Workload configuration optimization method and system under multi-instance GPU

Abstract

The invention discloses a workload configuration optimization method and system under a multi-instance GPU, and the method comprises the steps: obtaining a set of undistributed but distributable regions of each GPU, obtaining an undistributed feasible partition of each GPU, and obtaining the current workload of each GPU; for the non-allocated feasible partition and the current workload of each GPU, determining a GPU allocation scheme of workload placement with maximum income and minimum cost according to a preset objective function and constraint conditions, the objective function including the income for maximizing the workload; the constraints are used to ensure that the workload placed on the GPU does not exceed its available compute instances and memory while minimizing GPU usage costs, re-partitioning costs, workload migration costs, and resource waste costs. According to the method, the partition function of the multi-instance GPU is efficiently utilized, the optimal workload placement and migration are realized, and the LLM reasoning task is more efficiently operated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and large-scale language models, and in particular to a method and system for optimizing workload configuration under a multi-instance GPU. Background Art

[0002] GPUs are at the heart of LLM pre-training, fine-tuning, and inference. They have become one of the most precious and expensive resources in computing, with demand far outstripping availability.

[0003] Large language models (LLMs) require powerful computing power, especially GPUs. As hardware devices with strong parallel processing capabilities, GPUs play a vital role in the reasoning process of large models. However, with the rapid development of generative AI and LLMs, GPUs also face many problems and challenges in reasoning applications.

[0004] LLM is one of the most advanced natural language processing technologies in the current field of artificial intelligence, capable of handling a variety of complex language tasks. Its large-scale parameters and powerful language modeling capabilities have enabled it to achieve remarkable results in the fields of dialogue, text generation, and translation, but it also faces problems such as high consumption of computing resources and inference efficiency.

[0005] MIG (Multi-Instance GPU) technology allows the GPU to be divided into multiple instances, so that multiple tasks can share one GPU resource. For example, LLM inference tasks can be deployed as workloads on MIG. However, in actual applications, it faces problems such as insufficient resource utilization, low migration efficiency, and GPU fragmentation.

[0006] In summary, the current GPU resource allocation for large language models has the following shortcomings:

[0007] 1. Lack of intelligent workload placement strategy: Traditional workload allocation strategies mostly adopt simple "first come, first served" or fixed load balancing algorithms, ignoring the specific requirements of different workloads for GPU resources and failing to achieve refined resource management.

[0008] 2. GPU fragmentation problem: As tasks increase or decrease, GPU resources may become fragmented, resulting in some GPU resources not being fully utilized. Existing partitioning and migration strategies cannot effectively solve this problem.

[0009] 3. Low migration efficiency: When a task is migrated from one GPU to another, there will be a certain amount of overhead. The current technology lacks efficient migration planning, which may lead to performance degradation during task migration.

[0010] 4. The LLM reasoning task runs on MIG, which does not fully utilize the computing and memory resources of MIG. Summary of the invention

[0011] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a method and system for optimizing workload configuration under a multi-instance GPU are provided to ensure efficient use of the partitioning function of the multi-instance GPU, achieve optimal workload placement and migration, and thereby achieve more efficient operation of LLM reasoning tasks.

[0012] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0013] A method for optimizing workload configuration under a multi-instance GPU comprises the following steps: obtaining unallocated feasible partitions of each GPU from currently existing GPUs, wherein the unallocated feasible partitions refer to a collection of regions that are not allocated but can be allocated by the GPU; obtaining the current workload of each GPU; and determining a GPU allocation scheme for workload placement that maximizes benefits and minimizes costs according to preset objective functions and constraints for the unallocated feasible partitions of each GPU and the current workload, wherein the objective function comprises maximizing the benefits of the workload while minimizing the GPU usage cost, the repartitioning cost, the workload migration cost, and the resource waste cost, and the constraints are used to ensure that the workload placed on the GPU does not exceed its available computing instances and memory.

[0014] Furthermore, the functional expression of the objective function is:

[0015]

[0016] In the above formula:

[0017] The maximum benefit is achieved by placing workload w on GPU g where the unassigned feasible partition is located, p w is the benefit of placing workload w, x wg is a decision variable, indicating whether workload w is assigned to GPU g, which is 1 if yes and 0 otherwise;

[0018] is the cost of using GPU g, q g is the cost of using GPU g, y g The usage flag of GPU g, if GPU g is used, it is 1, otherwise it is 0;

[0019] is the cost of repartitioning, represents the penalty factor for reconfiguring GPU g;

[0020] is the workload migration cost of workload w from partitioned GPU g′ to GPU g, represents the penalty factor for migrating workload w, x wg′ is the allocation flag of GPU g′. If workload w is allocated to GPU g′, it is 1, otherwise it is 0. wg is the allocation flag of GPU g, which is 1 if workload w is allocated to GPU g, otherwise it is 0;

[0021] is the resource waste cost of GPU g, is the penalty factor for unused resources on GPU g, U g is the wasted computing resources on GPU g, V g It is the wasted memory resource on GPU g.

[0022] Furthermore, the constraints include:

[0023] The workload on each GPU g where the unassigned feasible partition is located cannot exceed the total amount of its computing instances, as expressed by:

[0024]

[0025] For a partitioned GPU g′, the total workload in each partition cannot exceed the computing capacity of the partition, as expressed as follows:

[0026]

[0027] The number of partitions in a partitioned GPU g′ does not exceed the number of available compute instances, as expressed by:

[0028]

[0029] In the above formula, C g represents the total number of computational slices of GPU g, C g′ represents the total number of computing slices of GPU g′, z g′ is an allocation auxiliary variable, which is 1 if the workload w is allocated to a partitioned GPU g′ of the available partition set P in the set of GPUs G′ with existing workloads, and 0 otherwise.

[0030] Furthermore, the constraints also include:

[0031] Each workload can only be placed in one GPU or partition, as expressed below:

[0032]

[0033] The workload computational requirements on each GPU cannot exceed the total computational capacity:

[0034]

[0035] The workload memory requirement per GPU cannot exceed the total memory:

[0036]

[0037] In the above formula, c w is the computational slice required for workload w, u g is the computing resource usage on GPU g; m w is the memory required by workload w, S g is the common memory factor across GPU slices, M g is the total memory size of GPU g.

[0038] Furthermore, the constraints also include:

[0039] Calculate the wasted computing resources U on GPU g where the feasible partition is not allocated due to the imbalance of computing and memory usage g , the expression is as follows:

[0040]

[0041] Calculate the computing resource usage u on GPU g where the unassigned feasible partition is located g The interval is as follows:

[0042]

[0043] Calculate the wasted memory resources V when the computing resources on GPU g where the unallocated feasible partition is full g , the expression is as follows:

[0044]

[0045] In the above formula, v g is the memory usage on GPU g, δ g It is an auxiliary binary variable, indicating whether the computation slice of GPU g is full. If yes, it is 1, otherwise it is 0.

[0046] Furthermore, currently available GPUs include virtual GPUs, and the constraints also include:

[0047] When the virtual GPU is used, the original GPU is no longer used. The expression is as follows:

[0048]

[0049] In the above formula, y g′ The usage flag of GPU g′, which is 1 if GPU g′ is used, otherwise 0.

[0050] The present invention also proposes a workload configuration optimization system under a multi-instance GPU, comprising:

[0051] Preprocessing module: used to obtain the unallocated feasible partitions of each GPU from the currently existing GPUs, where the unallocated feasible partitions refer to the set of areas that are not allocated but can be allocated by the GPU; obtain the current workload of each GPU;

[0052] The resource allocation module is used to determine the GPU allocation scheme for workload placement that maximizes benefits and minimizes costs for each GPU's unallocated feasible partitions and current workloads according to preset objective functions and constraints. The objective function includes maximizing the benefits of the workload while minimizing GPU usage costs, repartitioning costs, workload migration costs, and resource waste costs. The constraints are used to ensure that the workload placed on the GPU does not exceed its available computing instances and memory.

[0053] The present invention also proposes a workload configuration optimization device under a multi-instance GPU, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute any one of the workload configuration optimization methods under the multi-instance GPU.

[0054] The present invention also proposes a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute any one of the workload configuration optimization methods under a multi-instance GPU through a processor.

[0055] The present invention also proposes a computer program product, including a computer program or instructions, which are programmed or configured to execute any one of the workload configuration optimization methods under a multi-instance GPU through a processor.

[0056] Compared with the prior art, the present invention mainly has the following advantages:

[0057] The objective function constructed by the present invention maximizes the benefits of workload placement and minimizes the costs of GPU use, migration and resource waste. The constructed constraints ensure that the workload can be reasonably placed in each GPU or partition while avoiding exceeding the computing or memory capabilities of the GPU. Based on the unallocated feasible partitions of each GPU, the present invention redistributes the workload through the objective function and constraints, which can maximize the GPU utilization, reduce computing and memory waste, and support efficient scaling of workloads to adapt to frequent workload switching requirements under multi-instance GPUs, and minimize the time and computing overhead of migrating workloads and repartitioning to improve overall computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of an embodiment of the present invention.

[0059] Figure 2 This is a comparison diagram of the effects of the embodiments of the present invention. DETAILED DESCRIPTION

[0060] Before introducing the specific embodiments of the present invention, related concepts or terms are first explained.

[0061] GPU: Full name Graphics Processing Unit, which is a computing hardware specially designed for fast processing and rendering of graphics. Its main difference from the central processing unit (CPU) is its parallel processing capability, which can process a large amount of data at the same time, making it perform well in graphics rendering and complex computing tasks.

[0062] MIG: The full name is Multi-Instance GPU, which is a technology launched by NVIDIA. MIG allows the creation of multiple isolated instances on the same physical GPU, and each instance can run different workloads independently.

[0063] LLM: Large Language Model. LLM is an artificial intelligence model based on deep learning, which usually has billions or even hundreds of billions of parameters and can perform well in natural language processing tasks. LLM is trained on large amounts of text data to learn language patterns, grammar, and context, so that it can generate, understand, and process natural language.

[0064] Profile ID: The profile ID is used to identify the specific configuration of the GPU partition to which each workload is assigned in a MIG (multi-instance GPU) environment. Each profile ID corresponds to a fixed set of compute slices and memory slices, such as 2g.20gb, which means using 2 compute slices and 20GB of memory. Workload: A task with specific computing and memory requirements, such as a large LLM model computing task running on a multi-instance GPU, and is described by a pre-defined profile.

[0065] Unallocated feasible partitions: refers to the resource space on a GPU that can still be used to place workloads. This space must fit within the GPU's compute and memory capabilities and take into account the allocated workloads.

[0066] The patent application with publication number CN118377598A provides a cluster resource scheduling system for multi-instance GPUs. It is implemented based on kubernetes and includes 4 modules: a multi-instance GPU reasoning service job management module, a multi-instance GPU reasoning service job resource performance prediction module, a reasoning service job online scheduling module and a multi-instance GPU elastic resource management module; the multi-instance GPU elastic resource management module is connected to each agent in the cluster, and manages all state changes in the life cycle through the reasoning service job controller, and further the multi-instance GPU reasoning service job management module is connected to the multi-instance GPU reasoning service job resource performance prediction module and the reasoning service job online scheduling module for calculation. In order to achieve the design of a cluster resource scheduling system in a multi-instance GPU scenario, make full use of the multi-instance GPU characteristics, and improve the resource utilization effect. The patent focuses on kubernetes, while the present invention is generally applicable to multi-instance GPU usage scenarios under various platforms; at the same time, the patent is applicable to the management of multi-instance GPUs using agents under clusters, while the present invention does not rely on agents and is also applicable under non-clusters. Furthermore, the patent uses a serialization model and a random forest regression model, and does not involve the optimization method proposed by the present invention.

[0067] The patent application with publication number CN114637590A relates to a method for realizing artificial intelligence computing by a multi-instance GPU device based on a container, including the following steps: S100, the user proposes a demand for creating a computing container and sends it to the scheduling center; S200, the scheduling center applies to the GPU instantiation module and the computing instance module respectively according to the computing container demand proposed by the user; S300, the GPU instantiation module obtains the available GPU instance type from the GPU instance type library in real time or creates the GPU instance type issued by the user; S400, the computing instance module obtains the corresponding image from the artificial intelligence container image library according to the artificial intelligence computing request issued by the user, and creates the corresponding computing instance. Each GPU instance cut according to this scheme has its own processor and memory system, ensuring that the latency and throughput of each user's work are predictable, and through a flexible resource allocation method, it avoids the situation of idle resources and improves the utilization rate of the GPU. The invention is based on containers, and the present invention is applicable to multi-instance GPU usage scenarios under various platforms, and the invention focuses on the process of creating computing instances for container images through the scheduling center, and does not involve the optimization method proposed by the present invention.

[0068] The patent with publication number CN105677486B provides a data parallel processing method, including the following steps: step 1, the main management node receives data and obtains the association relationship of the data; step 2, the main management node calculates the deployable GPU and GPU workload of the working computing node; step 3, the main management node divides the data and distributes the divided data to each working computing node; step 4, the working computing node processes the received data in parallel and transmits the processed results back to the main management node; step 5, the main management node merges and outputs the results. The beneficial effects of the present invention are as follows: the master / slave architecture mode is adopted for high-performance large-scale data parallel processing, and the specific jobs converted by the application are divided into job stages according to DNA feature modeling, and the node granularity level job deployment is performed according to the division results, and the thread parallel optimization mechanism is adopted to make full use of multiple computing cores to improve the execution efficiency of data flow parallel tasks in a single node. However, it is limited to the GPU level, and does not involve the multi-instance GPU focused on by the present invention, and the patent focuses on the data parallel processing method, and does not involve the workload optimization inside the multi-instance GPU focused on by the present invention.

[0069] The patent with publication number CN113419846B discloses a resource configuration method and device, an electronic device and a computer-readable storage medium. The method comprises: scheduling instances in a resource pool according to a user's resource request to generate allocation information, wherein the resource request at least includes the storage usage and computing power usage requested by the user, and the allocation information at least includes the instance identifier of the instance allocated to the user; according to the allocation information, the instance allocated to the user is bound to the user's application. The embodiment of the present application can schedule instances in a resource pool according to the user's resource request, thereby generating resource allocation information and binding the instance identified by the allocation information to the application requested by the user, so that it can respond to the user's resource configuration needs more flexibly and improve the utilization rate of the resources actually used. The patent focuses on scheduling instances in a resource pool according to the user's resource request, and does not involve the optimization model mentioned in the present invention. Moreover, it does not involve the intelligent workload placement strategy, reduction of workload migration cost, reduction of memory and computing resource waste, MIG fragmentation solution, and support for efficient workload scaling mentioned in the present invention. At the same time, the invention mainly involves binding resources with user applications to achieve scheduling; it does not involve the method of optimizing workload placement in the present invention.

[0070] Embodiment 1

[0071] To optimize workload placement under multi-instance GPUs, the following issues need to be addressed:

[0072] 1. Maximize GPU utilization: By optimizing the placement of workloads, ensure that as few GPU resources as possible are used to meet multi-tasking requirements and avoid resource waste.

[0073] 2. Reduce computing and memory waste: Avoid GPU resource waste caused by uneven allocation by optimizing the GPU partition configuration.

[0074] 3. Reduce migration overhead: When workload migration is required, minimize the migration time and computing overhead to improve overall computing efficiency.

[0075] 4. Support efficient scaling of workloads: Support changing task requirements by automatically adjusting GPU resource configuration.

[0076] This embodiment proposes a workload configuration optimization method under multi-instance GPU to solve the above problems. The method of this embodiment is based on an optimization model, which is used to optimize the workload placement under multi-instance GPU to maximize the benefit of the workload while minimizing the GPU usage cost, repartitioning cost, workload migration cost and resource waste cost.

[0077] like Figure 1 As shown, the method of this embodiment includes the following steps:

[0078] S1) obtaining the unassigned feasible partitions of each GPU from the currently existing GPUs, and obtaining the current workload of each GPU;

[0079] S2) For each GPU's unassigned feasible partitions and current workload, an optimization model is used to perform model calculations to obtain a GPU allocation solution for workload placement that maximizes benefits and minimizes costs.

[0080] The optimization model in this embodiment is implemented by defining an objective function and constraints, wherein the objective function includes maximizing the benefits of the workload while minimizing the GPU usage cost, repartitioning cost, workload migration cost, and resource waste cost, and the constraints are used to ensure that the workload placed on the GPU does not exceed its available computing instances and memory. At the same time, if the GPU uses a virtual instance (i.e., the original GPU has been repartitioned), the original GPU is not allowed to be used again.

[0081] Before introducing the objective function and constraints of the optimization model, the relevant parameters are explained first:

[0082] xg.ygb: indicates a configuration using x computational slices and yGB of memory.

[0083] w: workload, such as an LLM large model instance.

[0084] g,g′: GPU index.

[0085] W: The set of transferable and to-be-assigned models.

[0086] G: The collection of available GPUs and virtual GPUs.

[0087] G′: The set of GPUs with existing workloads.

[0088] Gi: A collection of virtual GPUs, Introduce for each g′∈G′.

[0089] P: the set of available partitions in G′, P = p1∪p2∪P3…P |G′| .

[0090] A: The allocation set of the current model and GPU, (w,g′)∈A means that the model w is deployed on GPU g′.

[0091] B: The set of pairs (g′,g) of GPUs and their virtual counterparts, where g′∈G′, g∈Gi.

[0092] p w: The benefit of placing workload w.

[0093] q g : The cost of using GPU g.

[0094] c w : The computational slices required for workload w.

[0095] m w : Memory required by workload w.

[0096] C g : The total number of compute slices on GPU g (e.g., C in A100 g =7).

[0097] M g : The total memory size of GPU g (for example, M in A100 g =80).

[0098] S g : Common memory factor across GPU slices (e.g., S in A100 g =10).

[0099] Penalty factor for migrating workload w.

[0100] Reconfigure the penalty factor for GPU g.

[0101] Penalty factor for unused resources on GPU g.

[0102] x wg : 1 if workload w is assigned to GPU g, 0 otherwise.

[0103] y g : 1 if GPU g is used, 0 otherwise.

[0104] z g : Auxiliary variable, equal to 1 when the model is assigned to partition g of P, and equal to 0 otherwise.

[0105] u g : Computational resource usage on GPU g.

[0106] v g : Memory usage on GPU g.

[0107] U g : Wasted computing resources.

[0108] V g : Wasted memory resources.

[0109] δ g : Auxiliary binary variable indicating whether the computation slice of GPU g is full.

[0110] In this embodiment, the function expression of the objective function is:

[0111]

[0112] In the above formula:

[0113] The maximum benefit is achieved by placing workload w on GPU g where the unassigned feasible partition is located, p w is the benefit of placing workload w, x wg is a decision variable, indicating whether workload w is assigned to GPU g, which is 1 if yes and 0 otherwise;

[0114] is the cost of using GPU g. In this embodiment, the cost of using GPU g needs to be minimized. g is the cost of using GPU g, y g The usage flag of GPU g, if GPU g is used, it is 1, otherwise it is 0;

[0115] is the cost of repartitioning. In this embodiment, the cost of repartitioning needs to be minimized. represents the penalty factor for reconfiguring GPU g;

[0116] is the workload migration cost of migrating workload w from the partitioned GPU g′ to GPU g. In this embodiment, the overhead of migrating workload from the original GPU g′ to the new GPU g is considered. represents the penalty factor for migrating workload w, x wg′ is the allocation flag of GPU g′. If workload w is allocated to GPU g′, it is 1, otherwise it is 0. wg is the allocation flag of GPU g, which is 1 if workload w is allocated to GPU g, otherwise it is 0;

[0117] is the resource waste cost of GPU g, is the penalty factor for unused resources on GPU g, U g is the wasted computing resources on GPU g, V g It is the wasted memory resource on GPU g.

[0118] In formula (1), max means solving the maximum value of the entire expression under the given constraints of all decision variables x, y, z, δ and u, v, U, V. In other words, the goal of the model is to find the optimal combination of decision variables to maximize the overall objective function. x, y, z, δ are binary variables with values ​​of 0 or 1, indicating a specific decision. u, v, U, V are non-negative continuous variables indicating the usage of computing resources and memory.

[0119] The constraints in this embodiment ensure that the system can properly place workloads in each GPU or partition to avoid exceeding the computational or memory capabilities of the GPU. For virtual and repartitioned GPUs, there are additional constraints to avoid resource conflicts or multiple allocations. These include:

[0120] 1. The workload on each GPU g where the unassigned feasible partition is located cannot exceed the total amount of its computing instances, as expressed as follows:

[0121]

[0122] 2. For a partitioned GPU g′, the total workload in each partition cannot exceed the computing capacity of the partition, as expressed as follows:

[0123]

[0124] 3. The number of partitions in the partitioned GPU g′ does not exceed the number of available computing instances, as expressed as follows:

[0125]

[0126] 4. Each workload can only be placed in one GPU or partition, as expressed below:

[0127]

[0128] 5. The workload computing requirements on each GPU cannot exceed the total computing capacity, as expressed as follows:

[0129]

[0130] 6. The workload memory requirement of each GPU cannot exceed the total memory, as expressed as follows:

[0131]

[0132] 7. The original GPU is no longer used when the virtual GPU is used:

[0133]

[0134] 8. Calculate the wasted computing resources U due to the imbalanced computing and memory usage on GPU g where the unallocated feasible partition is located g , the expression is as follows:

[0135]

[0136] 9. Calculate the computing resource usage u on GPU g where the unassigned feasible partition is located g When the computing resources are full, it is marked as full. The expression is as follows:

[0137]

[0138] 10. Calculate the wasted memory resources V when the computing resources on GPU g where the unallocated feasible partition is full g , the expression is as follows:

[0139]

[0140] In the above formula, C g represents the total number of computational slices of GPU g, C g′ represents the total number of computing slices of GPU g′, z g′ is an allocation auxiliary variable, which is 1 if the workload w is allocated to a partitioned GPU g′ in the available partition set P in the GPU set G′ with existing workloads, otherwise it is 0; c w is the computational slice required for workload w, u g is the computing resource usage on GPU g; m w is the memory required by workload w, S g is the common memory factor across GPU slices, M g is the total memory size of GPU g; v g is the memory usage on GPU g, δ g is an auxiliary binary variable, indicating whether the computation slice of GPU g is full, if yes, it is 1, otherwise, it is 0; g′ The usage flag of the partitioned original GPU g′, which is 1 if GPU g′ is used, otherwise 0.

[0141] It can be seen that the objective function and constraints of this embodiment maximize the benefits of workload placement and minimize the costs of GPU usage, migration, and resource waste. The system can optimize the use of GPU resources, while considering different cost items (including migration costs and resource waste costs) and making trade-offs based on actual needs, thereby achieving the purpose of maximizing GPU utilization and minimizing cost overhead.

[0142] The method of this embodiment is further explained below by taking NVIDIA A100 GPU (with 80GB of memory and an upper limit of 7 computing instances) and LLM reasoning tasks as workloads as examples.

[0143] Given a set of GPU clusters G and a set of workloads (LLM inference tasks) W to be placed. Each workload has specific computational requirements and memory requirements. The resource constraints of the GPU ensure that the computational and memory requirements do not exceed their total capacity. Figure 2 As shown, there is currently a load on each multi-instance GPU, and there are three multi-instance GPUs, but each GPU has idle resources, resulting in a waste of memory and computing resources.

[0144] In step S1, the GPU is regarded as a container, and each GPU may be empty, partially partitioned, or partitioned, to ensure that the computing and memory requirements do not exceed the total capacity of the GPU. It is necessary to preprocess the existing GPUs to obtain the set of unallocated feasible partitions, that is, the set of areas that are not allocated but can be allocated by the GPU.

[0145] As attached Figure 2 As shown, GPU1 currently has load W1 that uses 4 computing slices and 40GB of memory resources, and has 3 computing slices and 40GB of memory resources left; GPU2 currently has load W2 that uses 3 computing slices and 40GB of memory resources, and load W3 that uses 2 computing slices and 20GB of memory resources, and GPU2 has 2 computing slices and 20GB of memory resources left (because workload W2 uses 40GB of memory, and each computing slice has a mapping relationship with 10GB of memory resources, in order to meet the 40GB memory resource requirement, although W2 only applies for 3 computing slices, it also has to occupy the positions of 4 computing slices, wasting one computing slice); similarly, workload W6 in GPU3 also wastes one memory slice.

[0146] In step S2, after obtaining the unassigned feasible partitions of each GPU, an objective function (such as formula (1)) is constructed to maximize the benefit and minimize the cost according to the resources and workload defined in the previous text. The constraints (formulas (2) to (11)) ensure the rational use of resources and avoid overload and waste of resources. Figure 2 C of each GPU g =7 (number of slices to calculate), M g =80 (total memory size), subscript g indicates additional Figure 1Then, the model is operated to solve the optimization problem for the decision variables x, y, z, δ and u, v, U, V based on the objective function and constraints. During the calculation process, if the resources are insufficient, the model will introduce a virtual GPU to simulate the repartitioning requirement. The solution will provide guidance for GPU workload migration and resource reallocation to ensure maximum computing efficiency. For example, Figure 2 Workloads W2 and W6 require more memory resources than the corresponding memory of their own computing slices, so they are suitable to be placed in the last position, because A100 has 7 computing slices and 80GB of memory, and the last 10GB of memory resources can only be used by the last computing slice. Finally, the workload placement set that maximizes the benefits and minimizes the costs of multi-instance GPUs is obtained. Figure 2 As shown in , the optimal set is as follows:

[0147] GPU1: {4g.40gb, 3g.40gb}

[0148] GPU2: {2g.20gb,1g.10gb,1g.10gb,1g.10gb,1g.10gb,1g.20gb}

[0149] Figure 2 The blank space in GPU2 indicates that the space is currently idle, but can be allocated to 1g.10gb workloads (such as LLM inference tasks).

[0150] It can be seen that the result of the model operation first saves a GPU, and at the same time there is no waste of memory resources or computing slices, thereby achieving the purpose of maximizing GPU utilization efficiency and minimizing resource waste and cost.

[0151] Embodiment 2

[0152] This embodiment provides a workload configuration optimization system under a multi-instance GPU, including:

[0153] Preprocessing module: used to obtain the unallocated feasible partitions of each GPU from the currently existing GPUs, where the unallocated feasible partitions refer to the set of areas that are not allocated but can be allocated by the GPU; obtain the current workload of each GPU;

[0154] A resource allocation module is used to perform model calculations using an optimization model for each GPU's unallocated feasible partitions and current workloads, and determine a GPU allocation scheme for workload placement that maximizes benefits and minimizes costs based on the objective function and constraints preset in the optimization model, wherein the objective function is the objective function in Embodiment 1, including maximizing the benefits of the workload while minimizing GPU usage costs, repartitioning costs, workload migration costs, and resource waste costs, and the constraints are the constraints in Embodiment 1, which are used to ensure that the workload placed on the GPU does not exceed its available computing instances and memory.

[0155] Embodiment 3

[0156] This embodiment proposes a workload configuration optimization device under a multi-instance GPU, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the workload configuration optimization method under a multi-instance GPU described in Example 1.

[0157] This embodiment also proposes a computer-readable storage medium, which stores a computer program or instruction. The computer program or instruction is programmed or configured to execute the workload configuration optimization method under the multi-instance GPU described in Example 1 through a processor.

[0158] This embodiment also proposes a computer program product, including a computer program or instructions, which are programmed or configured to execute the workload configuration optimization method under the multi-instance GPU described in Example 1 through a processor.

[0159] In summary, the present invention proposes a method and system for optimizing workload configuration under multi-instance GPUs. By using optimization models applicable to various multi-instance GPU clusters, the workload placement on GPUs is optimized, the cost in the workload migration and repartitioning process is reduced, and the following goals under multi-instance GPUs are achieved:

[0160] a. Maximize GPU utilization: By optimizing the placement of workloads, ensure that as few GPU resources as possible meet multi-tasking requirements and avoid resource waste.

[0161] b. Reduce computing and memory waste: Avoid GPU resource waste caused by uneven allocation by optimizing GPU partition configuration.

[0162] c. Reduce migration overhead: When workload migration is required, minimize the migration time and computing overhead to improve overall computing efficiency.

[0163] d. Support efficient scaling of workloads: Support changing task requirements by automatically adjusting GPU resource configuration.

[0164] Compared with the prior art, the advantages of the present invention are:

[0165] (1) From scratch, we implemented a method for efficient workload orchestration under multi-instance GPUs.

[0166] (2) A low-cost method for migrating and repartitioning workloads on multi-instance GPUs can minimize the time and computational overhead of migrating and repartitioning workloads to improve overall computing efficiency.

[0167] (3) A workload placement optimization model for multi-instance GPUs that can maximize GPU utilization, reduce computational and memory waste, and support efficient scaling of workloads to adapt to frequent workload switching requirements under multi-instance GPUs.

[0168] (4) The optimization model in the present invention supports the optimization of workload deployment under a multi-instance GPU cluster.

[0169] (5) MIG uses the optimization model of the present invention to run LLM reasoning tasks, and can use less GPU resources to run LLM reasoning tasks of the same scale.

[0170] Those skilled in the art will appreciate that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0171] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A workload configuration optimization method under a multi-instance GPU, characterized in that: The method comprises the following steps: obtaining unallocated feasible partitions of each GPU from currently existing GPUs, wherein the unallocated feasible partitions refer to a collection of areas that are not allocated to the GPU but can be allocated; obtaining the current workload of each GPU; and determining a GPU allocation scheme for workload placement that maximizes benefits and minimizes costs according to preset objective functions and constraints for the unallocated feasible partitions and current workload of each GPU, wherein the objective function comprises maximizing the benefits of the workload while minimizing the GPU usage cost, repartitioning cost, workload migration cost, and resource waste cost, and the constraints are used to ensure that the workload placed on the GPU does not exceed its available computing instances and memory.

2. The workload configuration optimization method under multi-instance GPU according to claim 1, characterized in that: The functional expression of the objective function is: In the above formula: The maximum benefit is achieved by placing workload w on GPUg where the unassigned feasible partition is located, p w is the benefit of placing workload w, x wg is a decision variable, indicating whether workload w is assigned to GPUg, which is 1 if yes and 0 otherwise; is the cost of using GPUg, q g is the cost of using GPUg, y g The usage flag of GPUg, if GPUg is used, it is 1, otherwise it is 0; is the cost of repartitioning, represents the penalty factor for reconfiguring GPUg; is the workload w from the partitioned GPU g ′ Migration cost of workloads migrated to GPUg, represents the penalty factor for migrating workload w, x wg′ is the allocation tag of GPUg′. If workload w is allocated to GPUg ′ , then it is 1, otherwise it is 0, x wg is the allocation flag of GPUg, which is 1 if workload w is allocated to GPUg, otherwise it is 0; is the resource waste cost of GPUg, is the penalty factor for unused resources on GPUg, U g is the wasted computing resources on GPUg, V g It is a waste of memory resources on GPUg.

3. The workload configuration optimization method under multi-instance GPU according to claim 2, characterized in that: The constraints include: The workload on each GPUg where the unassigned feasible partition is located cannot exceed the total amount of its computing instances, as expressed in the following expression: For a partitioned GPUg′, the total workload in each partition cannot exceed the computing capacity of the partition, as expressed as follows: The number of partitions in a partitioned GPUg′ does not exceed the number of available compute instances, as expressed by: In the above formula, C g Indicates the total number of computing slices on GPUg, C g ′ represents the total number of computing slices of GPUg′, z g′ To allocate auxiliary variables, if workload w is assigned to the GPU set G with existing workloads ′ It is 1 if the GPU g′ is a partitioned GPU in the available partition set P, otherwise it is 0.

4. The method for optimizing workload configuration under a multi-instance GPU according to claim 3, characterized in that: The constraints also include: Each workload can only be placed in one GPU or partition, as expressed below: The workload computational requirements on each GPU cannot exceed the total computational capacity: The workload memory requirement per GPU cannot exceed the total memory: In the above formula, c w is the computational slice required for workload w, u g is the computing resource usage on GPUg; m w is the memory required by workload w, S g is the common memory factor across GPU slices, M g is the total memory size of GPUg.

5. The method for optimizing workload configuration under a multi-instance GPU according to claim 4, characterized in that: The constraints also include: Calculate the wasted computing resources U on the GPUg where the unallocated feasible partitions are located due to the imbalance in computing and memory usage g , the expression is as follows: Calculate the computing resource usage u on GPUg where the unassigned feasible partition is located g The interval is as follows: Calculate the wasted memory resources V when the computing resources on the GPUg where the unallocated feasible partition is located are full g , the expression is as follows: In the above formula, v g is the memory usage on GPUg, δ g It is an auxiliary binary variable, indicating whether the computation slice of GPUg is full. If yes, it is 1, otherwise it is 0.

6. The method for optimizing workload configuration under a multi-instance GPU according to claim 4, characterized in that: The existing GPUs include virtual GPUs, and the constraints also include: When the virtual GPU is used, the original GPU is no longer used. The expression is as follows: In the above formula, y g′ For GPU ′ If GPUg ′ is used, 1; otherwise 0.

7. A workload configuration optimization system under multi-instance GPU, characterized in that: include: Preprocessing module: used to obtain the unallocated feasible partitions of each GPU from the currently existing GPUs, where the unallocated feasible partitions refer to the set of areas that are not allocated but can be allocated by the GPU; obtain the current workload of each GPU; The resource allocation module is used to determine the GPU allocation scheme for workload placement that maximizes benefits and minimizes costs for each GPU's unallocated feasible partitions and current workloads according to preset objective functions and constraints. The objective function includes maximizing the benefits of the workload while minimizing GPU usage costs, repartitioning costs, workload migration costs, and resource waste costs. The constraints are used to ensure that the workload placed on the GPU does not exceed its available computing instances and memory.

8. A workload configuration optimization device under a multi-instance GPU, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the workload configuration optimization method under a multi-instance GPU as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the workload configuration optimization method under a multi-instance GPU described in any one of claims 1 to 6 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the workload configuration optimization method under a multi-instance GPU described in any one of claims 1 to 6 through a processor.

Images