A computing power card over-resolution and priority scheduling method, system, electronic device and storage medium

Abstract

The present invention discloses a method, system, electronic device and storage medium for super-resolution and priority scheduling of computing power cards, including dividing a single physical GPU into multiple logical GPU instances, presetting a single physical GPU super-score and a core number limit and a video memory limit for each logical GPU instance; registering the super-score, core number limit and video memory limit to the Kubernetes of the node; the Kubernetes scheduler identifies the declaration of the super-score, core number limit and video memory limit by the Pod during the scheduling process, and implements different scheduling strategies; the scheduling strategy includes a Binpack strategy and a Spread strategy, the Binpack strategy selects the highest score for scheduling, and the Spread strategy selects the lowest score for scheduling. The present invention effectively reduces the card fragmentation rate, and through a sophisticated scheduling strategy and super-resolution technology of GPU resources, can more reasonably allocate and manage GPU resources, reduce GPU fragmentation caused by improper resource allocation, and improve resource utilization efficiency and system performance.

Description

Technical Field

[0001] The present invention relates to the field of scheduling technology, and in particular to a computing power card over-resolution and priority scheduling method, system, electronic device and storage medium. Background Art

[0002] In the era of the rapid rise of large AI models, both the training of large models and the running of real-time reasoning tasks are highly dependent on the powerful computing capabilities of graphics processing units (GPUs). GPUs have demonstrated excellent performance in accelerating large-scale matrix operations and neural network calculations with their parallel processing architecture, becoming one of the key technologies supporting the development of AI. For machine learning platforms, how to efficiently manage and utilize GPU resources and improve resource utilization has become a crucial challenge.

[0003] As the de facto standard in the cloud native field, Kubernetes provides strong support for improving GPU utilization efficiency through its powerful container orchestration and cluster management capabilities. Kubernetes can automatically schedule containers to appropriate nodes based on resource requests and restrictions. Through fine-grained resource configuration strategies, such as the number and type of CPU, memory, and GPUs, it can ensure that tasks get enough resources while avoiding resource waste. In addition, the Kubernetes scheduler can support affinity and anti-affinity scheduling based on GPU types, so that specific types of tasks can run on specific GPU hardware and optimize resource utilization efficiency. In order to maximize the utilization of GPU resources, Kubernetes supports GPU sharing through specific device plug-ins (such as NVIDIA Device Plugin). These plug-ins allow multiple containers or Pods to share the same GPU, and allocate GPUs through virtualization technology or time slice rotation mechanism, thereby significantly improving resource utilization. At the same time, Kubernetes also provides a resource isolation mechanism to ensure that different workloads do not interfere with each other and ensure the stability and security of tasks.

[0004] Mainstream GPU sharing scheduling schemes aim to improve the utilization and scheduling fairness of GPU resources. To achieve this goal, these schemes usually use technical means such as resource isolation and scheduling algorithms. Resource isolation is a common technical means that divides the GPU device into multiple virtual GPUs, each of which can be assigned to a container for use. Tasks in different containers can be executed on independent virtual GPUs, improving the utilization of GPU resources. In addition to resource isolation, scheduling algorithms are also one of the core technologies of GPU sharing scheduling. Scheduling algorithms can reasonably allocate GPU resources according to the different characteristics and priorities of tasks, thereby ensuring the fairness and efficiency of GPU scheduling. Common scheduling algorithms include weight-based scheduling algorithms, queue-based scheduling algorithms, etc.

[0005] Although the current GPU sharing scheduling scheme provides flexible segmentation capabilities for high-performance computing power cards, allowing multiple containers or tasks to share the resources of the same GPU, thereby improving resource utilization and cost-effectiveness. However, this sharing mechanism is also accompanied by a significant problem - the fragmented card problem. The fragmented card problem mainly refers to the situation where when a GPU is shared by multiple tasks, the GPU resources are divided into multiple fragmented resources because each task has different requirements for GPU resources, and these requirements are often difficult to accurately predict and match. These fragmented resources may not be effectively utilized because their size may not meet the requirements of subsequent tasks, or they are difficult to recombine due to management complexity. Summary of the invention

[0006] In view of the shortcomings existing in the above problems, the present invention provides a computing power card over-resolution and priority scheduling method, system, electronic device and storage medium.

[0007] To achieve the above object, the present invention provides a computing power card over-resolution and priority scheduling method, comprising:

[0008] Divide a single physical GPU into multiple logical GPU instances, and pre-set the single physical GPU super score and the core number limit and video memory limit of each logical GPU instance;

[0009] Register the super score, the core number limit, and the video memory limit on Kubernetes of the node;

[0010] The Kubernetes scheduler identifies the Pod's declaration of the super score, the core number limit, and the video memory limit during the scheduling process, and implements different scheduling strategies;

[0011] The scheduling strategies include a Binpack strategy and a Spread strategy. The Binpack strategy selects the highest score for scheduling, and the Spread strategy selects the lowest score for scheduling.

[0012] Preferably, the scoring formula for the nodes is:

[0013] ;

[0014] Where: Indicates the number of physical GPUs that the current Pod needs to use. Indicates the number of physical GPUs used by the current node. Indicates the number of available GPUs of the current node. Represents weight.

[0015] Preferably, when a single Pod passes through the scheduler, the number of cards used, the video memory used, and the cores used by the Pod are obtained.

[0016] Preferably, the Pod is scored from three dimensions based on the number of cards used, the video memory used, and the cores used. The scoring formula is:

[0017] ;

[0018] ;

[0019] .

[0020] Preferably, the default weights of the super-resolution dimension and the video memory dimension are set to 0.4, and the default weight of the computing power dimension is set to 0.2. The default value is 10, and the final score is calculated as:

[0021] .

[0022] The present application also provides a computing power card over-resolution and priority scheduling system, including:

[0023] A partitioning module, used to partition a single physical GPU into multiple logical GPU instances, and pre-set the single physical GPU super-score and the core number limit and video memory limit of each logical GPU instance;

[0024] A registration module, used to register the super score, the core number limit and the video memory limit on Kubernetes of the node;

[0025] A scheduling module, used for the Kubernetes scheduler to identify the Pod's declaration of the super score, the core number limit, and the video memory limit during the scheduling process, and implement different scheduling strategies;

[0026] The strategy module is used for the scheduling strategies including Binpack strategy and Spread strategy. The Binpack strategy selects the highest score for scheduling, and the Spread strategy selects the lowest score for scheduling.

[0027] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0028] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention effectively reduces the card fragmentation rate, and through a sophisticated scheduling strategy and a super-resolution technology of GPU resources, can more reasonably allocate and manage GPU resources, reduce GPU fragmentation caused by improper resource allocation, and improve resource utilization efficiency and system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the computing power card super-resolution and priority scheduling method of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Reference Figure 1 The present invention provides a computing power card super-resolution and priority scheduling method, comprising:

[0034] Divide a single physical GPU into multiple logical GPU instances, and pre-set the single physical GPU super score and the core number limit and video memory limit for each logical GPU instance;

[0035] Register the super score, core limit, and memory limit to the node's Kubernetes;

[0036] The Kubernetes scheduler recognizes Pod's declarations of super scores, core limits, and memory limits during the scheduling process, and implements different scheduling strategies;

[0037] The scheduling strategies include Binpack strategy and Spread strategy. The Binpack strategy selects the highest score for scheduling, while the Spread strategy selects the lowest score for scheduling.

[0038] In this embodiment, in the Kubernetes environment, API hijacking technology and Device Plugin mechanism are combined to achieve super-resolution of computing power cards, which can finely control the allocation and isolation of GPU resources, such as Figure 1 As shown, it can be explained from the following three key dimensions:

[0039] GPU Super Score

[0040] Dividing a single physical GPU into multiple logical GPU instances enables multiple workloads to use GPU resources in parallel and independently. This super-resolution capability greatly improves the utilization of GPU resources and effectively meets diverse computing needs.

[0041] Number of GPU cores

[0042] Set a core limit for each logical GPU instance to intercept the use of GPU cores by user-mode drivers. Through time slicing, dynamically limit the GPU usage of each logical instance, which can more flexibly adapt to the performance requirements of different computing tasks, avoid excess or insufficient resources, and thus optimize the performance and cost-effectiveness of the overall system.

[0043] GPU memory size

[0044] A memory limit is set for each logical GPU instance. When a memory allocation request is intercepted, it is determined whether the requested memory exceeds the limit of the card corresponding to the container. If it exceeds, OOM is returned. Otherwise, the relevant information of the process memory allocation is recorded and the request is forwarded to the driver layer. Even if multiple workloads run on the same physical GPU at the same time, their memory usage will not interfere with each other, ensuring the stability and reliability of computing tasks.

[0045] To ensure the quality of service, the GPU super score is set to 10 by default; the number of GPU cores is set to 100 by default; the nvidia nvml library is called to obtain the actual GPU memory size. The above three resource indicators are registered to the Kubelet of the node through the Device Plugin. The Kubelet synchronizes the resource indicators to the Apiserver. The scheduler recognizes the Pod's declaration of the above three indicators during the scheduling process and implements different scheduling strategies.

[0046] Specifically, for node-level GPU scheduling strategies, the native Kubernetes scheduler supports multiple resource allocation strategies, including Binpack and Spread. The Binpack strategy aims to improve node resource utilization by tightly packing Pods onto a few nodes to reduce resource fragmentation. In contrast, the Spread strategy tends to disperse Pods onto multiple nodes to enhance the system's fault tolerance and fault isolation capabilities. After the introduction of super-division of computing power, GPUs will also be used as a scheduling topology domain, and the scheduler needs to consider the node-level GPU Binpack / Spread strategy.

[0047] Objective Function

[0048] The scoring formula for nodes is:

[0049] ;

[0050] Where: Indicates the number of physical GPUs that the current Pod needs to use. Indicates the number of physical GPUs used by the current node. Indicates the number of available GPUs of the current node. Represents the weight, which is used to balance the score information. It is used as a hyperparameter in the formula and its default value is 10.

[0051] If the scheduling strategy is Binpack, the node with the highest score is selected for scheduling; if the scheduling strategy is Spread, the node with the lowest score is selected for scheduling.

[0052] Single-card GPU scheduling strategy, when setting the single-card level Binpack strategy, our goal is to optimize resource usage and reduce fragmentation, so we prefer to schedule tasks on the same card to more effectively utilize all the resources of the card. Spread strategy is the opposite, the scheduler will tend to prioritize the deployment of tasks to the card with lower current utilization, to achieve balanced distribution of resources and reduce single point problems.

[0053] The scheduler cache maintains the total super score, used super score, total memory, used memory, total cores, and used cores of each GPU card. When a single Pod passes through the scheduler, the number of cards used by the Pod, the memory used, and the cores used are obtained, and the single card is scored from the following three dimensions.

[0054] ;

[0055] ;

[0056] .

[0057] Considering that computing power can be compressed, but super-resolution and video memory cannot be compressed, we set the default weights of super-resolution and video memory dimensions to 0.4, and the default weight of computing power dimension to 0.2. Used to balance the score information, as a hyperparameter in the formula, the default value is 10. The final score calculation formula is:

[0058] .

[0059] If the scheduling strategy is Binpack, the computing power card with the highest score is selected for scheduling; if the scheduling strategy is Spread, the computing power card with the lowest score is selected for scheduling.

[0060] This application significantly improves GPU resource utilization. Through GPU super-resolution technology, a single physical GPU is divided into multiple logical instances, so that GPU resources can be efficiently utilized to meet diverse computing needs; intelligent scheduling and optimization, node-level and single-card-level GPU scheduling strategies can be flexibly adjusted according to actual needs, optimize resource allocation, and improve overall system performance. The Spread strategy improves the system's fault tolerance and fault isolation capabilities by distributing Pods to multiple nodes or GPU cards, and enhances the system's resilience and availability. Effectively reduce the card fragmentation rate. Through sophisticated scheduling strategies and GPU resource super-resolution technology, GPU resources can be allocated and managed more reasonably, reducing GPU fragmentation caused by improper resource allocation, and improving resource utilization efficiency and system performance. Example 1

[0061] Single logical GPU scheduling

[0062] Create the following workload, using 1 logical GPU, 30 cores and 3000M video memory.

[0063] apiVersion: batch.volcano.sh / v1alpha1

[0064] kind: Job

[0065] metadata:

[0066] name: job-test

[0067] spec:

[0068] minAvailable: 1

[0069] schedulerName: volcano

[0070] queue: queue-vgpu

[0071] policies:

[0072] - event: PodEvicted

[0073] action: RestartJob

[0074] tasks:

[0075] - replicas: 1

[0076] name: nginx

[0077] policies:

[0078] - event: TaskCompleted

[0079] action: CompleteJob

[0080] template:

[0081] spec:

[0082] containers:

[0083] - command:

[0084] - sleep

[0085] - 10000

[0086] image: nginx:latest

[0087] name: nginx

[0088] resources:

[0089] requests:

[0090] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vgpu: 1

[0091] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vmemory: 3000

[0092] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vcore: 30

[0093] limits:

[0094] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vgpu:1

[0095] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vmemory: 3000

[0096] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vcore: 30

[0097] restartPolicy: Never

[0098] Verify the node-level GPU scheduling strategy under a single logical GPU

[0099] Two nodes are pre-set, node A contains 2 computing power cards, and node B contains 4 computing power cards. It is expected that under the Binpack scheduling strategy, tasks are scheduled to node A; under the Spread scheduling strategy, tasks are scheduled to node B.

[0100] Verify the GPU scheduling strategy at the card level under a single logical GPU

[0101] A node is pre-set, which contains 2 computing cards. Computing card A is used by other tasks, and computing card B is not used. It is expected that under the Binpack scheduling strategy, tasks are scheduled to computing card A; under the Spread scheduling strategy, tasks are scheduled to computing card B. Example 2

[0102] Multi-Logical GPU Scheduling

[0103] Create the following workload, using 2 logical GPUs, equally splitting 60 cores and 6000M of video memory.

[0104] apiVersion: batch.volcano.sh / v1alpha1

[0105] kind: Job

[0106] metadata:

[0107] name: job-test

[0108] spec:

[0109] minAvailable: 1

[0110] schedulerName: volcano

[0111] queue: queue-vgpu

[0112] policies:

[0113] - event: PodEvicted

[0114] action: RestartJob

[0115] tasks:

[0116] - replicas: 1

[0117] name: nginx

[0118] policies:

[0119] - event: TaskCompleted

[0120] action: CompleteJob

[0121] template:

[0122] spec:

[0123] containers:

[0124] - command:

[0125] - sleep

[0126] - 10000

[0127] image: nginx:latest

[0128] name: nginx

[0129] resources:

[0130] requests:

[0131] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vgpu: 2

[0132] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vmemory: 6000

[0133] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vcore: 60

[0134] limits:

[0135] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vgpu: 2

[0136] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vmemory: 6000

[0137] qkd.sh / NVIDIA-GeForce-RTX-2080-8GB-Vcore: 60

[0138] restartPolicy: Never

[0139] Verify node-level GPU scheduling strategy under multiple logical GPUs

[0140] Two nodes are pre-set, node A contains 2 computing power cards, and node B contains 4 computing power cards. It is expected that under the Binpack scheduling strategy, tasks are scheduled to node A; under the Spread scheduling strategy, tasks are scheduled to node B.

[0141] Verify the GPU scheduling strategy at the single card level under multiple logical GPUs

[0142] A node is pre-set, which contains 4 computing cards. Computing cards A and B are already used by other tasks, and computing cards C and D are not used. It is expected that under the Binpack scheduling strategy, it will be scheduled to computing cards A and B; under the Spread scheduling strategy, it will be scheduled to computing cards C and D.

[0143] The present application also provides a computing power card over-resolution and priority scheduling system, including:

[0144] The partitioning module is used to partition a single physical GPU into multiple logical GPU instances, pre-set the single physical GPU super score and the core number limit and video memory limit of each logical GPU instance;

[0145] A registration module, which is used to register the super score, core limit, and memory limit to the node's Kubernetes;

[0146] The scheduling module is used by the Kubernetes scheduler to identify Pod's declarations of super scores, core number limits, and video memory limits during the scheduling process, and implement different scheduling strategies;

[0147] The strategy module is used for scheduling strategies including Binpack strategy and Spread strategy. The Binpack strategy selects the highest score for scheduling, while the Spread strategy selects the lowest score for scheduling.

[0148] The present invention also provides an electronic device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0149] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0150] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for super-division and priority scheduling of computing power cards, characterized in that: include: Divide a single physical GPU into multiple logical GPU instances, and pre-set the single physical GPU super score and the core number limit and video memory limit of each logical GPU instance; Register the super score, the core number limit, and the video memory limit on Kubernetes of the node; The Kubernetes scheduler identifies the Pod's declaration of the super score, the core number limit, and the video memory limit during the scheduling process, and implements different scheduling strategies; The scheduling strategy includes a Binpack strategy and a Spread strategy. The Binpack strategy selects the highest score for scheduling, and the Spread strategy selects the lowest score for scheduling. The scoring formula for nodes is: ； Where: Indicates the number of physical GPUs that the current Pod needs to use. Indicates the number of physical GPUs used by the current node. Indicates the number of available GPUs of the current node. represents weight; When a single Pod passes through the scheduler, the number of cards, video memory, and cores used by the Pod are obtained; Based on the number of cards used, video memory used, and cores used, the Pod is scored from three dimensions. The scoring formula is: ； ； ； The default weights of the super-resolution dimension and the video memory dimension are set to 0.4, and the default weight of the computing power dimension is set to 0.

2. The default value is 10, and the final score is calculated as: 。 2. A computing power card super-resolution and priority scheduling system, characterized in that: include: A partitioning module, used to partition a single physical GPU into multiple logical GPU instances, and pre-set the single physical GPU super-score and the core number limit and video memory limit of each logical GPU instance; A registration module, used to register the super score, the core number limit and the video memory limit on Kubernetes of the node; A scheduling module, used for the Kubernetes scheduler to identify the Pod's declaration of the super score, the core number limit, and the video memory limit during the scheduling process, and implement different scheduling strategies; Strategy module, used for the scheduling strategies including Binpack strategy and Spread strategy, wherein the Binpack strategy selects the highest score for scheduling, and the Spread strategy selects the lowest score for scheduling; The scoring formula for nodes is: ； Where: Indicates the number of physical GPUs that the current Pod needs to use. Indicates the number of physical GPUs used by the current node. Indicates the number of available GPUs of the current node. represents weight; When a single Pod passes through the scheduler, the number of cards, video memory, and cores used by the Pod are obtained; Based on the number of cards used, video memory used, and cores used, the Pod is scored from three dimensions. The scoring formula is: ； ； ； The default weights of the super-resolution dimension and the video memory dimension are set to 0.4, and the default weight of the computing power dimension is set to 0.

2. The default value is 10, and the final score is calculated as: 。 3. An electronic device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to claim 1.

4. A storage medium, characterized in that: The computer program executable by an electronic device is stored therein, and when the program is run on the electronic device, the electronic device executes the method according to claim 1 .

Images