A general heterogeneous vgpu scheduling method and device

Abstract

This invention discloses a general heterogeneous VGPU scheduling method and device. The method includes: pre-creating a pre-occupied Pod and occupying physical GPU resources through Kubernetes' resource request mechanism; in response to receiving a VGPU task request, starting a VGPU task using a target pre-occupied Pod, wherein the target pre-occupied Pod is a pre-occupied Pod with shared quota determined based on pre-occupied Pod annotation information; in response to the completion of the VGPU task, updating the annotation of the target pre-occupied Pod, removing the identifier of the completed VGPU task, and releasing the shared quota occupied by the VGPU task to return the shared quota to the pre-occupied Pod for use by new VGPU tasks. This application solves the technical problems of poor versatility, the need for extensive customization and development, high system complexity, inability to dynamically adjust static solutions, poor flexibility, and high cost in existing VGPU solutions, achieving the technical effects of improving GPU versatility, improving GPU resource utilization, and being easy to implement and maintain.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a general heterogeneous VGPU scheduling method and device. Background Technology

[0002] With the rapid development of applications such as artificial intelligence and deep learning, the demand for GPUs (Graphics Processing Units), as the core resource for high-performance computing, is increasing daily. VGPU (Virtual Graphics Processing Unit) technology virtualizes physical GPU resources, allowing multiple tasks to share the same GPU, thereby improving resource utilization, reducing costs, and increasing scheduling flexibility.

[0003] Currently, the main implementation schemes of VGPU are as follows: (1) Utilize the virtualization capabilities provided by the GPU hardware itself to statically divide a physical GPU card into multiple virtual GPUs. (2) Statically divide a physical NPU (Neural Processing Unit) card into multiple vNPU (virtual Neural Processing Unit) instances. Each vNPU has its own dedicated computing core, video memory, and other hardware resources. They are completely isolated from each other and can only be used by different containers / virtual machines. The total number of divisions cannot exceed the hardware limit of the physical card. (3) Through the Kubernetes Webhook mechanism, intercept and modify the GPU resources when the Pod is created to achieve GPU resource virtualization.

[0004] However, solutions (1) and (2) are only applicable to high-end GPUs that support hardware virtualization, and do not support mid-to-low-end GPUs and most GPU models. Although the NPU supports virtualization capabilities, the number of partitions is limited by hardware and cannot be flexibly configured. Moreover, the partition configuration of solutions (1) and (2) is static and cannot be dynamically adjusted according to actual needs. Therefore, it cannot adapt to the different GPU resource requirements of different tasks. At the same time, static partitioning may cause some virtual GPU instances to be idle for a long time, resulting in waste. It is also necessary to purchase high-end GPUs that support hardware virtualization, which is costly. Solution (3) requires in-depth customization for specific GPU manufacturers, which involves a large amount of development work. Different GPU manufacturers require different implementation solutions, resulting in poor universality. At the same time, there may be compatibility issues with Kubernetes version upgrades, and the system architecture is complex, making troubleshooting difficult. Summary of the Invention

[0005] This invention provides a general heterogeneous VGPU scheduling method and device, which solves the technical problems of existing VGPU solutions, such as poor versatility, the need for a lot of customized development, high system complexity, static solutions that cannot be dynamically adjusted, poor flexibility, and high cost.

[0006] This invention provides a general heterogeneous VGPU scheduling method, the method comprising: Pre-create pre-occupied Pods and use Kubernetes' resource request mechanism to occupy physical GPU resources; In response to receiving a VGPU task request, a VGPU task is started using a target pre-occupied Pod, wherein the target pre-occupied Pod is a pre-occupied Pod with a shared quota determined based on the pre-occupied Pod annotation information, and the VGPU task is created without declaring a GPU resource request; In response to the completion of the VGPU task, the annotation of the target pre-occupied Pod is updated, the identifier of the completed VGPU task is removed, and the shared quota occupied by the VGPU task is released to return the shared quota to the pre-occupied Pod for use by new VGPU tasks.

[0007] Furthermore, the creation rules for the pre-owned Pod include: Each pre-occupied Pod occupies only one GPU card, using Kubernetes' resource request mechanism to allocate GPU resources; The pre-occupied Pod runs an idle service without occupying actual GPU computing resources; The pre-occupied Pod is configured to use different resource names for different GPU vendors when making resource requests.

[0008] Furthermore, pre-creating pre-occupied Pods and utilizing physical GPU resources through Kubernetes' native resource request mechanism includes: Plan GPU resource requirements; Based on the planning results, pre-occupied Pods are created and the corresponding physical GPU resources are occupied through the Kubernetes native resource request mechanism. Write annotations for the created pre-occupied Pod, wherein the annotations include at least the shared capacity of the pre-occupied Pod, the list of used tasks, GPU identification information, and current shared status information.

[0009] Furthermore, the VGPU task is identified during its creation, startup, and execution by at least one of the following methods: Identified by the annotations of the target pre-occupied Pod; Identify by setting labels; Identification is achieved by recording the association with the target pre-occupied Pod.

[0010] Furthermore, the planned GPU resource requirements include: Determine the number of GPUs that need to be reserved based on the total amount of GPU resources in the cluster and the expected VGPU task requirements. Based on a pre-defined heterogeneous GPU unified scheduling framework, it identifies and plans GPU resources from different manufacturers. Select a suitable node to deploy the pre-emptive Pod to avoid a single point of failure.

[0011] Furthermore, based on a pre-defined heterogeneous GPU unified scheduling framework, the system identifies and plans GPU resources from different vendors, including: Based on a pre-defined heterogeneous GPU unified scheduling framework, unified management and scheduling of VGPUs from different manufacturers and architectures can be achieved through at least one of the following methods: resource name mapping, device path mapping, and environment variable mapping.

[0012] Furthermore, in response to receiving a VGPU task request, launching a VGPU task using the target pre-owned Pod includes: In response to receiving the VGPU task request, the target pre-occupied Pod is determined by traversing all the pre-occupied Pods through custom scheduling logic; Update the annotation of the target pre-occupied Pod to add the identifier of the current VGPU task; Create the Pod for the VGPU task and configure node affinity to ensure that the VGPU task Pod and the target pre-occupied Pod are deployed on the same node; The GPU information is passed to the VGPU task through environment variables and device mounting, and the VGPU task is started.

[0013] Furthermore, the screening criteria for the target pre-occupied Pod include: The pre-occupied Pod is in an available or running state; The number of used tasks in the pre-occupied Pod is less than the shared capacity; The GPU type, model, or manufacturer of the pre-occupied Pod meets the requirements of the VGPU task; The location of the node where the pre-occupied Pod is located satisfies the same-node access constraint.

[0014] Furthermore, the method also includes: If, after traversing all the pre-occupied Pods, no target pre-occupied Pod is found that meets the shared capacity, GPU requirements, and same-node access constraints, a resource shortage warning will be returned, or the process will enter a waiting / retry phase.

[0015] This invention also provides a general heterogeneous VGPU scheduling device, the scheduling device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the general heterogeneous VGPU scheduling method described in any of the above embodiments.

[0016] This invention also provides a computer-readable storage medium storing computer instructions that cause a processor to execute the general heterogeneous VGPU scheduling method described in any of the above embodiments.

[0017] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the general heterogeneous VGPU scheduling method described in any of the above embodiments.

[0018] This invention discloses a general heterogeneous VGPU scheduling method and device. The method includes: pre-creating a pre-occupied Pod to occupy physical GPU resources through Kubernetes' resource request mechanism; in response to receiving a VGPU task request, starting a VGPU task using a target pre-occupied Pod, wherein the target pre-occupied Pod is a pre-occupied Pod with a shared quota determined based on pre-occupied Pod annotation information; in response to the completion of the VGPU task, updating the annotation of the target pre-occupied Pod, removing the identifier of the completed VGPU task, and releasing the shared quota occupied by the VGPU task to return the shared quota to the pre-occupied Pod for use by new VGPU tasks. This application pre-occupies GPU resources through a pre-occupied Pod mechanism and then records shared information through a Pod annotation mechanism, enabling multiple VGPU tasks to share the GPU resources occupied by the same pre-occupied Pod. This solves the technical problems of poor versatility, the need for extensive customization and development, high system complexity, inability to dynamically adjust static solutions, poor flexibility, and high cost in existing VGPU solutions. It achieves the technical effects of improving GPU versatility, improving GPU resource utilization, and being easy to implement and maintain. Attached Figure Description

[0019] Figure 1 This is a flowchart of a general heterogeneous VGPU scheduling method provided by an embodiment of the present invention;

[0020] Figure 2 This is an architecture diagram of a general heterogeneous VGPU scheduling system provided by an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of a general heterogeneous VGPU scheduling device provided in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish different objects, not to limit a specific order. The various embodiments of this invention described below can be performed individually or in combination with each other; the embodiments of this invention do not impose specific limitations in this regard.

[0024] Figure 1 This is a flowchart of a general heterogeneous VGPU scheduling method provided by an embodiment of the present invention.

[0025] like Figure 1 As shown, this general heterogeneous VGPU scheduling method specifically includes the following steps:

[0026] S101 pre-creates pre-occupied Pods and uses Kubernetes' resource request mechanism to occupy physical GPU resources.

[0027] Specifically, Kubernetes manages GPU resources through resource names, device plugins, and schedulers. However, Kubernetes' native GPU resource management mechanism adopts an exclusive mode, meaning that a GPU can only be used by one Pod, which cannot realize the sharing function of VGPU. Therefore, it is necessary to create a pre-occupied Pod, that is, to create a pre-occupied Pod on the node in advance that does almost no work. The pre-occupied Pod only needs to "occupy" the GPU resource through Kubernetes' native mechanism to prevent other full-GPU scheduling tasks from taking away this GPU card, thus preparing for the subsequent execution of VGPU tasks.

[0028] Optionally, the creation rules for preempted Pods include: each preempted Pod occupies only one GPU card and uses the Kubernetes resource request mechanism to occupy GPU resources; the preempted Pod runs idle services and does not occupy actual GPU computing resources; the preempted Pod is configured to use different resource names for different GPU vendors to request resources.

[0029] Specifically, the creation of preempted Pods follows these rules: (1) One-to-one allocation. Each preempted Pod occupies only one GPU card, using Kubernetes' resource request mechanism to allocate GPU resources. (2) Idle operation. Preempted Pods occupy GPU resources but do not consume actual GPU computing resources. They run lightweight idle services, consuming only minimal CPU resources. Their main function is to prevent other Pods from using these GPUs. (3) Heterogeneous GPU support. Based on the unified scheduling of heterogeneous GPUs, specific GPU types are specified through node selectors and resource names, supporting GPUs from different manufacturers such as NVIDIA and AMD. Among them, preempted Pods are configured to use different resource names for resource requests from different GPU manufacturers.

[0030] S102, in response to receiving a VGPU task request, a VGPU task is started using a target pre-occupied Pod, wherein the target pre-occupied Pod is a pre-occupied Pod with a shared quota determined based on the pre-occupied Pod annotation information, and the VGPU task is created without declaring a GPU resource request.

[0031] Specifically, when creating a VGPU task, it needs to be marked as using VGPU mode. This can be done through Pod annotations or tags. In essence, a VGPU task does not configure GPU resources in its resource requests, and from the perspective of Kubernetes resource management, it does not use GPU resources. This means that VGPU tasks will not directly participate in Kubernetes' regular competition for exclusive GPU resources, thus avoiding competition and conflicts for GPU resources with the Kubernetes scheduler.

[0032] Optionally, S102 specifically includes:

[0033] In response to receiving a VGPU task request, the system iterates through all pre-empted Pods using custom scheduling logic to determine the target pre-empted Pod; updates the annotation of the target pre-empted Pod, adding the identifier of the current VGPU task; creates the VGPU task Pod, configures node affinity to ensure that the VGPU task Pod and the target pre-empted Pod are deployed on the same node; and passes GPU information to the VGPU task through environment variables and device mounting, and starts the VGPU task.

[0034] Optionally, the selection criteria for target preempted Pods include: the preempted Pod is available or running; the number of tasks used by the preempted Pod is less than the shared capacity; the GPU type, model, or manufacturer of the preempted Pod meets the VGPU task requirements; and the node location of the preempted Pod meets the same-node access constraints.

[0035] Optionally, the method further includes:

[0036] If no target pre-empted Pod that meets the shared capacity, GPU requirements, and same-node access constraints is found after traversing all pre-empted Pods, a resource shortage warning will be returned, or the process will enter a waiting / retry phase.

[0037] Specifically, after receiving a VGPU task request, the VGPU task scheduling is implemented through custom scheduling logic. The custom VGPU scheduler acts as a "pre-allocated Pod allocator": when a new VGPU task needs a GPU, the custom scheduler automatically finds pre-allocated Pods with available slots, determines one of them as the target pre-allocated Pod based on the target pre-allocated Pod filtering criteria, and then assigns the task to the same node, transferring the GPU usage rights to the VGPU task so that multiple tasks can use the same GPU together.

[0038] Specifically, the VGPU scheduling logic includes the following steps:

[0039] Step 1: It is necessary to find available pre-occupied Pods. The scheduler traverses all pre-occupied Pods and finds pre-occupied Pods that meet the following conditions: (1) The pre-occupied Pod is in the running state; (2) The number of VGPU tasks used is less than the shared capacity; (3) The GPU model matches the GPU requirements of the VGPU task. That is, after receiving the VGPU task request, the scheduler traverses all pre-occupied Pods and checks the Pods with the status of Running, the number of tasks used is less than the shared capacity, and the GPU model matches.

[0040] Step 2: After traversing all pre-owned Pods, if no target pre-owned Pod is found that meets the shared capacity, GPU requirements, and same-node access constraints, the scheduler will directly return a resource shortage warning or enter a waiting / retry process.

[0041] Step 3: When resources are sufficient, the scheduler determines the target preempted Pod and obtains the corresponding GPU information from the annotations of the target preempted Pod;

[0042] Step 4: Update the annotation of the target pre-occupied Pod, add the identifier of the current VGPU task, and add the VGPU task to the used list.

[0043] Since the VGPU task Pod needs to access the physical GPU device occupied by the target pre-occupied Pod, forced node affinity is used to ensure that the two are located on the same node to meet the same node device access condition. After adding the VGPU task to the used list, forced node affinity is configured to match the node hostname of the target pre-occupied Pod to ensure that the VGPU task and the target pre-occupied Pod are on the same node and can access the same GPU device. Subsequently, the GPU information of the target pre-occupied Pod is obtained through two methods: environment variables and device mounting. (1) Environment variables: add environment variables in the Pod configuration of the VGPU task; (2) Device mounting: mount the GPU device of the target pre-occupied Pod to the VGPU task through the Kubernetes device mounting mechanism. Device mounting enables the VGPU task to access the physical GPU device and supports GPUs from different vendors through device path mapping.

[0044] It should be noted that GPUs from different manufacturers have different device paths. The device path needs to be determined based on the GPU manufacturer and serial number to achieve heterogeneous GPU device path mapping. Furthermore, VGPU tasks do not configure GPU resources in their resource requests; from the perspective of Kubernetes resource management, no GPU resources are used, thus avoiding conflicts with Kubernetes' native scheduling mechanism.

[0045] S103, in response to the end of a VGPU task, updates the annotation of the target pre-occupied Pod, removes the identifier of the ended VGPU task, and releases the shared quota occupied by the VGPU task, so that the shared quota can be returned to the pre-occupied Pod for use by new VGPU tasks.

[0046] Specifically, when a VGPU task ends, the scheduler needs to update the annotation of the target pre-owned Pod and release shared resources. The specific execution steps are as follows: when the VGPU task Pod is detected to be in a state of ending, deletion, termination, or no longer running, update the shared record in the target pre-owned Pod; remove the corresponding VGPU task identifier and release the corresponding shared quota.

[0047] Clearly, the lifecycle of a preempted Pod includes: pre-creating preempted Pods based on VGPU resource requirements; monitoring the running status of preempted Pods; automatically recreating preempted Pods while maintaining their original annotation configurations when they exit abnormally; and dynamically adjusting the number of preempted Pods according to actual needs.

[0048] In this embodiment of the invention, GPU resources are pre-occupied through a pre-occupied Pod mechanism, and shared information is recorded through a Pod annotation mechanism. This allows multiple VGPU tasks to share the GPU resources occupied by the same pre-occupied Pod, solving the technical problems of poor versatility, the need for extensive customization, high system complexity, inability to dynamically adjust static solutions, poor flexibility, and high cost in existing VGPU solutions. This achieves the technical effects of improving GPU versatility, increasing GPU resource utilization, and being easy to implement and maintain.

[0049] Based on the above technical solutions, S101 specifically includes:

[0050] Plan GPU resource requirements; create pre-occupied Pods based on the planning results, and occupy the corresponding physical GPU resources through the Kubernetes native resource request mechanism; write annotations for the created pre-occupied Pods, where the annotations should include at least the shared capacity of the pre-occupied Pod, the list of tasks already used, GPU identification information, and current shared status information.

[0051] Optionally, planning GPU resource requirements includes: determining the number of GPUs to be pre-allocated based on the total amount of GPU resources in the cluster and the expected VGPU task requirements; identifying and planning GPU resources from different vendors based on a pre-defined heterogeneous GPU unified scheduling framework; and selecting appropriate nodes to deploy pre-allocated Pods to avoid single points of failure.

[0052] Specifically, before creating a pre-empted Pod, the number of GPU cards to be used for VGPU needs to be planned in advance, and pre-empted Pods are created based on the planned structure. GPU resources are then allocated through a resource request mechanism. Subsequently, annotations need to be written for the created pre-empted Pod. The pre-empted Pod records shared information through Kubernetes' annotation mechanism, which is crucial for VGPU scheduling. The annotation content includes at least the shared capacity of the pre-empted Pod, a list of used tasks, GPU identification information, and current shared status information. The GPU identification information includes one or more of the following: GPU serial number, GPU UUID, GPU model, and GPU manufacturer. Specifically, the shared capacity annotation defines how many VGPU tasks a GPU can be shared with; the used task list annotation records currently used VGPU tasks to determine if there is still available capacity; and the GPU information annotation records detailed GPU information for device mounting and heterogeneous GPU matching.

[0053] For example, the specific fields of an annotation can be represented as follows:

[0054] 1. `vgpu.share.capacity` (total shared capacity of the virtual GPU);

[0055] 2. `vgpu.share.used` (Virtual GPU shared capacity is used);

[0056] 3. `gpu.gpu.index` (GPU device index number);

[0057] 4. `gpu.gpu.uuid` (GPU identifier);

[0058] 5. `gpu.gpu.model` (GPU model name);

[0059] 6. `gpu.gpu.vendor` (GPU manufacturer name).

[0060] In summary, by using Pod annotations to record how many VGPU tasks each pre-occupied Pod can share with, as well as the list of currently used VGPU tasks, shared management of GPU resources is achieved. Utilizing Kubernetes' native annotation mechanism, no additional resource management components are required. The recording and querying of shared information are achieved through annotations, and dynamic updates of shared information are supported, making it more flexible.

[0061] Optionally, during the creation, startup, and execution of a VGPU task, it may be identified by at least one of the following methods: by annotating the target preempted Pod; by setting a tag; or by recording the association with the target preempted Pod.

[0062] Specifically, the identification mechanism is one of the fundamental conditions of the scheduling process. Only after identifying that a task belongs to a VGPU task can it be determined whether it needs to go through the pre-emptive and shared scheduling logic. Therefore, VGPU tasks need to be identified through a defined method to achieve their recognition.

[0063] Optionally, based on a preset heterogeneous GPU unified scheduling framework, identifying and planning GPU resources from different vendors includes: based on a preset heterogeneous GPU unified scheduling framework, achieving unified management and scheduling of VGPUs from different vendors and architectures through at least one of the following methods: resource name mapping, device path mapping, and environment variable mapping.

[0064] Specifically, resource name mapping achieves GPU resource scheduling by reusing the resource name definitions in existing heterogeneous VGPU scheduling schemes; device path mapping achieves GPU resource scheduling by defining device mount paths for GPUs from different vendors; and environment variable mapping achieves GPU resource scheduling by defining environment variable names for GPUs from different vendors. The scheduling logic adapts to scheduling rules for different GPU types, specifically including: matching the corresponding pre-allocated Pod based on the GPU requirements of the VGPU task; creating a pre-allocated Pod using the corresponding resource name based on the GPU vendor; and determining the correct device mount path based on the GPU vendor and sequence number. Therefore, in this embodiment of the invention, it does not rely on specific GPU hardware virtualization characteristics, but rather achieves unified shared scheduling of GPU resources from different vendors and models through resource name mapping, device path mapping, environment variable mapping, and a pre-allocated Pod sharing mechanism.

[0065] For example, resource name mapping can be represented as: `nvidia.com / gpu`, `amd.com / gpu`, `ascend.com / npu`; environment variable mapping can be represented as: `NVIDIA_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES`; device path mapping can be represented as: ` / dev / nvidia0`.

[0066] In summary, the general heterogeneous VGPU scheduling method provided by the embodiments of the present invention has the following characteristics:

[0067] (1) Pre-emption mechanism: Idle pre-empted Pods are created in advance and GPU resources are occupied through Kubernetes' resource request mechanism. The pre-empted Pods run lightweight idle services and do not consume actual GPU computing resources, but prevent other Pods from using these GPUs.

[0068] (2) Sharing mechanism: The Pod annotation records how many VGPU tasks each pre-occupied Pod can share with, as well as the list of currently used VGPU tasks. Multiple VGPU tasks can share the GPU occupied by the same pre-occupied Pod.

[0069] (3) Scheduling mechanism: When a VGPU task is created, it searches for available pre-occupied Pods through custom scheduling logic, and then passes GPU information to the VGPU task through environment variables and device mounting. It also ensures that the VGPU task and the pre-occupied Pod are on the same node through node affinity.

[0070] (4) Heterogeneous support: Based on the already implemented unified scheduling of heterogeneous GPUs, the unified management and scheduling of VGPUs of heterogeneous GPUs is realized through resource names, device path mapping and other methods.

[0071] Therefore, the general heterogeneous VGPU scheduling method provided by the embodiments of the present invention has the following advantages:

[0072] (1) High versatility: It supports all GPU manufacturers and models, does not depend on specific GPU hardware characteristics, and manages heterogeneous GPUs in a unified manner through resource name mapping, device path mapping, etc., which reduces hardware cost requirements and development and maintenance costs.

[0073] (2) Simple implementation: Based on the native Kubernetes mechanism (Pod annotations, node affinity, device mounting), no modification to the core Kubernetes components is required, and no additional virtualization layer is needed. There are few main components (VGPU scheduler, pre-empted Pods, VGPU task Pods), the system architecture is clear, the development workload is small, the development cycle is short, and the maintenance cost is low.

[0074] (3) Flexible and efficient: Supports dynamic resource allocation and release, and can flexibly adjust the number of pre-allocated Pods and shared capacity according to actual needs. Multiple VGPU tasks can share the same GPU, improving GPU utilization and reducing hardware costs. The pre-allocation mechanism avoids resource contention during real-time scheduling, and uses annotation mechanism to quickly find available resources, resulting in low scheduling latency.

[0075] (4) Easy to maintain: The system architecture is clear with few components. Shared information recorded through Pod annotations is easy to understand and track. Based on the native Kubernetes mechanism, it has good compatibility with Kubernetes version upgrades and does not require continuous follow-up on version updates. Problems can be quickly located through Pod annotations, and GPU access can be quickly verified through environment variables and device mounting.

[0076] Figure 2 This is an architecture diagram of a general heterogeneous VGPU scheduling system provided by an embodiment of the present invention.

[0077] like Figure 2 As shown, the VGPU scheduler, built on top of a Kubernetes cluster, features custom scheduling logic. The VGPU scheduler includes:

[0078] The pre-allocated Pod management module is used to pre-allocate GPU resources, record shared information through annotations, and run lightweight idle services; the VGPU scheduler is used for scheduling and allocating resources for VGPU tasks. The VGPU task Pod does not configure GPU resource requests, but obtains GPU information through environment variables and device mounting, and is on the same node as the pre-allocated Pod through node affinity; the heterogeneous GPU adaptation module is used for adapting to and managing heterogeneous GPUs; the VGPU resource release module is used to remove the identifier of completed VGPU tasks and release the shared quota occupied by VGPU tasks.

[0079] Depend on Figure 2 As can be seen, a GPU can only be used by one preempted Pod, and one or more preempted Pods can be deployed on a node according to the GPU resource situation; VGPU tasks and their target preempted Pods should be deployed on the same node; multiple VGPU tasks can share a preempted Pod.

[0080] The heterogeneous VGPU scheduling system provided in this embodiment of the invention executes the general heterogeneous VGPU scheduling method provided in the above embodiments. Therefore, the general heterogeneous VGPU scheduling system provided in this embodiment of the invention also has the beneficial effects described in the above embodiments, which will not be repeated here.

[0081] Figure 3 This is a schematic diagram of a general heterogeneous VGPU scheduling device provided in an embodiment of the present invention. This heterogeneous VGPU scheduling device is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0082] like Figure 3 As shown, the heterogeneous VGPU scheduling device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 and a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the heterogeneous VGPU scheduling device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0083] Multiple components in the heterogeneous VGPU scheduling device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network interface card, modem, wireless transceiver, etc. The communication unit 19 allows the heterogeneous VGPU scheduling device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0084] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a general heterogeneous VGPU scheduling method.

[0085] In some embodiments, the general heterogeneous VGPU scheduling method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed onto the heterogeneous VGPU scheduling device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the general heterogeneous VGPU scheduling method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the general heterogeneous VGPU scheduling method by any other suitable means (e.g., by means of firmware).

[0086] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0087] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0088] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0089] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous VGPU scheduling device, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0090] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0091] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0092] This invention also provides a computer program product, which includes computer-executable instructions that, when executed by a computer processor, are used to perform the general heterogeneous VGPU scheduling method provided in any embodiment of this invention.

[0093] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] Of course, the computer program product provided in the embodiments of this application has computer-executable instructions that are not limited to the method operations described above, and can also perform related operations in the methods provided in any embodiment of this application.

[0095] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is made herein.

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A general heterogeneous VGPU scheduling method, characterized in that, The method includes: Pre-create pre-occupied Pods and use Kubernetes' resource request mechanism to occupy physical GPU resources; In response to receiving a VGPU task request, a VGPU task is started using a target pre-occupied Pod, wherein the target pre-occupied Pod is a pre-occupied Pod with a shared quota determined based on the pre-occupied Pod annotation information, and the VGPU task is created without declaring a GPU resource request; In response to the completion of the VGPU task, the annotation of the target pre-occupied Pod is updated, the identifier of the completed VGPU task is removed, and the shared quota occupied by the VGPU task is released to return the shared quota to the pre-occupied Pod for use by new VGPU tasks. The rules for creating the pre-emptive Pod include: Each pre-occupied Pod occupies only one GPU card, using Kubernetes' resource request mechanism to allocate GPU resources; The pre-occupied Pod runs an idle service without occupying actual GPU computing resources; The pre-occupied Pod is configured to use different resource names for different GPU vendors when making resource requests.

2. The universal heterogeneous VGPU scheduling method of claim 1, wherein, Pre-creating pre-occupied Pods and using Kubernetes' native resource request mechanism to occupy physical GPU resources includes: Plan GPU resource requirements; Based on the planning results, pre-occupied Pods are created and the corresponding physical GPU resources are occupied through the Kubernetes native resource request mechanism. Write annotations for the created pre-occupied Pod, wherein the annotations include at least the shared capacity of the pre-occupied Pod, the list of used tasks, GPU identification information, and current shared status information.

3. The universal heterogeneous VGPU scheduling method of claim 2, wherein, The VGPU task is identified during its creation, startup, and execution by at least one of the following: Identified by the annotations of the target pre-occupied Pod; Identify by setting labels; Identification is achieved by recording the association with the target pre-occupied Pod.

4. The universal heterogeneous VGPU scheduling method of claim 2, wherein, The planned GPU resource requirements include: Determine the number of GPUs that need to be reserved based on the total amount of GPU resources in the cluster and the expected VGPU task requirements. Based on a pre-defined heterogeneous GPU unified scheduling framework, it identifies and plans GPU resources from different manufacturers. Select a suitable node to deploy the pre-occupied Pod to avoid a single point of failure.

5. The universal heterogeneous VGPU scheduling method of claim 4, wherein, Based on a pre-defined heterogeneous GPU unified scheduling framework, it identifies and plans GPU resources from different vendors, including: Based on a pre-defined heterogeneous GPU unified scheduling framework, unified management and scheduling of VGPUs from different manufacturers and architectures can be achieved through at least one of the following methods: resource name mapping, device path mapping, and environment variable mapping.

6. The general heterogeneous VGPU scheduling method according to claim 1, characterized in that, In response to receiving a VGPU task request, launching a VGPU task using the target pre-empted Pod includes: In response to receiving the VGPU task request, the target pre-occupied Pod is determined by traversing all the pre-occupied Pods through custom scheduling logic; Update the annotation of the target pre-occupied Pod to add the identifier of the current VGPU task; Create the Pod for the VGPU task and configure node affinity to ensure that the VGPU task Pod and the target pre-occupied Pod are deployed on the same node; The GPU information is passed to the VGPU task through environment variables and device mounting, and the VGPU task is started.

7. The general heterogeneous VGPU scheduling method according to claim 6, characterized in that, The selection criteria for the target pre-occupied Pod include: The pre-occupied Pod is in an available or running state; The number of used tasks in the pre-occupied Pod is less than the shared capacity; The GPU type, model, or manufacturer of the pre-occupied Pod meets the requirements of the VGPU task; The location of the node where the pre-occupied Pod is located satisfies the same-node access constraint.

8. The general heterogeneous VGPU scheduling method according to claim 6, characterized in that, The method further includes: If, after traversing all the pre-occupied Pods, no target pre-occupied Pod is found that meets the shared capacity, GPU requirements, and same-node access constraints, a resource shortage warning will be returned, or the process will enter a waiting / retry phase.

9. A general-purpose heterogeneous VGPU scheduling device, characterized in that, The scheduling device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the general heterogeneous VGPU scheduling method according to any one of claims 1-8.

Images