A method and apparatus for scheduling GPU computing resources

Abstract

A method for scheduling GPU computing resources includes: in response to a target computing task created in a computing cluster, determining the task type of the target computing task; if the target computing task is a first type of computing task, scheduling the target computing task to a first GPU hardware in the computing cluster whose remaining computing resources meet the computing power requirements of the target computing task for execution; in response to a first indication reported by a first computing node equipped with the first GPU hardware that the first type of computing task will exclusively occupy the computing power resources of the first GPU hardware, rescheduling a second type of computing task that has been scheduled to run on the first GPU hardware to a second GPU hardware in the computing cluster whose remaining computing resources meet the computing power requirements of the second type of computing task for execution; wherein the service level of the first type of computing task is higher than that of the second type of computing task.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a method and apparatus for scheduling GPU computing resources. Background Technology

[0002] With the rapid development of the AI ​​field, the demand for computing power related to AI model training and inference has grown far faster than the development of hardware. As a core computing hardware widely used in AI scenarios, GPUs have naturally become relatively scarce. Against the backdrop of a global shortage of GPU computing resources, GPUs have become a valuable computing resource for companies, institutions, and schools. However, current usage of GPU hardware still results in significant waste of computing resources. Summary of the Invention

[0003] This specification proposes a method for scheduling GPU computing resources, applied to a scheduler corresponding to a computing cluster; wherein at least some computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; the service level of the first type of computing task is higher than that of the second type of computing task; the method includes:

[0004] In response to a target computing task created in the computing cluster, the task type of the target computing task is determined;

[0005] If the target computing task is the first type of computing task, the target computing task is scheduled to run on the first GPU hardware in the computing cluster; wherein, the first GPU hardware is GPU hardware whose remaining computing power resources meet the computing power requirements of the target computing task;

[0006] In response to a first instruction reported by the first computing node, the second type of computing tasks that have been scheduled to run on the first GPU hardware are rescheduled to run on the second GPU hardware in the computing cluster; wherein, the first computing node is a computing node equipped with the first GPU hardware; the first instruction is used to indicate that the first type of computing tasks will exclusively occupy the computing resources of the first GPU hardware; the first instruction is reported by the first computing node to the scheduler when the computing resources of the first GPU hardware occupied by the first type of computing tasks reach a preset threshold; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing tasks.

[0007] Optionally, the method further includes:

[0008] If the target computing task is the second type of computing task, the target computing task is scheduled to run on the third GPU hardware in the computing cluster; wherein, the third GPU hardware is GPU hardware whose computing resources are not exclusively occupied by the first type of computing task, and whose remaining computing resources meet the computing power requirements of the target computing task.

[0009] Optionally, the scheduler maintains hardware modes corresponding to each GPU in the computing cluster. These hardware modes include a resource-sharing mode and a resource-exclusive mode; the resource-sharing mode indicates that the GPU's computing resources support running multiple types of computing tasks; the resource-exclusive mode indicates that the GPU's computing resources are used only to execute the first type of computing task.

[0010] Optionally, the method further includes:

[0011] In response to the first instruction reported by the first computing node equipped with the first GPU hardware, the first GPU hardware is switched from resource sharing mode to resource exclusive mode.

[0012] If the target computing task is the second type of computing task, scheduling the target computing task to the third GPU hardware in the computing cluster for execution includes:

[0013] If the target computing task is the second type of computing task, the target computing task is scheduled to run on the third GPU hardware in the computing cluster; wherein, the third GPU hardware is GPU hardware that is in the resource sharing mode and whose remaining computing power resources meet the computing power requirements of the target computing task.

[0014] Optionally, the method further includes:

[0015] In response to a second instruction from the first computing node, the first GPU hardware is switched from a resource-exclusive mode to a resource-sharing mode; wherein, the second instruction is reported by the first computing node to the scheduler when the computing power resources of the first GPU hardware occupied by the first type of computing task are lower than a preset threshold.

[0016] Optionally, the computing node supports virtualizing the computing resources of its onboard GPU hardware into a virtual GPU; the virtual GPU includes a first type of virtual GPU for performing the first type of computing tasks and a second type of virtual GPU for performing the second type of computing tasks.

[0017] Optionally, the scheduler maintains a first remaining computing power and a second remaining computing power corresponding to each GPU hardware in the computing cluster; the first remaining computing power represents the number of first-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware; the second remaining computing power represents the number of second-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware.

[0018] If the target computing task is the first type of computing task, scheduling the target computing task to the first GPU hardware in the computing cluster for execution includes:

[0019] If the target computing task is the first type of computing task, based on the maintained first computing power remaining amount corresponding to each GPU hardware, a first GPU hardware in the computing cluster whose first computing power remaining amount meets the first requirement of the target computing task for the first type of virtual GPU is determined, and the target computing task is scheduled to run on the first GPU hardware in the computing cluster.

[0020] If the target computing task is the second type of computing task, scheduling the target computing task to the third GPU hardware in the computing cluster for execution includes:

[0021] If the target computing task is the second type of computing task, based on the maintained second computing power remaining amount corresponding to each GPU hardware, a third GPU hardware that is not exclusively occupied by the first type of computing task and whose second computing power remaining amount meets the second requirement of the target computing task for the second type of virtual GPU is determined from the computing cluster, and the target computing task is scheduled to run on the third GPU hardware in the computing cluster.

[0022] Optionally, scheduling the target computing task to the first GPU hardware in the computing cluster for execution includes:

[0023] The first requirement of the target computing task for the first type of virtual GPU and the hardware identifier of the first GPU hardware are sent to the first computing node equipped with the first GPU hardware, so that the first computing node virtualizes the first GPU hardware to obtain a plurality of first type virtual GPUs corresponding to the first requirement, and runs the target computing task based on the plurality of first type virtual GPUs.

[0024] Scheduling the target computing task to run on a third GPU in the computing cluster includes:

[0025] The second requirement of the target computing task for the second type of virtual GPU and the hardware identifier of the third GPU hardware are sent to the second computing node equipped with the third GPU, so that the second computing node virtualizes the third GPU hardware to obtain a plurality of second type virtual GPUs corresponding to the second requirement, and runs the target computing task based on the plurality of second type virtual GPUs.

[0026] Optionally, the scheduler maintains a global topology corresponding to each computing node in the computing cluster; wherein the global topology is a topology composed of topology information reported by each computing node in the computing cluster.

[0027] Sending the first requirement of the target computing task for the first type of virtual GPU, and the hardware identifier of the first GPU hardware, to the first computing node equipped with the first GPU hardware includes:

[0028] The global topology is queried to determine the first computing node equipped with the first GPU hardware, and the first requirement of the target computing task for the first type of virtual GPU, as well as the hardware identifier of the first GPU hardware, are sent to the first computing node.

[0029] Sending the second requirement of the target computing task for the second type of virtual GPU, and the hardware identifier of the third GPU hardware, to the second computing node equipped with the third GPU includes:

[0030] The global topology is queried to determine the second computing node equipped with the third GPU hardware, and the second requirement of the target computing task for the second type of virtual GPU, as well as the hardware identifier of the third GPU hardware, are sent to the second computing node.

[0031] Optionally, the method further includes:

[0032] The system obtains the initial values ​​of the first and second remaining computing power reported by each computing node when joining the computing cluster, and maintains these initial values ​​locally. In response to scheduling a first-type or second-type computing task created in the computing cluster to any GPU hardware in the computing cluster, the system updates the maintained initial values ​​of the first or second remaining computing power of the GPU hardware based on the number of first-type or second-type virtual GPUs occupied by the first-type or second-type computing task. Alternatively,

[0033] The first and second remaining computing power are obtained from the real-time reports of each computing node in the computing cluster, and the obtained first and second remaining computing power are maintained locally. The first and second remaining computing power reported by each computing node in real time are obtained by updating the initial values ​​of the first and second remaining computing power based on the number of first or second type virtual GPUs occupied by the first or second type of computing tasks scheduled to run on the GPU hardware on the computing node.

[0034] Optionally, the second GPU hardware includes GPU hardware on the first computing node whose remaining computing resources, other than the first GPU hardware, meet the computing power requirements of the second type of computing task; or, the second GPU hardware includes GPU hardware on other computing nodes in the computing cluster whose remaining computing resources, other than the first computing node, meet the computing power requirements of the second type of computing task.

[0035] The second type of computing tasks that have already been scheduled to run on the first GPU hardware are rescheduled to run on the second GPU hardware, including:

[0036] Determine whether there are any second GPUs among the other GPUs on the first computing node besides the first GPU hardware that have remaining computing power resources to meet the computing power requirements of the second type of computing tasks that have been scheduled to run on the first GPU hardware.

[0037] If the second GPU hardware exists among the other GPU hardware, the second type of computing task is further scheduled to be run on the second GPU hardware.

[0038] If the second GPU hardware is not present in the other GPU hardware, determine whether there is a second GPU hardware with remaining computing power resources that can meet the computing power requirements of the second type of computing task among the GPU hardware on the other computing nodes in the computing cluster besides the first computing node; if so, further schedule the second type of computing task to the second GPU hardware for operation.

[0039] Optionally, the computing cluster is a Kubernetes cluster; the computing nodes support the mixed deployment of multiple containers for running different types of computing tasks on the same GPU hardware; the target computing task is a computing task running in containers deployed on various GPU hardware in the Kubernetes cluster.

[0040] Optionally, the first type of computing task is a computing task run by a container with a QoS service level of Guaranteed; the second type of computing task is a computing task run by a container with a QoS service level of BestEffort.

[0041] This specification also proposes a method for scheduling GPU computing resources, applied to any computing node in a computing cluster; wherein at least some computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; the service level of the first type of computing task is higher than that of the second type of computing task; the method includes:

[0042] In response to the scheduler corresponding to the computing cluster scheduling the first type of computing task to the first GPU hardware mounted on the computing node, the first type of computing task is run on the first GPU hardware; wherein, the first type of computing task is scheduled to the computing node by the scheduler when it is determined that the remaining computing power resources of the first GPU hardware meet the computing power requirements of the first type of computing task;

[0043] Determine whether the computing resources of the first GPU hardware occupied by the first type of computing task have reached a preset threshold;

[0044] If the computing resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold, a first indication is reported to the scheduler, wherein the first indication is used to indicate that the first type of computing task will exclusively occupy the computing resources of the first GPU hardware, so that the scheduler will reschedule the second type of computing task that has been scheduled to run on the first GPU hardware to run on the second GPU hardware in the computing cluster; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing task.

[0045] Optionally, determining whether the computing resources of the first GPU hardware occupied by the first type of computing task have reached a preset threshold includes:

[0046] Determine whether the values ​​of computing power indicators related to the computing power resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold; wherein, the computing power indicators include any or more of the following indicators: response time of requests related to the first type of computing task, utilization rate of the first GPU hardware, and utilization rate of the video memory of the first GPU hardware.

[0047] If the value of the computing power indicator reaches the preset threshold, it is determined that the computing power resources of the first GPU hardware occupied by the first type of computing task have reached the preset threshold.

[0048] Optionally, the computing node supports virtualizing the computing resources of its onboard GPU hardware into a virtual GPU; the virtual GPU includes a first type of virtual GPU for performing the first type of computing tasks and a second type of virtual GPU for performing the second type of computing tasks;

[0049] The method further includes:

[0050] In response to this computing node joining the computing cluster, the initial values ​​of the first and second remaining computing power corresponding to the GPU hardware on this computing node are reported to the scheduler for maintenance locally by the scheduler; wherein, the first remaining computing power represents the number of first-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware; the second remaining computing power represents the number of second-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware; or...

[0051] The system obtains the number of first-type or second-type virtual GPUs occupied by the first-type or second-type computing tasks scheduled by the scheduler to run on the GPU hardware mounted on this computing node. Based on the obtained number of first-type or second-type virtual GPUs, the system further updates the initial values ​​of the first and second remaining computing power corresponding to the GPU hardware mounted on this computing node, and reports the updated first and second remaining computing power to the scheduler in real time so that the scheduler can maintain them locally.

[0052] Optionally, the second GPU hardware includes GPU hardware on the first computing node whose remaining computing resources, excluding the first GPU hardware, meet the computing power requirements of the second type of computing task; or, the second GPU hardware includes GPU hardware on other computing nodes in the computing cluster whose remaining computing resources, excluding the first computing node, meet the computing power requirements of the second type of computing task.

[0053] Optionally, the computing cluster is a Kubernetes cluster; the computing nodes support the mixed deployment of multiple containers for running different types of computing tasks on the same GPU hardware; the target computing task is the computing task running in the container.

[0054] Optionally, the first type of computing task is a computing task run by a container with a QoS service level of Guaranteed; the second type of computing task is a computing task run by a container with a QoS service level of BestEffort.

[0055] This specification also proposes a GPU computing resource scheduling device, applied to a scheduler corresponding to a computing cluster; wherein at least some computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; the service level of the first type of computing task is higher than that of the second type of computing task; the device includes:

[0056] The first determining module, in response to a target computing task created in the computing cluster, determines the task type of the target computing task;

[0057] The scheduling module, if the target computing task is a first type of computing task, schedules the target computing task to the first GPU hardware in the computing cluster for execution; wherein, the first GPU hardware is GPU hardware with remaining computing power resources sufficient to meet the computing power requirements of the target computing task; in response to a first indication reported by a first computing node, reschedules the second type of computing task already scheduled to run on the first GPU hardware to the second GPU hardware in the computing cluster for execution; wherein, the first computing node is a computing node equipped with the first GPU hardware; the first indication is used to indicate that the first type of computing task will exclusively occupy the computing power resources of the first GPU hardware; the first indication is reported by the first computing node to the scheduler when the computing power resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold; the second GPU hardware is GPU hardware with remaining computing power resources sufficient to meet the computing power requirements of the second type of computing task.

[0058] This specification also proposes a GPU computing resource scheduling device, applied to any computing node in a computing cluster; wherein at least some computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; the service level of the first type of computing task is higher than that of the second type of computing task; the device includes:

[0059] The running module, in response to the scheduler corresponding to the computing cluster, schedules the first type of computing tasks to the first GPU hardware mounted on the computing node, and runs the first type of computing tasks on the first GPU hardware; wherein, the first type of computing tasks are scheduled to the computing node by the scheduler when it is determined that the remaining computing power resources of the first GPU hardware meet the computing power requirements of the first type of computing tasks;

[0060] The second determining module determines whether the computing resources of the first GPU hardware occupied by the first type of computing task have reached a preset threshold.

[0061] The reporting module, if the computing resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold, reports a first indication to the scheduler, wherein the first indication is used to indicate that the first type of computing task will exclusively occupy the computing resources of the first GPU hardware, so that the scheduler will reschedule the second type of computing task that has been scheduled to run on the first GPU hardware to run on the second GPU hardware in the computing cluster; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing task.

[0062] In the above technical solutions, on the one hand, by supporting the running of multiple types of computing tasks on the same GPU hardware on the computing nodes in the computing cluster, the scheduler of the computing cluster can schedule multiple types of computing tasks created in the computing cluster to run on the same GPU hardware, thereby improving the utilization rate of GPU computing resources.

[0063] On the other hand, by determining that when a first-type computing task with a higher service level, which is scheduled to a certain GPU hardware, consumes a preset threshold of computing resources on that GPU hardware, a second-type computing task with a lower service level, which has already been scheduled to that GPU hardware, is rescheduled to other GPU hardware in the computing cluster for execution. This allows for elastic scheduling of the second-type computing task, while ensuring that the first-type computing task can exclusively occupy the computing resources of the GPU hardware when its consumption reaches a certain level. It also allows the second-type computing task to be scheduled to run normally on other GPU hardware. This enables the first and second-type computing tasks to share GPU computing resources in a more reasonable way, reducing interference between different types of tasks during the sharing of GPU computing resources. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating a method for scheduling GPU computing resources in one embodiment of this specification;

[0066] Figure 2 This is a flowchart illustrating another method for scheduling GPU computing resources in one embodiment of this specification;

[0067] Figure 3 This is a schematic diagram illustrating the scheduling of computing resources of GPU hardware on compute nodes in a Kubernetes cluster, as shown in one embodiment of this specification.

[0068] Figure 4 This is a schematic diagram illustrating an embodiment of the present specification of elastic scheduling of low-service-level computing tasks that have been scheduled to GPU hardware on compute nodes in a Kubernetes cluster.

[0069] Figure 5 This is a schematic structural diagram of an electronic device shown in one embodiment of this specification;

[0070] Figure 6 This is a block diagram of a GPU computing resource scheduling device shown in one embodiment of this specification;

[0071] Figure 7 This is a block diagram of another GPU computing resource scheduling device shown in one embodiment of this specification. Detailed Implementation

[0072] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0073] Given the relative scarcity of GPU computing power, GPUs have become a precious resource. However, current usage of GPU hardware still results in significant waste of these resources.

[0074] For example, taking the inference task of executing AI models through GPU as an example, in this type of inference scenario, the use of GPU will have obvious tidal phenomenon. That is, when there is an inference task to be executed, the utilization rate of GPU is high, and when there is no inference task to be executed, the utilization rate of GPU is low, and a large amount of computing resources will be idle, resulting in a large waste of computing resources.

[0075] To fully utilize the computing power of GPUs, virtualization technology can be used in practical applications to virtualize the computing power of physical GPU hardware into multiple virtual GPUs. The computing power of these multiple virtual GPUs can then be allocated to different types of computing tasks, allowing different types of computing tasks to share the computing power of the same GPU hardware and run on the same GPU hardware.

[0076] However, in practical applications, different types of computing tasks running on the same GPU hardware may have different service levels. Simply relying on virtualizing the GPU hardware may not guarantee the service level of these computing tasks.

[0077] For example, assuming that the above-mentioned various types of computing tasks include a first type of computing task and a second type of computing task with a service level lower than that of the first type of task, in related technologies, when the first type of computing task and the second type of computing task are run on the same GPU hardware, the computing resources are usually scheduled for the first type of computing task and the second type of computing task from the perspective of sharing the computing resources of a single physical GPU.

[0078] In one scenario, after a first type of computing task is scheduled to a GPU on a computing node, if the first type of computing task requires a significant amount of computing resources from the virtual GPUs obtained by virtualizing the GPU hardware, and a second type of computing task is already running on the GPU hardware, the second type of computing task can usually be paused. When the computing resources of the virtual GPUs occupied by the first type of computing task decrease, virtual GPUs can be reallocated to the second type of computing task from the idle virtual GPUs.

[0079] However, in this situation, the second type of computing task may remain in a state of starvation and unable to obtain computing resources from the GPU hardware because the first type of computing task continuously consumes a large amount of virtual GPU computing resources.

[0080] In another scenario, virtual GPUs can be statically partitioned from the virtual GPUs obtained by virtualizing the GPU hardware for the first type of computing task and the second type of computing task, respectively; for example, a portion of virtual GPUs can be reserved from these virtual GPUs for the first type of computing task and the second type of computing task, respectively.

[0081] However, in this case, since the partitioning is static, the first type of computing task and the second type of computing task cannot share the virtual GPUs allocated to them. If there are idle virtual GPUs in the virtual GPUs reserved for the first type of computing task, they obviously cannot be reallocated to the second type of computing task, resulting in low utilization of GPU hardware computing resources.

[0082] Furthermore, since a portion of the virtual GPU needs to be reserved for the second type of computing task, if the first type of computing task with a high service level is itself a computing task that requires providing users with a user experience that exclusively uses the computing power resources of the GPU hardware, then the service level of the second type of computing task obviously cannot be guaranteed.

[0083] It is evident that scheduling computing resources for the first type of computing task and the second type of computing task from the perspective of sharing computing resources of a single physical GPU will result in an unreasonable allocation of GPU hardware computing resources to the first type of computing task and the second type of computing task, thus failing to guarantee the service level of these computing tasks.

[0084] Based on this, this specification proposes a technical solution for elastic scheduling of low-service-level computing tasks among various types of computing tasks in application scenarios where the same GPU hardware supports running multiple types of computing tasks. This solution is based on the perspective of sharing computing resources among the various physical GPUs in a computing cluster.

[0085] In this technical solution, the scheduler corresponding to the computing cluster can determine the task type of the target computing task when a target computing task is created in the computing cluster. If the target computing task is a first-class computing task with a high service level, the target computing task can be scheduled to the first GPU hardware in the computing cluster with the remaining computing power resources that meet the computing power requirements of the target computing task for operation.

[0086] The computing node equipped with the first GPU hardware can determine whether the computing resources of the first type of computing task running on the first GPU hardware have reached a preset threshold during the process of the target computing task running on the first GPU hardware. If the computing resources of the first type of computing task running on the first GPU hardware have reached the preset threshold, it can report a first indication to the scheduler that the first type of computing task will exclusively occupy the computing resources of the first GPU hardware.

[0087] After receiving the first instruction, the scheduler can reschedule the second type of computing tasks that have been scheduled to run on the first GPU hardware to the second GPU hardware in the computing cluster whose remaining computing power resources meet the computing power requirements of the second type of computing tasks.

[0088] In the above technical solutions, on the one hand, by supporting the running of multiple types of computing tasks on the same GPU hardware on the computing nodes in the computing cluster, the scheduler of the computing cluster can schedule multiple types of computing tasks created in the computing cluster to run on the same GPU hardware, thereby improving the utilization rate of GPU computing resources.

[0089] On the other hand, by determining that when a first-type computing task with a higher service level, which is scheduled to a certain GPU hardware, consumes a preset threshold of computing resources on that GPU hardware, a second-type computing task with a lower service level, which has already been scheduled to that GPU hardware, is rescheduled to other GPU hardware in the computing cluster for execution. This allows for elastic scheduling of the second-type computing task, while ensuring that the first-type computing task can exclusively occupy the computing resources of the GPU hardware when its consumption reaches a certain level. It also allows the second-type computing task to be scheduled to run normally on other GPU hardware. This enables the first and second-type computing tasks to share GPU computing resources in a more reasonable way, reducing interference between different types of tasks during the sharing of GPU computing resources.

[0090] Please see Figure 1 , Figure 1This document presents a flowchart of a GPU computing resource scheduling method, which is applied to a scheduler corresponding to a computing cluster. At least some computing nodes in the computing cluster are equipped with at least one GPU hardware. Each computing node supports running multiple types of computing tasks on the same GPU hardware. The multiple types of computing tasks include a first type of computing task and a second type of computing task. The service level of the first type of computing task is higher than that of the second type of computing task. The method includes the following execution process:

[0091] Step 102: In response to the target computing task created in the computing cluster, determine the task type of the target computing task;

[0092] The aforementioned computing cluster may include several computing nodes, at least some of which may be equipped with at least one GPU hardware and may run computing tasks through the GPU hardware.

[0093] The aforementioned computational tasks may include tasks that support the use of GPU hardware computing resources; for example, they may be training or inference tasks related to LLM (Large Language Model) models.

[0094] In practical applications, a scheduler can also be deployed for the aforementioned computing cluster. Specifically, the scheduler can manage the computing resources of the GPU hardware on each computing node in the computing cluster, and reasonably schedule the created computing tasks in the computing cluster based on the remaining computing resources of the GPU hardware on each computing node in the computing cluster.

[0095] For example, in one scenario, the aforementioned computing cluster could be a Kubernetes cluster, where the compute nodes can be physical machines or virtual machines. As a distributed cluster built on container technology, in a Kubernetes cluster, computing tasks can typically be run as containerized applications on various compute nodes. In other words, the computing task can be understood as a computing task running in containers deployed on the compute nodes of the Kubernetes cluster.

[0096] In a Kubernetes cluster, the smallest unit for managing containers is called a Pod. A Pod contains one or more containers and typically represents a process running within the Kubernetes cluster. When deploying containerized applications (i.e., computing tasks) on a Kubernetes cluster, a Pod is usually created for the containerized application, and the application's executable code is packaged into a container image within that Pod. Then, the Kubernetes cluster scheduler can schedule the Pod to a specific GPU on a compute node in the cluster, based on the available computing resources of the GPUs on each compute node, and deploy the Pod to that GPU, thus utilizing the GPU's computing resources to run the containerized application within the Pod.

[0097] It should be noted that, in practical applications, the scheduler mentioned above can be an independent service device that interfaces with the computing cluster, or it can be a computing node selected from the computing cluster as the scheduler. No special limitation is made in this specification.

[0098] To fully utilize the computing power of the GPUs on the compute nodes, compute nodes in a compute cluster equipped with GPUs can also support running multiple types of computing tasks on the same GPU. These multiple types of computing tasks can specifically include a first type of computing task and a second type of computing task.

[0099] For example, taking the computing cluster as a Kubernetes cluster again, in order to make full use of the computing resources of the GPU hardware, the computing nodes in the Kubernetes cluster can support the mixed deployment of multiple containers for running different types of computing tasks on the same GPU hardware. In this case, the aforementioned computing tasks can be computing tasks that run in containers deployed on the GPU hardware of the computing nodes in the Kubernetes cluster.

[0100] The first type of computing task can be a computing task with a higher service level; for example, this first type of computing task could be a computing task that requires providing users with a user experience that utilizes dedicated GPU hardware computing resources. The second type of computing task can be a computing task with a lower service level than the first type of computing task.

[0101] It should be noted that, in practical applications, the aforementioned service level can specifically refer to the service level promised to the user by the service provider based on the SLA (service-level agreement).

[0102] For example, taking the computing cluster as a Kubernetes cluster as an example, as a distributed cluster built on container technology, the computing nodes in the Kubernetes cluster can usually support the mixed deployment of multiple containers with different QoS (Quality of Service) priorities on the same GPU hardware. In this scenario, the service level can be set for the containerized applications (i.e. computing tasks) created in the cluster based on the QoS mechanism for containers.

[0103] Among them, the QoS mechanism for containers in the Kubernetes cluster is a resource management strategy for Pods (i.e., containers) in the Kubernetes cluster. It can reasonably allocate resources to Pods based on their resource requests and limitations, thereby enabling reasonable control over the resources used by Pods in the cluster.

[0104] Based on the QoS mechanism of the Kubernetes cluster, Pods can be divided into three QoS levels according to their resource requests and restrictions: Guaranteed, Burstable, and BestEffort.

[0105] For Pods with a QoS level of Guaranteed, their declared resource requests and resource limits are equal. This means that the Pod will receive all the resources it requests and will not use more than that amount. Guaranteed Pods have the highest priority in resource allocation. The cluster can provide deterministic resource guarantees for Pods with a QoS level of Guaranteed, making them suitable for running applications with strict resource requirements.

[0106] For Pods with a QoS level of Burstable, these Pods request less resources than their set limits. This means that when resources are plentiful, the Pod can use more resources than it requests, but when resources are scarce, its resource usage will be limited to the requested amount. For Pods with a QoS level of Burstable, the cluster provides resources with elasticity and flexibility, typically allowing these Pods to borrow additional resources as available in the cluster. This is suitable for applications that do not need to run at full load most of the time, but may require additional resources during peak periods.

[0107] For Pods with a QoS level of BestEffort, this category of Pods has no explicit resource requests or restrictions. This means that these Pods will use idle resources on the node that are not being used by other Pods. These Pods have the lowest priority in resource allocation and are the first to be sacrificed when resources are scarce to ensure the resource needs of Pods with QoS levels of Guaranteed and Burstable are met. These Pods are generally suitable for running applications that are not sensitive to resource requirements.

[0108] It's easy to understand that Pods with a QoS level of Guaranteed typically have the highest service level, while Pods with a QoS level of BestEffort typically have the lowest service level. In this case, the first type of computing task mentioned above can be a computing task run by a Pod with a QoS level of Guaranteed. Correspondingly, the second type of computing task mentioned above can be a computing task run by a Pod with a QoS level of BestEffort. It should be noted that since the technical solution in this specification aims to achieve elastic scheduling of the second type of computing task, and the cluster itself has elasticity and flexibility in providing resources to Pods with a QoS level of Burstable, this specification allows Pods with a QoS level of BestEffort to be considered as the second type of computing task, rather than the computing tasks run by Pods with a QoS level of Burstable.

[0109] Of course, when the above computing cluster is a Kubernetes cluster, in addition to setting the service level for containerized applications created in the cluster based on the QoS mechanism for containers, other methods can also be used to set the service level for containerized applications created in the cluster, which will not be listed one by one in this specification.

[0110] In this specification, when a user creates a target computing task in a computing cluster based on specific computing needs, the scheduler of the computing cluster can respond to the target computing task and determine the task type of the target computing task.

[0111] Specifically, the scheduler can determine whether a target computing task is a Class I computing task with a higher service level or a Class II computing task with a lower service level, based on the service level set by the user for that target computing task.

[0112] For example, taking the aforementioned computing cluster as a Kubernetes cluster, the first type of computing task is a computing task run by a Pod with a QoS service level of Guaranteed, and the second type of computing task is a computing task run by a Pod with a QoS service level of BestEffort. When a user creates a new Pod (i.e., the aforementioned target computing task) in the computing cluster based on specific computing needs, the scheduler can determine the QoS service level of the Pod when determining the task type of the computing task run by the Pod. If the QoS service level of the Pod is Guaranteed, it means that the computing task run by the Pod is the first type of computing task mentioned above; if the QoS service level of the Pod is BestEffort, it means that the computing task run by the Pod is the second type of computing task mentioned above.

[0113] Step 104: If the target computing task is the first type of computing task, schedule the target computing task to the first GPU hardware in the computing cluster whose remaining computing power resources meet the computing power requirements of the target computing task for execution;

[0114] In this specification, in order to enable the first type of computing tasks and the second type of computing tasks to share the computing resources of the same GPU hardware more reasonably, the scheduler of the computing cluster, when scheduling the first type of computing tasks and the second type of computing tasks, may no longer schedule the computing resources from the perspective of sharing the computing resources of a single physical GPU hardware, but may instead schedule the computing resources from the perspective of sharing the computing resources of each physical GPU hardware in the computing cluster.

[0115] When scheduling computing resources from the perspective of sharing computing resources among the physical GPUs in the computing cluster, the scheduler can maintain the remaining computing resources of each GPU in the computing cluster in advance, so as to grasp the remaining computing resources of all GPUs in the computing cluster from a global perspective.

[0116] Once the scheduler determines the task type of the target computing task, if the target computing task is the first type of computing task, the scheduler can refer to the remaining computing resources of each GPU hardware and determine the first GPU hardware with remaining computing resources that can meet the computing power requirements of the target computing task from the physical GPU hardware in the computing cluster.

[0117] For example, in practical applications, the scheduler can refer to the remaining computing resources of each GPU hardware it maintains, first determine the computing nodes in the computing cluster whose remaining computing resources meet the computing power requirements of the target computing task, and then further determine the first GPU hardware whose remaining computing resources meet the computing power requirements of the target computing task from the GPU hardware on the determined computing nodes.

[0118] If the scheduler determines that there are multiple computing nodes in the computing cluster with remaining computing power resources that meet the computing power requirements of the target computing task, then these multiple computing nodes can be scored, and the suitability of each computing node among the multiple computing nodes for the target computing task can be evaluated based on the score. Then, a suitable computing node can be determined for the target computing task from the multiple computing nodes, and then the first GPU hardware with remaining computing power resources that meet the computing power requirements of the target computing task can be further determined from the GPU hardware on the suitable computing node.

[0119] It should be noted that the specific implementation details of scoring the above-mentioned multiple computing nodes will not be elaborated in this specification; for example, taking the above computing cluster as a Kubernetes cluster, the scoring mechanism of computing nodes in the Kubernetes cluster can be used to score each computing node.

[0120] In one embodiment shown, to achieve precise scheduling of the computing resources of the GPU hardware, the computing nodes in the computing cluster can also support virtualizing the computing resources of their GPU hardware into virtual GPUs. Specifically, these virtual GPUs can include a first type of virtual GPU for performing a first type of computing task and a second type of virtual GPU for performing a second type of computing task.

[0121] Correspondingly, the scheduler maintains the remaining computing resources of each GPU hardware in the computing cluster, specifically the first and second remaining computing resources corresponding to each GPU hardware in the computing cluster; wherein, the first remaining computing resources can represent the number of first-type virtual GPUs that can be created based on the remaining computing resources of the GPU hardware; and the second remaining computing resources can represent the number of second-type virtual GPUs that can be created based on the remaining computing resources of the GPU hardware.

[0122] For example, to facilitate fine-grained management of GPU hardware computing resources, the computing resources of a single GPU can be abstracted into 100 parts by default. For instance, for a single GPU, its computing resources could be abstracted into 100 Type 1 virtual GPUs and 100 Type 2 virtual GPUs by default. It's important to note that this abstraction into 100 Type 1 and 100 Type 2 virtual GPUs refers to the number of Type 1 and Type 2 virtual GPUs that can be created when all of the GPU's computing resources are idle. If the GPU's remaining computing resources change, the number of Type 1 and Type 2 virtual GPUs that can be abstracted based on those resources will typically change simultaneously.

[0123] In this scenario, if the target computing task is a type 1 computing task, when the scheduler determines the first GPU hardware whose remaining computing power resources meet the computing power requirements of the target computing task, it can specifically determine the GPU hardware whose remaining computing power resources meet the first requirement of the target computing task for the first type of virtual GPUs based on the maintained first remaining computing power corresponding to each GPU hardware in the computing cluster (that is, the number of type 1 virtual GPUs that can be created based on the remaining computing power resources). Then, the determined GPU hardware is used as the first GPU hardware whose remaining computing power resources meet the computing power requirements of the target computing task.

[0124] It should be noted that in practical applications, the aforementioned first GPU hardware may refer to a single GPU hardware or multiple GPU hardware, and no special limitation is made in this specification.

[0125] For example, if the computing power requirement of the target task does not exceed the computing power resources of a single GPU, the scheduler can, by default, schedule the target task to run on a single GPU in the computing cluster with sufficient remaining computing power resources. If the target task has a large computing power requirement, exceeding the computing power resources of a single GPU, the scheduler may need to schedule the target task to run on multiple GPUs in the computing cluster with sufficient remaining computing power resources.

[0126] It's important to clarify that the first and second remaining computing power values ​​maintained by the scheduler for each GPU in the computing cluster are typically only the estimated number of first and second type virtual GPUs that can be created based on the remaining computing resources of each GPU in the cluster. In reality, the virtualization process for each GPU in the computing cluster may not yet be complete. In other words, before any computing tasks are scheduled to the GPUs in the computing cluster, the individual GPUs do not need to be pre-virtualized. Instead, virtualization can be triggered only after a computing task is scheduled to a particular GPU.

[0127] In one embodiment shown, the first and second remaining computing power values ​​maintained by the scheduler corresponding to each GPU hardware in the computing cluster can be obtained by the scheduler updating the initial values ​​of the first and second remaining computing power values ​​reported by each computing node in the computing cluster when it joins the computing cluster in real time.

[0128] In this scenario, when each computing node in the computing cluster joins the computing cluster, it can report the initial values ​​of the first and second remaining computing power corresponding to each GPU hardware it is equipped with—that is, the maximum number of the first type of virtual GPUs and the second type of virtual GPUs that can be created when all the computing power resources of each GPU hardware are idle—to the scheduler.

[0129] After obtaining the initial values ​​of the first and second remaining computing power reported by each computing node when joining the computing cluster, the scheduler can maintain these initial values ​​locally. Subsequently, after scheduling a first-type or second-type computing task created in the computing cluster to any GPU hardware in the cluster, the scheduler can update the maintained initial values ​​of the first or second remaining computing power corresponding to that GPU hardware based on the number of first-type or second-type virtual GPUs occupied by that first-type or second-type computing task.

[0130] In another embodiment shown, the scheduler maintains a first and a second remaining computing power corresponding to each GPU hardware in the computing cluster, which may specifically be the remaining power reported in real time by each computing node in the computing cluster.

[0131] In this scenario, each computing node in the computing cluster can locally maintain initial values ​​for the first and second remaining computing power corresponding to each GPU hardware it carries. Then, based on the number of first or second type virtual GPUs occupied by the first or second type of computing tasks scheduled to run on the GPU hardware carried by this computing node, it can update the locally maintained initial values ​​for the first and second remaining computing power in real time. The updated first and second remaining computing power can then be reported to the scheduler in real time, and the scheduler will maintain them locally.

[0132] In this specification, the scheduler, referring to the remaining computing resources of each GPU hardware, determines the first GPU hardware in the computing cluster whose remaining computing resources meet the computing power requirements of the target computing task, and then schedules the target computing task to the first GPU hardware for execution.

[0133] In one embodiment shown, when the scheduler schedules a target computing task to run on a first GPU hardware in a computing cluster, it can specifically send the first demand of the target computing task for a first type of virtual GPU, and the hardware identifier of the first GPU hardware, to a first computing node; wherein, the first computing node is a computing node in the computing cluster that carries the first GPU hardware.

[0134] After receiving the first demand for a first type of virtual GPU from the target computing task, and the hardware identifier of the first GPU hardware, the first computing node can virtualize the first GPU hardware as needed according to the computing resource requirements of the target computing task, obtaining several first-type virtual GPUs corresponding to the first demand for first-type virtual GPUs from the target computing task. Then, the target computing task can be run based on these virtualized first-type virtual GPUs. For example, assuming the first demand for first-type virtual GPUs from the target computing task is N, then N first-type virtual GPUs can be virtualized as needed.

[0135] Of course, in practical applications, in addition to virtualizing the GPU hardware as needed according to the computing resource requirements of the computing task scheduled to a certain GPU hardware, the GPU hardware can also be virtualized in advance. After the computing task is scheduled to the GPU hardware, the required virtual GPU can be allocated for the computing task from the virtual GPUs obtained by virtualizing the GPU hardware in advance.

[0136] The detailed process of virtualizing the GPU hardware will not be described in detail in this specification. Those skilled in the art can refer to the relevant technical records.

[0137] For example, if the computing cluster mentioned above is a Kubernetes cluster, and the computing nodes in the cluster are equipped with GPUs, and the computing nodes support virtualizing the computing resources of the GPUs into virtual GPUs for use by containerized applications, then a GPU virtualization driver module is usually implemented on the computing node for the GPU hardware on the computing node, and the GPU virtualization driver module is used to perform virtualization processing on the GPU hardware on the computing node.

[0138] In one embodiment shown, the scheduler can also maintain a global topology corresponding to each computing node in the computing cluster; specifically, the global topology can be a topology composed of topology information reported by each computing node in the computing cluster.

[0139] For example, in practical applications, each computing node in a computing cluster can report topology information to the scheduler after joining the computing cluster. The scheduler can then integrate the topology information reported by each computing node to generate a global topology that describes the topological relationships between the GPU hardware on each computing node in the entire computing cluster.

[0140] When the scheduler sends the first demand of the target computing task for the first type of virtual GPU and the hardware identifier of the first GPU hardware to the first computing node equipped with the first GPU hardware, it can first query the global topology to determine the first computing node equipped with the first GPU hardware, and then send the first demand of the target computing task for the first type of virtual GPU and the hardware identifier of the first GPU hardware to the first computing node.

[0141] For example, in practical applications, the aforementioned global topology may specifically include the correspondence between the node identifiers of each computing node in the computing cluster and the hardware identifiers of the GPU hardware mounted on each computing node. In addition, it may also include network connection information (such as RDMA connection information, NVLINK connection information, etc.) between the scheduler and each computing node in the computing cluster.

[0142] In this scenario, when the scheduler sends the first demand of the target computing task for the first type of virtual GPU and the hardware identifier of the first GPU hardware to the first computing node equipped with the first GPU hardware, it can first query the above-mentioned correspondence contained in the global topology to determine the node identifier of the first computing node corresponding to the hardware identifier of the first GPU hardware.

[0143] After determining the node identifier of the first computing node, the network connection information contained in the global topology can be further queried. Then, based on the queried network connection information, the first requirement of the target computing task for the first type of virtual GPU and the hardware identifier of the first GPU hardware are sent to the first computing node.

[0144] Step 106: In response to the first instruction reported by the first computing node, the second type of computing task that has been scheduled to run on the first GPU hardware is rescheduled to run on the second GPU hardware in the computing cluster; wherein, the first computing node is a computing node equipped with the first GPU hardware; the first instruction is used to indicate that the first type of computing task will exclusively occupy the computing resources of the first GPU hardware; the first instruction is reported by the first computing node to the scheduler when the computing resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing task.

[0145] After the scheduler schedules the target computing task to the first GPU hardware determined from the computing cluster, the first computing node equipped with the first GPU hardware can run the target computing task on the first GPU hardware.

[0146] For example, as mentioned above, the first computing node can virtualize the first GPU hardware as needed according to the computing resource requirements of the target computing task, and obtain a number of first-type virtual GPUs corresponding to the first demand of the target computing task for the first type of virtual GPUs. Then, the target computing task can be run based on the virtualized number of first-type virtual GPUs.

[0147] In addition, the first computing node can also monitor in real time the computing resources of the first GPU hardware occupied by the first type of computing task running on the first GPU hardware; for example, a monitor can be implemented on the first computing node to monitor the computing resources of the first GPU hardware occupied by the target computing task.

[0148] It should be noted that, in practical applications, when the first computing node monitors the computing resources of the first type of computing tasks running on the first GPU hardware, it can specifically understand the resource usage of these first type of computing tasks on the first GPU hardware by monitoring the changes in the values ​​of computing power indicators related to the computing power resources of the first GPU hardware. These computing power indicators can specifically include any form of indicator that can describe the resource usage of the target computing task on the first GPU hardware.

[0149] Then, the first computing node can further determine whether the computing resources of the first GPU hardware occupied by these first type of computing tasks have reached a preset threshold based on the changes in the values ​​of these computing power indicators monitored.

[0150] In one embodiment shown, the aforementioned computing power metrics may specifically include one or a combination of metrics such as the response time of requests related to the target computing task, the utilization rate of the first GPU hardware, and the utilization rate of the video memory of the first GPU hardware. The first computing node can determine whether the values ​​of these monitored computing power metrics reach a preset threshold, thereby determining that the computing power resources of the first GPU hardware occupied by these first-type computing tasks have reached the preset threshold.

[0151] Of course, in practical applications, in addition to the indicators listed above, the above computing power indicators may also include other types of indicators that can describe the computing power resources occupied by the target computing task on the first GPU hardware, which will not be listed one by one in this specification.

[0152] If the first computing node determines that the computing resources of the first GPU hardware occupied by these first-type computing tasks have reached a preset threshold, since the first-type computing tasks are high-service-level computing tasks, the first computing node can report a first indication to the scheduler. This first indication can be used to indicate that the first-type computing tasks will exclusively occupy the computing resources of the first GPU hardware.

[0153] For example, an eviction mechanism can be implemented on the first compute node. This eviction mechanism can listen to the monitoring results of a monitor implemented on the first compute node regarding the computing resources of the first type of computing tasks occupying the first GPU hardware. When the eviction mechanism detects from the monitor that the computing resources of the first type of computing tasks occupying the first GPU hardware have reached a preset threshold, it can initiate the eviction process for the second type of computing tasks already scheduled to run on the first GPU hardware. This stops the second Pod from running on the first GPU hardware and releases the computing resources of the first GPU hardware occupied by the second Pod. Then, the first compute node can further report a first indication to the scheduler that the first type of computing tasks will exclusively occupy the computing resources of the first GPU hardware, instructing the scheduler not to schedule the aforementioned second type of computing tasks to the first GPU hardware.

[0154] After receiving the first instruction reported by the first computing node, the scheduler can respond to the first instruction by rescheduling the second type of computing tasks that have been scheduled to run on the first GPU hardware to run on the second GPU hardware in the computing cluster.

[0155] In one embodiment shown, the second GPU hardware may specifically be the GPU hardware among the remaining GPU hardware on the first computing node, excluding the first GPU hardware, whose remaining computing power resources meet the computing power requirements of the second type of computing tasks scheduled to run on the first GPU hardware; or, the second GPU hardware may also be the GPU hardware among the GPU hardware on other computing nodes in the computing cluster besides the first computing node equipped with the first GPU hardware, whose remaining computing power resources meet the computing power requirements of the second type of computing tasks scheduled to run on the first GPU hardware.

[0156] In this scenario, when the scheduler reschedules a second type of computing task that has been scheduled to run on the first GPU hardware to run on a second GPU hardware with remaining computing power resources that meet the computing power requirements of the second type of computing task, it can first determine whether there are any second GPU hardwares with remaining computing power resources that meet the computing power requirements of the second type of computing task that has been scheduled to run on the first GPU hardware among the other GPU hardwares on the first computing node besides the first GPU hardware.

[0157] On the one hand, if there are other GPUs on the first computing node besides the first GPU hardware that have remaining computing power resources to meet the computing power requirements of the second type of computing task that has been scheduled to run on the first GPU hardware, the second type of computing task can be rescheduled within the first computing node and further scheduled to run on the second GPU hardware on the first computing node.

[0158] On the other hand, if there are no other GPUs on the first computing node besides the first GPU hardware that have sufficient computing power to meet the computing power requirements of the second type of computing task already scheduled to run on the first GPU hardware, the scheduler can further determine whether there are other GPUs on the computing nodes in the computing cluster besides the first computing node that have sufficient computing power to meet the computing power requirements of the second type of computing task. If so, the second type of computing task can be rescheduled across nodes and further scheduled to run on the second GPU hardware on the other computing node.

[0159] In this way, the rescheduling of the second type of computing tasks running on the first GPU hardware can be avoided as much as possible.

[0160] The specific process by which the scheduler reschedules the second type of computing task that has been scheduled to run on the first GPU hardware to the second GPU hardware can be referred to the specific process by which the scheduler schedules the target computing task to the first GPU hardware as described in the above embodiments, and will not be repeated in this specification.

[0161] The above embodiments only describe in detail the process of rescheduling the second type of computing task running on the first computing node to other second GPU hardware in the computing cluster from the perspective of the first computing node. It should be emphasized that in practical applications, for any computing node in the computing cluster, if the first type of computing task running on a certain GPU hardware on the computing node occupies the computing resources of the GPU hardware to a preset threshold, it can also report the above-mentioned first instruction to the scheduler in the same way as the first computing node, and the scheduler will rescheduling the second type of computing task running on the computing node to other GPU hardware in the cluster. These will not be listed one by one in this specification.

[0162] In one embodiment shown, after the scheduler determines the task type of the target computing task, if the target computing task is the second type of computing task, the scheduler may also refer to the remaining computing resources of each GPU hardware maintained, and determine a third GPU hardware from the physical GPU hardware in the computing cluster that is not exclusively occupied by the first type of computing task, and whose remaining computing resources can meet the computing power requirements of the target computing task.

[0163] For example, in practical applications, the scheduler can still refer to the remaining computing resources of each GPU hardware it maintains. First, it can identify the computing nodes in the computing cluster whose remaining computing resources are not exclusively occupied by the first type of computing task and can meet the computing power requirements of the target computing task. Then, from the GPU hardware on the identified computing nodes that are not exclusively occupied by the first type of computing task, it can further identify a third GPU hardware whose remaining computing resources can meet the computing power requirements of the target computing task.

[0164] If the scheduler identifies multiple computing nodes in the computing cluster whose remaining computing resources, which are not exclusively used by the first type of computing task, meet the computing power requirements of the target computing task, then these multiple computing nodes can still be scored. Based on the scores, the suitability of each computing node among these multiple computing nodes for the target computing task can be evaluated. In this way, a suitable computing node can be identified for the target computing task from these multiple computing nodes. Then, from the GPU hardware on the suitable computing node that is not exclusively used by the first type of computing task, a third GPU hardware whose remaining computing resources meet the computing power requirements of the target computing task can be further identified.

[0165] In one embodiment shown, the scheduler maintains the remaining computing resources of each GPU hardware in the computing cluster, which may specifically be a first remaining computing power and a second remaining computing power corresponding to each GPU hardware in the computing cluster; wherein, the first remaining computing power may represent the number of first-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware; and the second remaining computing power may represent the number of second-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware.

[0166] In this scenario, if the target computing task is a second type of computing task, when the scheduler determines a third GPU hardware in the computing cluster that is not exclusively occupied by a first type of computing task and whose remaining computing power resources meet the computing power requirements of the target computing task, it can specifically determine the GPU hardware whose second remaining computing power meets the second requirement of the target computing task for second type virtual GPUs from among the physical GPU hardware in the computing cluster that is not exclusively occupied by a first type of computing task. Then, the determined GPU hardware is used as the third GPU hardware whose remaining computing power resources meet the computing power requirements of the target computing task.

[0167] It should be noted that, in practical applications, the aforementioned third GPU hardware may refer to a single GPU hardware or multiple GPU hardware, and this specification does not impose any specific limitations.

[0168] In this specification, the scheduler, referring to the remaining computing resources of each GPU hardware maintained by the scheduler, determines a third GPU hardware from the physical GPU hardware in the computing cluster that is not exclusively occupied by the first type of computing task, and then further schedules the target computing task to the third GPU hardware for execution.

[0169] In one embodiment shown, when the scheduler schedules a target computing task to run on a third GPU in the computing cluster, it can specifically send the second demand of the target computing task for the second type of virtual GPU, as well as the hardware identifier of the third GPU, to a second computing node; wherein, the second computing node is a computing node in the computing cluster that carries the third GPU.

[0170] After receiving the second demand for the second type of virtual GPU from the target computing task and the hardware identifier of the third GPU hardware, the second computing node can virtualize the third GPU hardware as needed according to the computing resource requirements of the target computing task, thereby obtaining several second type of virtual GPUs corresponding to the second demand for the second type of virtual GPU from the target computing task. Then, the target computing task can be run based on the several virtualized second type of virtual GPUs.

[0171] In one embodiment shown, the scheduler can also maintain a global topology corresponding to each compute node in the computing cluster. When the scheduler sends the second demand of the target computing task for the second type of virtual GPU and the hardware identifier of the third GPU hardware to the first compute node equipped with the first GPU hardware, it can first query the maintained global topology corresponding to each GPU hardware in the computing cluster to determine the second compute node equipped with the third GPU hardware, and then send the second demand of the target computing task for the second type of virtual GPU and the hardware identifier of the third GPU hardware to the second compute node.

[0172] The specific implementation process of determining the second computing node equipped with the third GPU hardware by querying the global topology, and then sending the second requirement of the target computing task for the second type of virtual GPU, as well as the hardware identifier of the third GPU hardware, to the second computing node, will not be elaborated here.

[0173] In one embodiment shown, the scheduler can maintain not only the remaining computing resources of each GPU in the computing cluster, but also the hardware mode corresponding to each GPU in the computing cluster.

[0174] This hardware mode can include resource sharing mode and resource exclusive mode.

[0175] In this context, resource-sharing mode indicates that the GPU hardware's computing resources support running multiple types of computing tasks; resource-exclusive mode indicates that the GPU hardware's computing resources are only used to execute the first type of computing task. By default, the hardware mode of each GPU in the computing cluster can be resource-sharing mode, meaning that the GPU hardware's computing resources can support running multiple types of computing tasks.

[0176] Once the scheduler receives a first instruction reported by a computing node, it can switch the GPU hardware from resource-sharing mode to resource-exclusive mode. For example, after receiving the first instruction reported by the first computing node, the scheduler can respond to the first instruction by switching the first GPU hardware from resource-sharing mode to resource-exclusive mode.

[0177] In one scenario, since only GPU hardware in resource-sharing mode supports running the second type of computing task, the scheduler needs to consider the specific mode of each GPU hardware in the computing cluster when determining which GPU hardware has remaining computing power resources to meet the computing power requirements of the second type of computing task from among the GPU hardware in the computing cluster.

[0178] In this scenario, the scheduler can identify GPUs in the computing cluster that are in resource-sharing mode and have sufficient remaining computing power to meet the computational requirements of the second type of computing task. For example, if the target computing task is a second type of computing task, it can be scheduled to run on a third GPU in the computing cluster that is in resource-sharing mode and has sufficient remaining computing power to meet the computational requirements of the target computing task.

[0179] In another scenario, since both resource-sharing and resource-exclusive GPU hardware can support running Type I computing tasks, the scheduler can determine which GPU hardware has remaining computing power resources to meet the computing power requirements of Type I computing tasks from among the GPU hardware in the computing cluster. This is because the hardware mode of each GPU hardware in the computing cluster is not considered. In other words, the scheduler only needs to determine which GPU hardware has remaining computing power resources to meet the computing power requirements of Type I computing tasks from among the GPU hardware in both resource-sharing and resource-exclusive modes.

[0180] In one embodiment shown, a computing node in the computing cluster can also report a second instruction to the scheduler if it determines that a first type of computing task running on a GPU hardware mounted on the computing node occupies less than a preset threshold of the computing resources of the GPU hardware. After receiving the second instruction reported by a computing node, the scheduler can switch the GPU hardware from a resource-exclusive mode to a resource-sharing mode in response to the second instruction.

[0181] For example, if the target computing task is a first type of computing task, after the first type of computing task is scheduled to run on the first GPU hardware in the computing cluster, the scheduler can switch the first GPU hardware from the resource exclusive mode to the resource sharing mode after receiving the second instruction reported by the first computing node equipped with the first GPU hardware.

[0182] Please see Figure 2 , Figure 2 This is a flowchart illustrating another method for scheduling GPU computing resources, as shown in this specification. The method is applied to any computing node in a computing cluster; at least some computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; the service level of the first type of computing task is higher than that of the second type of computing task; the method includes the following execution process:

[0183] Step 202: In response to the scheduler corresponding to the computing cluster scheduling the first type of computing task to the first GPU hardware on the computing node, the first type of computing task is run on the first GPU hardware; wherein, the first type of computing task is scheduled to the computing node by the scheduler when it is determined that the remaining computing power resources of the first GPU hardware meet the computing power requirements of the first type of computing task.

[0184] Step 204: Determine whether the computing resources of the first GPU hardware occupied by the first type of computing task have reached a preset threshold;

[0185] Step 206: If the computing resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold, a first indication is reported to the scheduler, wherein the first indication is used to indicate that the first type of computing task will exclusively occupy the computing resources of the first GPU hardware, so that the scheduler will reschedule the second type of computing task that has been scheduled to run on the first GPU hardware to run on the second GPU hardware in the computing cluster; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing task.

[0186] It should be noted that the above is a specific embodiment provided from the perspective of a single computing node in the computing cluster. The specific implementation details of this embodiment are different from those of the previous embodiment. Figure 1 The specific implementation details disclosed in the illustrated embodiments are the same and will not be repeated in this specification.

[0187] The following will use the aforementioned computing cluster as a Kubernetes cluster as an example to explain the above technical solution in detail.

[0188] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the scheduling of computing resources of the GPU hardware on a compute node in a Kubernetes cluster.

[0189] In a Kubernetes cluster, the first type of computing task mentioned above can be a computing task run by a Pod with a QoS service level of Guaranteed (hereinafter referred to as GT task), and the second type of computing task mentioned above can be a computing task run by a Pod with a QoS service level of BestEffort (hereinafter referred to as BE task).

[0190] If the compute nodes in a Kubernetes cluster are equipped with GPUs, and these compute nodes support virtualizing the GPU's computing resources into virtual GPUs for use by containerized applications, a plugin called GPUDevice Plugin is typically implemented on the compute nodes. This plugin is primarily used to discover, manage, and schedule the computing resources of all the GPU hardware on the compute node. It acts as a bridge between the kubelet (daemon) on the Kubernetes node and the GPU hardware, allowing GPU resources to be transparently used by Pods in the Kubernetes cluster through resource requests. In addition, each compute node can also implement a GPU virtualization driver module, which is specifically used to virtualize the GPU hardware on the compute node.

[0191] When a compute node (such as a physical machine or virtual machine) joins a Kubernetes cluster, to facilitate fine-grained management of the computing resources of the physical GPU hardware on that compute node, the GPU virtualization driver module implemented on that compute node can abstract the remaining computing resources of each GPU hardware into 100 GT resources (i.e., the first type of remaining computing power mentioned above) and 100 BE resources (i.e., the second type of remaining computing power mentioned above) according to a percentage. It should be noted that GT resources represent the first type of virtual GPUs used to execute GT tasks, and BE resources represent the second type of virtual GPUs used to execute BE tasks.

[0192] Please see Figure 3The GPU Device Plugin implemented on this compute node detects the GPU model and quantity of each GPU hardware unit on the compute node at startup. It then obtains the quantity of GT (Gross Ground) and BE (Browser) resources abstracted from the remaining computing power resources of each GPU hardware unit from the GPU virtualization driver module. This obtained quantity of GT and BE resources is cached within the plugin as resource information for the compute node. For example, the hardware identifier of each physical GPU can be associated with the quantity of GT and BE resources abstracted from each physical GPU and cached within the GPU Device Plugin.

[0193] Please continue reading Figure 3 The GPU Device Plugin can also expose a ListAndWatch interface. Once the compute node joins the Kubernetes cluster, the Kubelet daemon on that compute node can call the ListAndWatch interface to retrieve the compute node's resource information cached in the GPU Device Plugin's internal data structure. This retrieved resource information is then reported to the Kubernetes cluster's API Server. The API Server is a core component of the Kubernetes cluster and serves as the entry point for cluster control.

[0194] Please continue reading Figure 3 When a compute node joins a Kubernetes cluster, the GPU Device Plugin can also collect the topology information of the GPU hardware on that compute node and then report the collected topology information to the API Server. For example, the collected topology information can be encapsulated into a Device data object and reported to the API Server.

[0195] Specifically, the aforementioned topology information may include the correspondence between the node identifiers of each computing node in the computing cluster and the hardware identifiers of the GPU hardware mounted on each computing node. In addition, it may include network connection information between the scheduler and each computing node in the computing cluster (such as RDMA connection information between the scheduler and each computing node), as well as network connection information between the GPU hardware mounted on each computing node in the computing cluster (such as NVLINK connection information between each GPU hardware).

[0196] Please continue reading Figure 3Once a compute node joins the Kubernetes cluster, the Kubernetes cluster scheduler can read the compute node's resource information (i.e., the number of GT and BE resources of the compute node) and the topology information of the GPU hardware on the compute node from the API Server, and cache this information inside the scheduler as a reference for subsequent task scheduling.

[0197] Please continue reading Figure 3 Once a user creates a first Pod in a Kubernetes cluster to execute GT tasks based on specific computing needs, the scheduler can first determine the GT resource requirements of that first Pod. For example, the GT resource requirements of the first Pod can typically include the number of GPUs it occupies and the number of GT resources required on each GPU.

[0198] After determining the GT resource requirements of the first Pod, the scheduler can make scheduling decisions for the first Pod based on the resource information of each compute node in the cached Kubernetes cluster.

[0199] Specifically, the scheduler can traverse the resource information of each compute node in the cached Kubernetes cluster, filter out the number of remaining GT resources of the GPU hardware, and select the compute nodes that meet the GT resource requirements of the first Pod.

[0200] If the Kubernetes cluster does not have a compute node that meets the GT resource requirements of the first Pod at this time, the scheduling round will be terminated and the scheduling of the first Pod will be marked as a failure. If there are multiple compute nodes in the Kubernetes cluster that meet the GT resource requirements of the first Pod, the scheduler can also score each compute node based on the scoring mechanism of the compute nodes in the Kubernetes cluster, and evaluate the suitability of each compute node among the multiple compute nodes for the GT task executed by the first Pod based on the score. Then, a suitable first compute node can be determined for the GT task from the multiple compute nodes, and then the number of remaining GT resources on the GPU hardware of the first compute node can be further determined to meet the GT resource requirements of the GT task.

[0201] Furthermore, after the scheduler determines the number of remaining GT resources to satisfy the GT resource requirements of the GT task executed by the first Pod, the scheduling decision for the first Pod is completed. The scheduler can then update the scheduling result to the first Pod. For example, it can update the annotations field of the first Pod with information such as the hardware identifier of the first GPU allocated to the first Pod and the number of GT resources required by the first Pod.

[0202] After the decision result is updated in the first Pod, the decision maker can send the first Pod to the first compute node determined by the decision based on the maintained topology information, so that the Kubelet daemon of the first compute node can prepare the Pod for startup.

[0203] Please continue reading Figure 3 Assuming that the first compute node is the newly added compute node mentioned above, the Kubelet daemon of the first compute node can start preparing to start the first Pod after listening to the scheduler's scheduling result for the first Pod.

[0204] Before the first Pod starts, the Kubelet daemon of the first compute node can call the Allocate interface provided by the GPU Device Plugin of the first compute node to bind the first Pod to the aforementioned first GPU hardware.

[0205] Specifically, the Kubelet daemon of the first compute node can read the scheduling results recorded in the annotations field of the first Pod, obtain the hardware identifier of the first GPU hardware allocated to the first Pod, and the number of GT resources required by the first Pod. Then, it can further call the Allocate interface provided by the GPU DevicePlugin plugin of the first compute node to mount the corresponding device file on the first GPU hardware, bind the first Pod to the first GPU hardware, and continue to send a virtual resource allocation request to the GPU virtualization driver module of the first compute node. The virtual resource allocation request may contain the number of GT resources required by the first Pod, that is, the number of first-type virtual GPUs required.

[0206] After receiving the resource allocation request, the GPU virtualization driver module of the first computing node can virtualize the first GPU hardware as needed according to the number of GT resources required by the first Pod contained in the resource allocation request, obtain a number of GT resources corresponding to the number of GT resources required by the first Pod, and then allocate these GT resources to the first Pod.

[0207] Once the necessary GT resources are allocated to the first Pod, the Kubelet daemon of the first compute node can start the first Pod and run it based on the GT resources allocated to it.

[0208] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the elastic scheduling of low-service-level computing tasks that have been scheduled to GPU hardware on compute nodes in a Kubernetes cluster.

[0209] On a compute node in a Kubernetes cluster, a monitor and an eviction mechanism can be implemented. The monitor tracks the computing resources of the GPU hardware used by Pods scheduled to execute ground truth (GT) tasks on that compute node. The eviction mechanism listens to the monitor's results and, if it detects from the monitor that the computing resources used by a Pod executing a GT task on the GPU hardware have reached a preset threshold, terminates the Pod's operation on that GPU hardware and releases the computing resources occupied by that Pod.

[0210] like Figure 4 As shown, after the scheduler schedules the first Pod, which is used to execute the GT task, to the first GPU hardware on the first computing node, the monitor on the first computing node can monitor the computing resources of the first GPU hardware occupied by the first Pod. The eviction mechanism on the first computing node can listen to the monitoring results. When the computing resources of the first GPU hardware occupied by the first Pod continue to increase, and the eviction mechanism detects from the monitor that the computing resources of the first GPU hardware occupied by the first Pod have reached a preset threshold, the eviction mechanism can initiate the eviction process for the second Pod, which is scheduled to run on the first GPU hardware and is used to execute the BE task. This process stops the second Pod from running on the first GPU hardware, releases the computing resources of the first GPU hardware occupied by the second Pod, and reports an instruction to the Kubernetes cluster API Server to evict the second Pod from the first GPU hardware.

[0211] The process of suspending the operation of the second Pod on the first GPU hardware and releasing the computing resources of the first GPU hardware occupied by the second Pod can be executed after the second Pod is rescheduled, or it can be executed immediately before the second Pod is rescheduled. In this embodiment, no special limitation is made.

[0212] In addition, the GPU Device Plugin on the first compute node can also listen to the monitoring results of this monitor. When the GPU Device Plugin detects from the monitor that the computing resources of the first GPU hardware occupied by the first Pod have reached a preset threshold, it can also report an indication to the Kubernetes cluster's API Server that the first Pod will exclusively occupy the computing resources of the first GPU hardware.

[0213] The scheduler of a Kubernetes cluster can read instructions reported by the first compute node from the API Server.

[0214] On the one hand, when the scheduler reads from the API Server an indication from the GPU Device Plugin on the first compute node that the first Pod will exclusively occupy the computing resources of the first GPU hardware, it can respond to the indication and switch the first GPU hardware from resource sharing mode to resource exclusive mode. Subsequently, the computing resources of the first GPU hardware will no longer be allocated to Pods that execute BE tasks.

[0215] On the other hand, when the scheduler reads from the API Server an instruction reported by the eviction device on the first compute node to evict the second Pod from the first GPU hardware, it can respond to the instruction and initiate a process to reschedule the second Pod to run on other GPU hardware in the cluster.

[0216] Of course, in practical applications, the eviction mechanism may not report the instruction to evict the second Pod from the first GPU hardware. In this case, the instruction reported by the GPU Device Plugin on the first compute node that the first Pod will exclusively occupy the computing resources of the first GPU hardware can not only trigger the scheduler to switch the first GPU hardware from resource sharing mode to resource exclusive mode, but also trigger the scheduler to start the process of rescheduling the second Pod.

[0217] Please continue reading Figure 4After a period of time, if the computing resources occupied by the first GPU continuously decrease, and the GPU Device Plugin of the first compute node detects from the monitor that the computing resources occupied by the first Pod on the first GPU hardware are below a preset threshold, it can also report an instruction to the Kubernetes cluster API Server to switch the first GPU hardware from exclusive resource mode to shared resource mode. After the scheduler reads this instruction reported by the GPU Device Plugin on the first compute node from the API Server, it can switch the first GPU hardware from exclusive resource mode to shared resource mode. At this time, the first GPU hardware is restored to support the hybrid deployment of Pods for executing GT tasks and Pods for executing BE tasks. Subsequently, idle computing resources on the first GPU hardware can be allocated to Pods executing BE tasks.

[0218] As demonstrated by the above embodiments, in scenarios where Pods for executing GT tasks and Pods for executing BE tasks are deployed in a hybrid manner on the same GPU hardware, the introduction of elastic scheduling for Pods executing BE tasks not only allows for the removal of BE tasks running on the GPU hardware when the computing resources consumed by GT tasks reach a preset threshold, thus ensuring the service level of GT tasks, but also allows for the rescheduling of BE tasks to other GPU hardware, ensuring that BE tasks are not disrupted by GT tasks and cannot function properly. Furthermore, when the computing resources consumed by GT tasks on the GPU hardware fall below a preset threshold, the idle computing resources of the GPU hardware can be allocated to other BE tasks, allowing Pods executing BE tasks and Pods executing GT tasks to share GPU computing resources in a more rational manner.

[0219] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of apparatus, electronic devices, and storage media.

[0220] Figure 5 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. Please refer to... Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other necessary hardware. One or more embodiments of this specification can be implemented in software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0221] like Figure 6 As shown, Figure 6 This is a block diagram illustrating a GPU computing resource scheduling apparatus according to an exemplary embodiment, which can be applied to, for example... Figure 5 The illustrated electronic device implements the technical solution of this specification. In this device, at least some computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; the service level of the first type of computing task is higher than that of the second type of computing task; the device 60 includes:

[0222] The first determining module 601, in response to a target computing task created in the computing cluster, determines the task type of the target computing task;

[0223] The scheduling module 602, if the target computing task is a first type of computing task, schedules the target computing task to the first GPU hardware in the computing cluster for execution; wherein, the first GPU hardware is GPU hardware whose remaining computing power resources meet the computing power requirements of the target computing task; in response to a first instruction reported by a first computing node, reschedules the second type of computing task that has been scheduled to run on the first GPU hardware to the second GPU hardware in the computing cluster for execution; wherein, the first computing node is a computing node equipped with the first GPU hardware; the first instruction is used to indicate that the first type of computing task will exclusively occupy the computing power resources of the first GPU hardware; the first instruction is reported by the first computing node to the scheduler when the computing power resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold; the second GPU hardware is GPU hardware whose remaining computing power resources meet the computing power requirements of the second type of computing task.

[0224] like Figure 7 As shown, Figure 7This is a block diagram illustrating another GPU computing resource scheduling apparatus according to an exemplary embodiment of this specification. This apparatus can also be applied to, for example... Figure 5 The illustrated electronic device implements the technical solution of this specification. In this device, at least some computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; the service level of the first type of computing task is higher than that of the second type of computing task; the device 70 includes:

[0225] The running module 701 responds to the scheduler corresponding to the computing cluster scheduling the first type of computing tasks to the first GPU hardware mounted on the computing node, and runs the first type of computing tasks on the first GPU hardware; wherein, the first type of computing tasks are scheduled to the computing node by the scheduler when it is determined that the remaining computing power resources of the first GPU hardware meet the computing power requirements of the first type of computing tasks.

[0226] The second determining module 702 determines whether the computing resources of the first GPU hardware occupied by the first type of computing task have reached a preset threshold.

[0227] The reporting module 703, if the computing resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold, reports a first indication to the scheduler, wherein the first indication is used to indicate that the first type of computing task will exclusively occupy the computing resources of the first GPU hardware, so that the scheduler will reschedule the second type of computing task that has been scheduled to run on the first GPU hardware to run on the second GPU hardware in the computing cluster; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing task.

[0228] Accordingly, this specification also provides an electronic device including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement all the steps in the previously described method flow.

[0229] Accordingly, this specification also provides a computer-readable storage medium having stored thereon executable computer program instructions; wherein, when executed by a processor, the instructions implement all the steps in the previously described method flow.

[0230] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, they implement all the steps in the previously described method flow.

[0231] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0232] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C2051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0233] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, it is not excluded that with the future development of computer technology, the computer implementing the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0234] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, it is not excluded that the process, method, product, or apparatus that includes the elements may also have other identical or equivalent elements. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0235] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0236] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0237] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0238] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0239] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0240] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0241] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0242] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0243] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0244] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0245] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for scheduling GPU computing resources, applied to a scheduler corresponding to a computing cluster; wherein, At least some of the computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; The service level of the first type of computing task is higher than that of the second type of computing task; the method includes: In response to a target computing task created in the computing cluster, the task type of the target computing task is determined; If the target computing task is the first type of computing task, the target computing task is scheduled to run on the first GPU hardware in the computing cluster; wherein, the first GPU hardware is GPU hardware whose remaining computing power resources meet the computing power requirements of the target computing task; In response to a first instruction reported by the first computing node, the second type of computing tasks that have been scheduled to run on the first GPU hardware are rescheduled to run on the second GPU hardware in the computing cluster; wherein, the first computing node is a computing node equipped with the first GPU hardware; the first instruction is used to indicate that the first type of computing tasks will exclusively occupy the computing resources of the first GPU hardware; the first instruction is reported by the first computing node to the scheduler when the computing resources of the first GPU hardware occupied by the first type of computing tasks reach a preset threshold; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing tasks.

2. The method of claim 1, further comprising: If the target computing task is the second type of computing task, the target computing task is scheduled to run on the third GPU hardware in the computing cluster; wherein, the third GPU hardware is GPU hardware whose computing resources are not exclusively occupied by the first type of computing task, and whose remaining computing resources meet the computing power requirements of the target computing task.

3. The method as described in claim 2, wherein the scheduler maintains a hardware mode corresponding to each GPU hardware in the computing cluster; wherein, The hardware modes include a resource-sharing mode and a resource-exclusive mode; the resource-sharing mode means that the computing resources of the GPU hardware support running multiple types of computing tasks; the resource-exclusive mode means that the computing resources of the GPU hardware are only used to execute the first type of computing task.

4. The method of claim 3, further comprising: In response to the first instruction reported by the first computing node equipped with the first GPU hardware, the first GPU hardware is switched from resource sharing mode to resource exclusive mode. If the target computing task is the second type of computing task, scheduling the target computing task to the third GPU hardware in the computing cluster for execution includes: If the target computing task is the second type of computing task, the target computing task is scheduled to run on the third GPU hardware in the computing cluster; wherein, the third GPU hardware is GPU hardware that is in the resource sharing mode and whose remaining computing power resources meet the computing power requirements of the target computing task.

5. The method of claim 4, further comprising: In response to a second instruction from the first computing node, the first GPU hardware is switched from a resource-exclusive mode to a resource-sharing mode; wherein, the second instruction is reported by the first computing node to the scheduler when the computing power resources of the first GPU hardware occupied by the first type of computing task are lower than a preset threshold.

6. The method as described in claim 1, wherein the computing node supports virtualizing the computing resources of the GPU hardware it carries into a virtual GPU; the virtual GPU includes a first type of virtual GPU for performing the first type of computing tasks and a second type of virtual GPU for performing the second type of computing tasks.

7. The method of claim 6, wherein the scheduler maintains a first remaining computing power and a second remaining computing power corresponding to each GPU hardware in the computing cluster; the first remaining computing power represents the number of first-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware; and the second remaining computing power represents the number of second-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware. If the target computing task is the first type of computing task, scheduling the target computing task to the first GPU hardware in the computing cluster for execution includes: If the target computing task is the first type of computing task, based on the maintained first computing power remaining amount corresponding to each GPU hardware, a first GPU hardware in the computing cluster whose first computing power remaining amount satisfies the first demand of the target computing task for the first type of virtual GPU is determined from the computing cluster, and the target computing task is scheduled to run on the first GPU hardware in the computing cluster. If the target computing task is the second type of computing task, scheduling the target computing task to the third GPU hardware in the computing cluster for execution includes: If the target computing task is the second type of computing task, based on the maintained second computing power remaining amount corresponding to each GPU hardware, a third GPU hardware that is not exclusively occupied by the first type of computing task and whose second computing power remaining amount meets the second requirement of the target computing task for the second type of virtual GPU is determined from the computing cluster, and the target computing task is scheduled to run on the third GPU hardware in the computing cluster.

8. The method as described in claim 7, Scheduling the target computing task to the first GPU hardware in the computing cluster for execution includes: The first requirement of the target computing task for the first type of virtual GPU and the hardware identifier of the first GPU hardware are sent to the first computing node equipped with the first GPU hardware, so that the first computing node virtualizes the first GPU hardware to obtain a plurality of first type virtual GPUs corresponding to the first requirement, and runs the target computing task based on the plurality of first type virtual GPUs. Scheduling the target computing task to run on a third GPU in the computing cluster includes: The second requirement of the target computing task for the second type of virtual GPU and the hardware identifier of the third GPU hardware are sent to the second computing node equipped with the third GPU, so that the second computing node virtualizes the third GPU hardware to obtain a plurality of second type virtual GPUs corresponding to the second requirement, and runs the target computing task based on the plurality of second type virtual GPUs.

9. The method of claim 7, wherein the scheduler maintains a global topology corresponding to each computing node in the computing cluster; wherein, The global topology is a topology composed of the topology information reported by each computing node in the computing cluster; Sending the first requirement of the target computing task for the first type of virtual GPU, and the hardware identifier of the first GPU hardware, to the first computing node equipped with the first GPU hardware includes: The global topology is queried to determine the first computing node equipped with the first GPU hardware, and the first requirement of the target computing task for the first type of virtual GPU, as well as the hardware identifier of the first GPU hardware, are sent to the first computing node. Sending the second requirement of the target computing task for the second type of virtual GPU, and the hardware identifier of the third GPU hardware, to the second computing node equipped with the third GPU includes: The global topology is queried to determine the second computing node equipped with the third GPU hardware, and the second requirement of the target computing task for the second type of virtual GPU, as well as the hardware identifier of the third GPU hardware, are sent to the second computing node.

10. The method of claim 7, further comprising: The system obtains the initial values ​​of the first and second remaining computing power reported by each computing node when it joins the computing cluster, and maintains the obtained initial values ​​of the first and second remaining computing power locally. In response to scheduling the first or second type of computing task created in the computing cluster to any GPU hardware in the computing cluster, the system updates the maintained initial values ​​of the first or second remaining computing power of the GPU hardware based on the number of the first or second type of virtual GPUs occupied by the first or second type of computing task. or, The first and second remaining computing power are obtained from the real-time reports of each computing node in the computing cluster, and the obtained first and second remaining computing power are maintained locally. The first and second remaining computing power reported by each computing node in real time are obtained by updating the initial values ​​of the first and second remaining computing power based on the number of first or second type virtual GPUs occupied by the first or second type of computing tasks scheduled to run on the GPU hardware on the computing node.

11. The method of claim 1, wherein the second GPU hardware includes GPU hardware on the first computing node whose remaining computing resources, other than the first GPU hardware, meet the computing power requirements of the second type of computing task; or, the second GPU hardware includes GPU hardware on other computing nodes in the computing cluster whose remaining computing resources, other than the first computing node, meet the computing power requirements of the second type of computing task. The second type of computing tasks that have already been scheduled to run on the first GPU hardware are rescheduled to run on the second GPU hardware, including: Determine whether there are any second GPUs among the other GPUs on the first computing node besides the first GPU hardware that have remaining computing power resources to meet the computing power requirements of the second type of computing tasks that have been scheduled to run on the first GPU hardware. If the second GPU hardware exists among the other GPU hardware, the second type of computing task is further scheduled to be run on the second GPU hardware. If the second GPU hardware is not present in the other GPU hardware, determine whether there is a second GPU hardware with remaining computing power resources that can meet the computing power requirements of the second type of computing task among the GPU hardware on the other computing nodes in the computing cluster besides the first computing node; if so, further schedule the second type of computing task to the second GPU hardware for operation.

12. The method of claim 1, wherein the computing cluster is a Kubernetes cluster; the computing nodes support the mixed deployment of multiple containers for running different types of computing tasks on the same GPU hardware; and the target computing task is a computing task running in containers deployed on various GPU hardware in the Kubernetes cluster.

13. The method of claim 12, wherein the first type of computing task is a computing task run by a container with a QoS service level of Guaranteed; and the second type of computing task is a computing task run by a container with a QoS service level of BestEffort.

14. A method for scheduling GPU computing resources, applied to any computing node in a computing cluster; wherein, At least some of the computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; The service level of the first type of computing task is higher than that of the second type of computing task; the method includes: In response to the scheduler corresponding to the computing cluster scheduling the first type of computing task to the first GPU hardware mounted on the computing node, the first type of computing task is run on the first GPU hardware; wherein, the first type of computing task is scheduled to the computing node by the scheduler when it is determined that the remaining computing power resources of the first GPU hardware meet the computing power requirements of the first type of computing task; Determine whether the computing resources of the first GPU hardware occupied by the first type of computing task have reached a preset threshold; If the computing resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold, a first indication is reported to the scheduler, wherein the first indication is used to indicate that the first type of computing task will exclusively occupy the computing resources of the first GPU hardware, so that the scheduler will reschedule the second type of computing task that has been scheduled to run on the first GPU hardware to run on the second GPU hardware in the computing cluster; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing task.

15. The method of claim 14, wherein determining whether the computing resources of the first GPU hardware occupied by the first type of computing task have reached a preset threshold includes: Determine whether the values ​​of computing power indicators related to the computing power resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold; wherein, the computing power indicators include any or more of the following indicators: response time of requests related to the first type of computing task, utilization rate of the first GPU hardware, and utilization rate of the video memory of the first GPU hardware. If the value of the computing power indicator reaches the preset threshold, it is determined that the computing power resources of the first GPU hardware occupied by the first type of computing task have reached the preset threshold.

16. The method of claim 15, wherein the computing node supports virtualizing the computing resources of its onboard GPU hardware into a virtual GPU; the virtual GPU includes a first type of virtual GPU for performing the first type of computing tasks and a second type of virtual GPU for performing the second type of computing tasks; The method further includes: In response to this computing node joining the computing cluster, the initial values ​​of the first and second remaining computing power corresponding to the GPU hardware on this computing node are reported to the scheduler for maintenance locally by the scheduler; wherein, the first remaining computing power represents the number of first-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware; the second remaining computing power represents the number of second-type virtual GPUs that can be created based on the remaining computing power resources of the GPU hardware; or... The system obtains the number of first-type or second-type virtual GPUs occupied by the first-type or second-type computing tasks scheduled by the scheduler to run on the GPU hardware mounted on this computing node. Based on the obtained number of first-type or second-type virtual GPUs, the system further updates the initial values ​​of the first and second remaining computing power corresponding to the GPU hardware mounted on this computing node, and reports the updated first and second remaining computing power to the scheduler in real time so that the scheduler can maintain them locally.

17. The method of claim 14, wherein the second GPU hardware includes GPU hardware on the first computing node whose remaining computing resources, other than the first GPU hardware, meet the computing power requirements of the second type of computing task; or, the second GPU hardware includes GPU hardware on other computing nodes in the computing cluster whose remaining computing resources, other than the first computing node, meet the computing power requirements of the second type of computing task.

18. The method of claim 14, wherein the computing cluster is a Kubernetes cluster; the computing node supports the mixed deployment of multiple containers for running different types of computing tasks on the same GPU hardware; and the computing task is a computing task running in the container.

19. The method of claim 18, wherein the first type of computing task is a computing task run by a container with a QoS service level of Guaranteed; and the second type of computing task is a computing task run by a container with a QoS service level of BestEffort.

20. A scheduling device for GPU computing resources, applied to a scheduler corresponding to a computing cluster; wherein, At least some of the computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; The service level of the first type of computing task is higher than that of the second type of computing task; the device includes: The first determining module, in response to a target computing task created in the computing cluster, determines the task type of the target computing task; The scheduling module, if the target computing task is a first type of computing task, schedules the target computing task to the first GPU hardware in the computing cluster for execution; wherein, the first GPU hardware is GPU hardware with remaining computing power resources sufficient to meet the computing power requirements of the target computing task; in response to a first indication reported by a first computing node, reschedules the second type of computing task already scheduled to run on the first GPU hardware to the second GPU hardware in the computing cluster for execution; wherein, the first computing node is a computing node equipped with the first GPU hardware; the first indication is used to indicate that the first type of computing task will exclusively occupy the computing power resources of the first GPU hardware; the first indication is reported by the first computing node to the scheduler when the computing power resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold; the second GPU hardware is GPU hardware with remaining computing power resources sufficient to meet the computing power requirements of the second type of computing task.

21. A GPU computing resource scheduling device, applied to any computing node in a computing cluster; wherein, At least some of the computing nodes in the computing cluster are equipped with at least one GPU hardware; the computing nodes support running multiple types of computing tasks on the same GPU hardware; the multiple types of computing tasks include a first type of computing task and a second type of computing task; The service level of the first type of computing task is higher than that of the second type of computing task; the device includes: The running module, in response to the scheduler corresponding to the computing cluster, schedules the first type of computing tasks to the first GPU hardware mounted on the computing node, and runs the first type of computing tasks on the first GPU hardware; wherein, the first type of computing tasks are scheduled to the computing node by the scheduler when it is determined that the remaining computing power resources of the first GPU hardware meet the computing power requirements of the first type of computing tasks; The second determining module determines whether the computing resources of the first GPU hardware occupied by the first type of computing task have reached a preset threshold. The reporting module, if the computing resources of the first GPU hardware occupied by the first type of computing task reach a preset threshold, reports a first indication to the scheduler, wherein the first indication is used to indicate that the first type of computing task will exclusively occupy the computing resources of the first GPU hardware, so that the scheduler will reschedule the second type of computing task that has been scheduled to run on the first GPU hardware to run on the second GPU hardware in the computing cluster; the second GPU hardware is GPU hardware whose remaining computing resources meet the computing power requirements of the second type of computing task.

22. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 19.

23. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 19.

24. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method of any one of claims 1 to 19.

Images