GPU scheduling method and device, equipment and storage medium

Abstract

The invention discloses a GPU scheduling method, device and equipment and a storage medium, and relates to the technical field of financial science and technology, and the method comprises the steps: responding to a GPU resource request of a to-be-scheduled container, and determining GPU demand information of the to-be-scheduled container; determining candidate nodes based on the GPU demand information, and taking the GPUs with the residual available capacity on the candidate nodes as candidate GPUs; determining a target GPU from the candidate GPUs based on the GPU demand information; and deploying the to-be-scheduled container to the target node where the target GPU is located, and mounting the target GPU to the to-be-scheduled container. According to the invention, the utilization rate of GPU resources can be improved.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a GPU scheduling method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of fields such as artificial intelligence, deep learning, and high-performance computing, GPUs (Graphics Processing Units) have become a core resource supporting complex computing tasks in cloud computing environments due to their superior parallel computing capabilities.

[0003] Currently, Kubernetes (K8s) is widely used as a mainstream container orchestration and resource scheduling platform for the management and allocation of GPU resources. Specifically, K8s uses a static allocation mechanism for GPU resource scheduling. Its core feature is that the entire physical GPU card is used as the smallest scheduling unit, and it is exclusively allocated to a single container (Pod). However, this GPU resource allocation method results in low GPU resource utilization.

[0004] Therefore, improving the utilization rate of GPU resources is an urgent problem that needs to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a GPU scheduling method, apparatus, device and storage medium, which aims to improve the utilization of GPU resources.

[0006] To achieve the above objectives, this application provides a GPU scheduling method, the GPU scheduling method comprising: In response to a GPU resource request from a container to be scheduled, determine the GPU requirement information of the container to be scheduled; Based on the GPU demand information, candidate nodes are determined, and GPUs with remaining available capacity on each candidate node are selected as candidate GPUs. The target GPU is determined from each of the candidate GPUs based on the GPU requirement information; The container to be scheduled is deployed to the target node where the target GPU is located, and the target GPU is mounted to the container to be scheduled.

[0007] In one embodiment, the GPU demand information includes a first GPU capacity requirement, and the step of determining the target GPU from the candidate GPUs based on the GPU demand information includes: Based on the remaining available capacity of each of the candidate GPUs, the GPU that meets the capacity requirement of the first GPU among the candidate GPUs is selected as the first GPU. Identify the non-idle second GPUs among the first GPUs and determine whether the second GPUs meet the GPU sharing requirements of the container to be scheduled. If the conditions are met, then the second GPU will be used as the target GPU; If the conditions are not met, then a third GPU other than the second GPU is determined from each of the first GPUs, and the target GPU is determined from each of the third GPUs.

[0008] In one embodiment, the GPU demand information further includes a first virtualization mode and a first affinity label, wherein the first virtualization mode represents the priority requirement of the scheduled container for GPU resource utilization, and the first affinity label represents the business line affiliation of the scheduled container. The step of determining whether the second GPU meets the GPU sharing requirements of the container to be scheduled includes: Determine the second virtualization mode and second affinity tag of the container already mounted in the second GPU; If the first virtualization mode and the second virtualization mode are the same, and the first affinity label is the same as the second affinity label, then the second GPU is determined to meet the GPU sharing requirements of the container to be scheduled.

[0009] In one embodiment, the GPU demand information further includes a first service identifier, which represents the business service affiliation of the container to be scheduled; The step of determining that the second GPU meets the GPU sharing requirements of the container to be scheduled includes: When the first virtualization mode is virtual GPU mode, determine whether the first service identifier and the second service identifier of the mounted container are consistent; If the first service identifier and the second service identifier are inconsistent, it is determined that the second GPU meets the GPU sharing requirements of the container to be scheduled.

[0010] In one embodiment, the GPU demand information further includes a first user identifier, and the step of determining that the second GPU meets the GPU sharing requirements of the container to be scheduled includes: When the first virtualization mode is a multi-process service mode, determine whether the first user identifier and the second user identifier of the mounted container are consistent. If the first user identifier and the second user identifier are the same, it is determined that the second GPU meets the GPU sharing requirements of the container to be scheduled.

[0011] In one embodiment, the step of determining candidate nodes based on the GPU demand information includes: Obtain the first schedulable node in the cluster to which the container to be scheduled belongs, and determine the total remaining GPU capacity of each first node; Based on the total remaining GPU capacity, the nodes in each of the first nodes that meet the first GPU capacity requirements are selected as candidate nodes.

[0012] In one embodiment, the step of mounting the target GPU to the scheduling container includes: When the first virtualization mode is virtual GPU mode, the resource isolation control library and limiting parameters of the target GPU are mounted to the scheduling container through the first component on the target node. The resource isolation control library is used to isolate and manage the resources of the target GPU, and the limiting parameters include maximum video memory usage and maximum computing power share. When the first virtualization mode is a multi-process service mode, the process communication pipeline directory and shared memory space of the target GPU are mounted to the container to be scheduled through the second component on the target node. The process communication pipeline directory includes a communication interface between the container to be scheduled and the target node, and the shared memory space is used to store temporary computing data.

[0013] Furthermore, to achieve the above objectives, this application also provides a GPU scheduling device, the GPU scheduling device comprising: The determination module is used to determine the GPU requirement information of the container to be scheduled in response to the GPU resource request of the container to be scheduled. The node scheduling module is used to determine candidate nodes based on the GPU demand information, and to select GPUs with remaining available capacity on each candidate node as candidate GPUs. The resource scheduling module is used to determine the target GPU from each of the candidate GPUs based on the GPU demand information; The scheduling module is used to deploy the container to be scheduled to the target node where the target GPU is located, and to mount the target GPU to the container to be scheduled.

[0014] In addition, to achieve the above objectives, this application also proposes a GPU scheduling device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the GPU scheduling method as described above.

[0015] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, on which a program implementing a GPU scheduling method is stored, and the program implementing the GPU scheduling method is executed by a processor to implement the steps of the GPU scheduling method as described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the GPU scheduling method described above.

[0017] This application provides a GPU scheduling method. In response to a GPU resource request issued by a container to be scheduled, the method determines the GPU requirement information of the container to be scheduled. Then, based on the GPU requirement information, candidate nodes are determined, and GPUs with remaining available capacity on each candidate node are selected as candidate GPUs. Based on the GPU requirement information, a target GPU is determined from the candidate GPUs. Finally, the container to be scheduled is deployed to the target node where the target GPU is located, and the target GPU is mounted to the container to be scheduled for use by the container.

[0018] In summary, this application determines candidate nodes based on the GPU demand information corresponding to GPU resource requests. Then, it selects GPUs with remaining available capacity on each candidate node as candidate GPUs. This is equivalent to using idle GPUs and non-idle GPUs with remaining available capacity as candidate GPUs. Finally, it identifies the target GPUs that can be allocated to the container to be scheduled from the candidate GPUs and determines the corresponding target nodes. In this way, this application avoids the problem of traditional Kubernetes using the entire physical GPU card as the smallest scheduling unit. Once a GPU card is allocated to a container, it no longer participates in subsequent resource scheduling, leading to the waste of remaining resources on that GPU card, thus improving the utilization rate of GPU resources. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the GPU scheduling method of this application; Figure 2This is a schematic diagram of the system architecture involved in an embodiment of the GPU scheduling method of this application; Figure 3 This is a schematic diagram of conventional resource scheduling involved in an embodiment of the GPU scheduling method of this application; Figure 4 This is a schematic diagram of the WCS scheduling process involved in an embodiment of the GPU scheduling method of this application; Figure 5 This is a schematic diagram of the overall GPU scheduling process according to an embodiment of the GPU scheduling method of this application; Figure 6 This is a schematic diagram illustrating the virtualization mode involved in an embodiment of the GPU scheduling method of this application; Figure 7 This is a schematic diagram of the module structure of the GPU scheduling device in this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the GPU scheduling method in this application embodiment.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] Currently, Kubernetes (K8s) is widely used as a mainstream container orchestration and resource scheduling platform for the management and allocation of GPU resources. Specifically, K8s uses a static allocation mechanism for GPU resource scheduling. Its core feature is that the entire physical GPU card is used as the smallest scheduling unit, and it is exclusively allocated to a single container (Pod). However, this GPU resource allocation method results in low GPU resource utilization.

[0026] Therefore, improving the utilization rate of GPU resources is an urgent problem that needs to be solved.

[0027] The main solution of this application is as follows: in response to a GPU resource request from a container to be scheduled, determine the GPU requirement information of the container to be scheduled; determine candidate nodes based on the GPU requirement information, and select GPUs with remaining available capacity on each candidate node as candidate GPUs; determine the target GPU from each candidate GPU based on the GPU requirement information; deploy the container to be scheduled to the target node where the target GPU is located, and mount the target GPU to the container to be scheduled.

[0028] This application determines candidate nodes based on GPU demand information corresponding to GPU resource requests. Then, it selects GPUs with remaining available capacity on each candidate node as candidate GPUs. This is equivalent to using idle GPUs and non-idle GPUs with remaining available capacity as candidate GPUs. Finally, it identifies the target GPUs that can be allocated to the container to be scheduled from the candidate GPUs and determines the corresponding target nodes. In this way, this application avoids the problem of traditional Kubernetes using the entire physical GPU card as the smallest scheduling unit. Once a GPU card is allocated to a container, it no longer participates in subsequent resource scheduling, leading to the waste of remaining resources on that GPU card, thus improving the utilization rate of GPU resources.

[0029] It should be noted that the execution subject of the GPU scheduling method in various embodiments of this application can be a GPU scheduling system, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a GPU scheduling device capable of implementing the above functions. This embodiment does not specifically limit this. The following uses a GPU scheduling system as the execution subject as an example to describe this embodiment and the following embodiments.

[0030] First, let's explain the terms and abbreviations mentioned below: WCS (Webank Common Container Platform): Webank's self-developed general container management platform, based on the Kubernetes architecture, supporting cross-cluster container resource scheduling, deployment, and operation and maintenance; AIEP (AI Engineering Platform): Webank's full-process engineering support platform for AI business, covering model development, training, deployment, and operation and maintenance, providing standardized toolchains and resource management capabilities; MLSS (Machine Learning Service Suite): A core submodule of AIEP, focusing on the development and online services of machine learning models, providing model encapsulation, deployment scheduling, monitoring, and other functions, and serving as a key entry point for user interaction with underlying resources; VGPU (Virtual Graphics Processing Unit): One of the GPU virtualization modes, implemented based on user-space CUDA (Compute Unified Device Architecture) hijacking technology, without relying on hardware virtualization, and performing fine-grained control of GPU resources by intercepting container CUDA calls; MPS (Multi-Process Service): One of the GPU virtualization modes, using CUDA... Context fusion enables multi-process sharing of GPU resources; gpu-core, the unit of GPU resource quantification, is used to measure the scale of virtual GPU resources requested by a container, where "100 vgpu-core" corresponds to one complete physical GPU card, and supports fractional partitioning (e.g., 50 vgpu-core represents 0.5 physical cards); Pod, the smallest unit of deployment and scheduling in Kubernetes, is not a single container, but a collection of one or more closely related containers, logically regarded as an "application instance" that work together to complete specific business functions; Based on this, this application proposes a GPU scheduling method according to a first embodiment, please refer to... Figure 1 The GPU scheduling method includes steps S10 to S40: Step S10: In response to the GPU resource request of the container to be scheduled, determine the GPU requirement information of the container to be scheduled; It's important to note that a container awaiting scheduling refers to a container instance (typically corresponding to a business container within a Pod) that has been submitted to the container orchestration platform and is waiting for GPU resources to be allocated for startup. This container relies on the GPU to perform computational tasks (such as AI inference and model training). A GPU resource request is the request initiated by the container awaiting scheduling to the container orchestration platform for GPU resources; it is the core signal that triggers the scheduling process. Parsing the GPU resource request yields the GPU requirement information of the container awaiting scheduling. This GPU requirement information is a multi-dimensional feature set used to match GPU resources, which is then used for subsequent node and GPU selection.

[0031] Step S20: Based on the GPU demand information, determine candidate nodes and select GPUs with remaining available capacity on each candidate node as candidate GPUs; It should be noted that candidate nodes refer to physical nodes selected by the container orchestration system (such as WCS WeBank General Container Platform) that meet the basic resource requirements of the containers to be scheduled and have available GPU resources. Remaining available capacity refers to the amount of virtual GPU resources remaining on a single physical GPU card after deducting the vgpu-core share already allocated to other containers (the total capacity of a single physical GPU card is 100 vgpu-cores, i.e., 1 unit). For example, if a GPU card has been allocated 0.5 units, the remaining available capacity is 0.5 units.

[0032] Based on GPU demand information, candidate nodes that meet the requirements of the containers to be scheduled are identified, and GPUs with remaining available capacity on each candidate node are selected as candidate GPUs.

[0033] In this embodiment, step S20 may include: Step S201: Obtain the first schedulable node in the cluster to which the container to be scheduled belongs, and determine the total remaining GPU capacity of each first node. Step S202: Based on the total remaining GPU capacity, select the nodes in each of the first nodes that meet the first GPU capacity requirements as candidate nodes.

[0034] It should be noted that the cluster to which the container to be scheduled belongs refers to the container cluster pre-associated with the container to be scheduled (such as an AI inference model container) for deployment and operation. In this embodiment, the cluster specifically refers to the cross-K8s online cluster managed by the WCS WeBank General Container Platform. The cluster contains multiple nodes equipped with physical GPU cards and supports both VGPU / MPS virtualization modes. Scheduled nodes refer to physical nodes in the cluster that meet the basic resource requirements (CPU, memory, disk, network IP, etc.) of the container to be scheduled and are not marked as "unschedulable" (e.g., node failure, maintenance). These nodes have completed GPU card management (recording physical GPU card ID, total capacity, etc.) through WCS-RM (ResourceManager), and possess the basic conditions to participate in GPU resource scheduling. The total remaining GPU capacity of a node refers to the sum of the remaining available virtual GPU resources of all physical GPU cards on the node.

[0035] The schedulable nodes in the cluster containing the container to be scheduled are referred to as first nodes for distinction. The total remaining GPU capacity of each first node is determined. Based on the total remaining GPU capacity, the nodes in each first node that meet the capacity requirements of the container to be scheduled (hereinafter referred to as first GPU capacity requirements for distinction) are determined as candidate nodes.

[0036] Step S30: Determine the target GPU from each of the candidate GPUs based on the GPU requirement information; Based on GPU demand information, GPUs that meet the GPU resource requirements of the container to be scheduled are selected from the candidate GPUs (hereinafter referred to as target GPUs for distinction).

[0037] Step S40: Deploy the container to be scheduled to the target node where the target GPU is located, and mount the target GPU to the container to be scheduled.

[0038] After identifying the target GPU, the candidate node where the target GPU is located is called the target node. Then, the container to be scheduled is deployed to the target node, and the target GPU is mounted to the container to be scheduled so that the container can use the GPU resources.

[0039] In one feasible implementation, such as Figure 2The diagram shows the system architecture. Starting with the AIEP user, after submitting an AI model service deployment request through the MLSS-MF (Model Factory) module, MLSS-MF first requests container resources from WCS-ADMIN. Then, WCS-ADMIN's resource manager, WCS-RM, performs container allocation and scheduling, synchronizing GPU resource information with the vgpu-extender component in the WCS K8s online cluster (an online cluster of the WeBank general container platform built on Kubernetes). Next, MLSS-MF instructs the deployment service WCS-DEPLOY to perform container deployment. WCS-DEPLOY creates a model service Pod on the target GPU node through kube-apiserver. Finally, on the target GPU node, the vgpu-device-plugin provides virtual GPU resources for the Pod, and the mps-daemon component starts to provide multi-process service support. Ultimately, the model service Pod successfully starts and runs, completing the AI ​​service deployment.

[0040] It should be understood that when each GPU device is partitioned by percentage, the sufficiency of a single-node GPU resource cannot be determined directly by addition or subtraction of numbers. For example, ... Figure 3 The diagram illustrates traditional resource scheduling. In a two-card node, the two cards occupy 0.7 and 0.8 cards respectively. Directly calculating the remaining 0.5 card resources reveals that the actual remaining resources are 0.2 and 0.3 cards respectively, which cannot meet the needs of containers requiring 0.5 cards. Therefore, shared GPU scheduling requires two layers: node scheduling and card scheduling. When scheduling is complete, in addition to Kubernetes' usual allocation of a suitable Node (node, server) to the Pod, it also allocates a specific GPU device on the target node to the Pod and updates the device number in the annotation. After the Pod is scheduled to the node, the corresponding device-plugin reads the card information in the annotation and performs device mounting. Each Pod is assigned a unique card ID, and multiple card IDs are written for multiple cards. The device-plugin then mounts the GPU card into the container for its use based on the card ID.

[0041] Furthermore, Pod scheduling in a WCS cluster does not follow the conventional Kubernetes scheduling model. Creating a Pod in a WCS cluster involves two steps, specifically, as follows: Figure 4The diagram illustrates the WCS scheduling process. WCS Pod creation involves two steps: container application (scheduling) and container deployment (creation). During the application phase, the user fills in resource and target cluster requirements. The WCS backend calculates and filters nodes that meet the CPU, memory, disk, and network IP requirements, recording the mapping between Pods and Nodes. During deployment, pre-allocated nodes are retrieved directly from the database, and the Pod is created in the Kubernetes cluster with `nodeSelector:hostname=node_xxx` configured. However, directly using this scheduling logic cannot meet the needs of shared GPU scheduling. First, its resource judgment relies solely on simple numerical subtraction, lacking higher-order scheduling capabilities such as GPU affinity. Second, the completed container only allocates GPU resources without recording the specific card ID, failing to adapt to the fine-grained scheduling requirements of shared GPUs. Therefore, to meet the requirement of shared GPU Pod deployment in the WCS cluster and minimize intrusion into the WCS scheduling logic code, a component similar to the Kubernetes scheduler extender is proposed and deployed into the cluster, providing an HTTP interface. Figure 5 The diagram illustrates the overall GPU scheduling process. WCS first uses a pre-selection algorithm, such as basic resource deduction, to filter out a list of Nodes that meet the requirements for CPU, memory, etc. Then, it sends a request to the extender component deployed in the cluster and providing an HTTP interface, passing in the resource information requested by the Pod (e.g., 0.5 GPUs), the service identifier for distributed deployment (subsystem + DCN, restricting the same identifier from being scheduled to the same GPU card), the pre-selected list of Nodes, and the list of Pods that have requested GPUs in the corresponding Nodes (including Pod information pre-allocated by WCS but not yet created). Based on this information, the extender component constructs a scheduling view in conjunction with the current status of Nodes and Pods in the cluster, and uses vgpu scheduling logic to filter out suitable target nodes and their corresponding GPU IDs and return them to WCS. Finally, WCS records the target node and GPU ID in the database, completing the container request.

[0042] Thus, this embodiment determines candidate nodes based on the GPU demand information corresponding to the GPU resource request. Then, GPUs with remaining available capacity on each candidate node are selected as candidate GPUs. This is equivalent to using idle GPUs and non-idle GPUs with remaining available capacity as candidate GPUs. Finally, the target GPU that can be allocated to the container to be scheduled is determined from the candidate GPUs, and the corresponding target node is identified. In this way, this embodiment avoids the problem of traditional Kubernetes using the entire physical GPU card as the smallest scheduling unit. Once a GPU card is allocated to a container, it no longer participates in subsequent resource scheduling, leading to the waste of remaining resources on that GPU card, thus improving the utilization rate of GPU resources.

[0043] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the GPU requirement information includes a first GPU capacity requirement, and step S30 may include: Step S301: Based on the remaining available capacity of each of the candidate GPUs, the GPU that meets the capacity requirement of the first GPU among the candidate GPUs is selected as the first GPU. It should be noted that the GPU resource requirements of the containers to be scheduled are referred to as the first GPU capacity requirement for distinction. For example, "vgpu-core" is used as the core unit (100 vgpu-core corresponds to 1 full card), and fractional granularity is supported (e.g., 50 vgpu-core corresponds to 0.5 cards), which is the basic quantitative standard for screening GPU resources.

[0044] The remaining available capacity of each candidate GPU is determined. Based on the remaining available capacity, it is determined whether each candidate GPU meets the first GPU capacity requirement. The candidate GPU that meets the first GPU capacity requirement is designated as the first GPU for distinction. Specifically, if the remaining available capacity of a candidate GPU is greater than or equal to the amount of GPU resources represented by the first GPU capacity requirement, then the candidate GPU is determined to meet the first GPU capacity requirement.

[0045] Step S302: Determine the non-idle second GPU among the first GPUs, and determine whether the second GPU meets the GPU sharing requirements of the container to be scheduled; It should be noted that a non-idle GPU refers to a physical GPU card that has at least one container mounted on it.

[0046] Identify the non-idle GPUs in each of the first GPUs (hereinafter referred to as the second GPUs for distinction), and determine whether the second GPUs meet the GPU sharing requirements of the container to be scheduled. It should be understood that when there are multiple second GPUs, the GPU sharing requirements are determined sequentially for each second GPU.

[0047] Thus, this embodiment of the application maximizes the activation of GPU fragment resources by prioritizing the selection of available resources from non-idle second GPUs, thereby avoiding the long-term idle capacity of non-idle GPU cards due to "not being prioritized for scheduling".

[0048] In this embodiment, the GPU demand information further includes a first virtualization mode and a first affinity label. The first virtualization mode represents the priority requirement of the container to be scheduled for GPU resource utilization, and the first affinity label represents the business line affiliation of the container to be scheduled. Step S302 may include: Step A10: Determine the second virtualization mode and second affinity tag of the container already mounted in the second GPU; It's important to note that a container's virtualization mode refers to the priority selection of the container's GPU resource utilization. The container's virtualization mode can be VGPU mode or MPS mode. VGPU mode is a GPU virtualization technology based on user-space CUDA hijacking, suitable for scenarios requiring business isolation. When the container's virtualization mode is VGPU, it means that the container's priority in GPU resource utilization is isolation. MPS mode reduces switching overhead through CUDAContext fusion and is suitable for high-concurrency inference scenarios. When the container's virtualization mode is VGPU, it means that the container's priority in GPU resource utilization is processing performance. The container's affinity tag is a predefined, unique identifier representing the container's business line affiliation. For example, the affinity tag "loan_model_A" represents the loan risk model business line, and the affinity tag "repayment_model_B" represents the repayment prediction model business line.

[0049] The virtualization mode of the container to be scheduled is referred to as the first virtualization mode for distinction, and the affinity label of the container to be scheduled is referred to as the first affinity label for distinction. The virtualization mode (hereinafter referred to as the second virtualization mode for distinction) and affinity label (hereinafter referred to as the second affinity label for distinction) of the containers already mounted in the second GPU are determined.

[0050] In the first feasible implementation, the virtualization mode selected by the container to be allocated can be a single-container bare hybrid mode. Among them, the model bare hybrid mode of full-card scheduling (i.e., the above-mentioned single-container bare hybrid mode) does not require additional modification and adaptation at the scheduling layer, and continues the deployment method of single container exclusively occupying the entire GPU card. The core difference is that it supports a single container to load multiple AI models on the same GPU at the same time. The sharing of GPU resources by multiple models needs to be adapted by the business side. There are no isolation restrictions on computing power and video memory at the GPU hardware level (i.e., "bare hybrid"). Its sharing principle directly relies on NVIDIA GPU time slice segmentation technology. The CUDAContext of different models will freely preempt and dynamically switch with each other during the operation. This mode is applicable in three main scenarios: First, resource sharing scenarios involving more than one GPU card, such as deploying a smaller model requiring only 0.5 GPU cards in a GPU cluster that already has two full-card models deployed. Second, scenarios where the business side has already completed technical modifications, meaning other businesses have achieved the ability to deploy multiple models simultaneously by modifying their code frameworks. Third, scenarios where models are related to business operations, as multiple models are deployed in the same container, and maintenance operations (such as image creation, version release, and online / offline updates) need to be performed synchronously. Furthermore, if the container exits due to external failures, all models will be affected simultaneously. From a characteristic perspective, the core advantage of this mode is its high flexibility. Businesses can independently define the specific form of multi-model co-deployment, and businesses that have already achieved co-deployment can directly containerize and deploy without additional adaptation. However, there are two risks: First, due to the lack of a memory and computing power isolation mechanism, a single model may consume a large amount of resources due to sudden loads, causing other models to fail to operate normally due to insufficient resources. Second, multiple models sharing a single container increases operational complexity, as all models need to simultaneously complete image building, release updates, and online / offline operations, resulting in higher operational costs.

[0051] In the second feasible implementation, the virtualization mode selected by the container to be allocated can be VGPU. VGPU is an isolated priority mode for shared card scheduling. Its underlying GPU sharing principle is consistent with the single-container bare hybrid mode, both based on time-division multiplexing technology. Multiple CUDA Context switching occurs when multiple containers access concurrently. However, compared to the bare hybrid mode, this mode adds two key optimizations: First, it supports isolated deployment of multiple containers, with each container running independently and managed separately for on / off maintenance, eliminating the need to deploy multiple models in the same container, and allowing for adaptation without business-side modifications. Second, it achieves precise limiting of GPU memory and computing power. In time-division multiplexing scenarios, CUDA hijacking technology is used to control container-level GPU memory usage and computing power usage, preventing a single container from occupying a large amount of resources for an extended period, thus preventing other shared containers from becoming unavailable. The applicable scenarios for this mode are clearly defined: on the one hand, it is suitable for mixed deployments of models with low request volume, low concurrency performance requirements, and low latency sensitivity, where the GPU memory and computing power limiting mechanism ensures stable resource allocation; on the other hand, it is suitable for bypass models, which are not core business links and have low real-time requirements, allowing for efficient utilization of GPU fragment resources without affecting core business operations. Finally, in terms of characteristics, its core advantage is strong isolation. Through hard isolation of GPU memory and computing power, it completely eliminates the risk of a single model excessively consuming resources and causing other models to become unusable. However, there are also three risks: First, isolation will lead to performance loss, with inference performance slightly worse than the non-isolated mode; second, when multiple containers compete for computing power, inference latency spikes are prone to occur; and third, the CUDA hijacking library it relies on is implemented by a third party, which may have some CUDA API incompatibility issues. Therefore, when adopting this mode, performance adaptation needs to be a key focus. Before the model goes live, it must undergo rigorous stress testing to verify whether the inference latency, latency spikes, and other indicators meet business requirements.

[0052] In the third feasible implementation, the virtualization mode selected by the container to be configured can be MPS. The core advantage of MPS lies in its ability to optimize request latency and improve inference throughput in high-concurrency inference scenarios; however, it should be noted that MPS cannot provide performance improvements if the service request volume is small or concurrency is low. Its applicable scenarios include three categories: First, deployment of multiple replicas of the same model, such as converting two 1-GPU instances into two 0.5-GPU instances to save resources, and ensuring that CUDA Context failures do not affect other services (subject to new high availability rules); second, bypass models, which are non-core links and can efficiently utilize resources when adapted to MPS; and third, offline batch task scenarios. For example, in non-MPS mode, different tasks in multiple Streams are executed serially in a time-division multiplexing manner, with obvious CUDA Context switching and dispersed task execution timelines with intervals. However, in MPS mode, different tasks in multiple Streams can be executed in parallel, with tight timelines and high concurrency, achieving spatial multiplexing. From a performance perspective, MPS's core advantage lies in its strong performance. In high-concurrency scenarios, it reduces inference latency and GPU memory usage by integrating CUDAContext. However, it also presents two risks: firstly, fault propagation—after CUDAContext integration, an anomaly in a single Context can lead to errors in multiple concurrent requests; secondly, process UID (UserIdentifier) ​​restrictions—MPS mode requires that only processes with the same UID can share resources. Furthermore, adopting this mode requires close attention to fault impact and recovery mechanisms: Before model deployment, rigorous load testing is necessary to verify the impact of single container exits, restarts, or forced termination on the other model service in scenarios with two models deployed concurrently; downstream callers of the inference service must implement error retry mechanisms. If the service experiences a prolonged unrecoverable interruption due to MPS, an initial forced restart of nvidia-mps-server can be performed. If recovery fails, Pod reconstruction via configuration is supported, allowing CUDA requests to bypass the MPS layer and degrade to a multi-container hybrid mode.

[0053] It should be noted that, due to the adoption of a unified scheduler scheme, the usage of full-card scheduling and the two types of shared card scheduling (isolation priority and performance priority) are not significantly different, and it supports the simultaneous deployment of full-card containers and shared card containers on the same node without conflict. Specifically, for full-card applications, vgpu-core:100 needs to be configured in the resource field; for shared card applications (isolation priority mode), gpu-share-mode=isolation needs to be marked in the annotations field and vgpu-core:50 needs to be configured in the resource field; and for shared card applications (performance priority mode), gpu-share-mode=performance needs to be marked in the annotations field and vgpu-core:50 needs to be configured in the resource field.

[0054] For example, such as Figure 6 The diagram shown illustrates the virtualization mode, in which... Figure 6 In diagram (a), a single-container bare-metal hybrid deployment mode is used. In this mode, two models, Model A and Model B, are deployed simultaneously within a single container, sharing the same GPU-θ resource. Each model maintains its own independent CUDA Context and executes tasks alternately on the GPU through time-division multiplexing. This results in frequent CUDA Context switching during execution, which is a bare-metal hybrid deployment mode without isolation. Figure 6 In Figure (b), the VGPU mode is used. In this mode, Model A and Model B are deployed in separate containers. Each container uses the libvgpu.so library to hijack and isolate GPU resources. The two containers share the same GPU-θ resource and use the GPU alternately in a time-division multiplexing manner. CUDA Context switching still occurs during execution, but container-level isolation of GPU memory and computing power is achieved through vGPU technology, avoiding service unavailability issues caused by resource contention. Figure 6 (c) represents the MPS mode. In this mode, Model A and Model B are deployed in separate containers. Each container maintains its own CUDA Context and interacts with GPU-θ through the NVIDIA MPS Server (Multi-Process Service). The MPS Server merges the CUDA Contexts of multiple containers into a single underlying context, enabling spatial multiplexing of GPU resources. Multiple tasks can be executed in parallel on the GPU, significantly improving inference throughput and performance in high-concurrency scenarios.

[0055] In this embodiment, the GPU scheduling system includes a GPU pod management module, which is completed collaboratively by multiple core components, forming a complete link from resource awareness and management to scheduling and deployment. The core is centered around the WCS-RM (Resource Management System). WCS-RM primarily manages Pod container resources and coordinates GPU information. On one hand, it manages all resources of the Pod container, storing Pod metadata including CPU, memory, GPU, and node information. On the other hand, it retrieves detailed information about NVIDIA GPUs from Kubernetes and stores it in the host_info table. For GPU physical machines, it can automatically detect all GPU cards and assign them numbers, fully supporting single-machine multi-card deployment modes. A common configuration is 8 cards per physical machine. Each card is assigned a unique identifier, such as "GPU-4f37297a-de5f-6006-a11d-d84c02a7bad2d," ensuring the uniqueness of resource traceability. Simultaneously, WCS-RM provides flexible support for GPU sharing among Pods, with built-in MPS and VGPU virtualization modes for businesses to choose from as needed.

[0056] Around the core functions of WCS-RM, several specialized components provide technical support: VGPU-Extender (WCS Virtualized GPU Container Scheduling Extension), as the core of scheduling decisions, provides WCS-RM with a precise scheduling decision module at the Pod and Card levels, ensuring the rationality of resource allocation; vgpu-device-plugin (Virtualized GPU Device Plugin) exists as a device plugin for the kubelet on the K8s node, undertaking the connection between GPU devices and Pod containers, responsible for accurately mounting the selected GPU device to the target Pod container; mps-control-daemon (MPS Node Daemon) is primarily responsible for deploying and starting nvidia-mps-server on the node, providing the basic environment support for the normal operation of MPS mode. Through the division of labor and cooperation of these components, the entire process of GPU card management, from awareness and management to scheduling and deployment, is achieved.

[0057] Step A20: If the first virtualization mode and the second virtualization mode are the same, and the first affinity label and the second affinity label are the same, determine that the second GPU meets the GPU sharing requirements of the container to be scheduled.

[0058] Determine whether the first virtualization mode and the second virtualization mode are consistent, and determine whether the first affinity label and the second affinity label are consistent. If the first virtualization mode and the second virtualization mode are the same, and the first affinity label and the second affinity label are the same, determine that the second GPU meets the GPU sharing requirements of the container to be scheduled, that is, it allows the mounted container and the container to be scheduled to share a GPU card.

[0059] Step S303: If the conditions are met, then the second GPU is used as the target GPU; If the second GPU meets the GPU sharing requirements of the container to be scheduled, the second GPU will be used as the target GPU.

[0060] Understandably, the GPU scheduling in this application needs to meet the following three core requirements: First, because the WCS container platform adopts a pre-allocated container resource mode and spans multiple K8s clusters, containers and nodes are bound together and cannot directly use the K8s scheduler. Therefore, vgpu scheduling capabilities need to be integrated into the WCS allocation logic. Second, the scheduling of virtualized GPUs needs to support differentiated strategies, ensuring both affinity scheduling between different services to optimize resource aggregation and distributed scheduling of different instances of the same service on GPUs to avoid the risk of fault propagation. Third, both VGPU and MPS computing power isolation modes need to be provided for business selection to adapt to the isolation or performance requirements of different scenarios.

[0061] Step S304: If the condition is not met, then determine the third GPU in each of the first GPUs other than the second GPU, and determine the target GPU from each of the third GPUs.

[0062] If none of the second GPUs meet the GPU sharing requirements of the container to be scheduled, the GPUs other than the second GPUs in each first GPU (hereinafter referred to as the third GPUs for distinction) are identified. It can be understood that the third GPUs are idle cards; the target GPU is then determined from each third GPU.

[0063] In one feasible implementation, the step of determining the target GPU from each of the third GPUs includes: selecting any one of the third GPUs as the target GPU. In another feasible implementation, the step of determining the target GPU from each of the third GPUs includes: determining the GPU ranked first in the list of idle cards composed of the third GPUs as the target GPU, wherein the list of idle cards is generated during the traversal of each first node.

[0064] For example, in a shared card scenario where the GPU request volume does not exceed 1, the core objective of the scheduling algorithm is to select suitable target nodes and GPU cards for the Pod to be scheduled. The input information includes a list of candidate nodes and parameters of the Pod to be scheduled (such as virtualization mode, affinity tag, etc.). The algorithm execution starts with a dual traversal of nodes and GPU cards, and performs multiple rounds of verification for each physical GPU card of each node. First, resource verification is performed. If the remaining resources of the card cannot meet the GPU request volume of the Pod to be scheduled (for example, a card that has been allocated 0.7 units of resources cannot handle a new 0.5 unit request), the card is skipped directly. Next, virtualization mode verification is performed. If the VGPU or MPS mode of the container already on the card is inconsistent with that of the Pod to be scheduled, the card will be excluded. The process then proceeds to the affinity matching stage. If the card is not idle (already has a container running), its `gpu-affinity` tag (i.e., affinity tag) must be checked to ensure it fully matches the affinity tag of the Pod to be scheduled. If the Pod to be scheduled uses VGPU mode, it is also necessary to verify whether the card already has a Pod serving the same service to avoid violating the principle of distributed scheduling within the same service. If all conditions are met, the card and its corresponding node are immediately selected and the result is returned. If the card has no containers allocated, the "node + GPU card" combination is added to the list of available resources. After the traversal is complete, the final decision stage begins: if a matching non-idle card has been found in step 1, the scheduling result is returned directly; if not, the list of available resources is checked. If an available card exists, the first item in the list is selected for scheduling; otherwise, an insufficient resource error is returned.

[0065] The matching rules between affinity tags can be found in Table 1 below: Table 1

[0066] This application embodiment achieves fine-grained resource management and allocation of virtualized GPUs through three steps: "WCS record extension → Extender precise scheduling → information injection at creation time". Specifically, it performs WCS record extension first. At the node data level, it needs to record a list of all physical GPU IDs of the vGPU node that has been enabled. At the container data level, it extends the Pod record table, adds a "GPU card number list" field to store multiple physical GPU card IDs, and a "GPU resource request amount" field to support decimal virtual GPU request amounts. It also modifies the WCS scheduling logic to determine the resource availability by subtracting the total GPU resource request amount of the allocated containers from the total GPU resources of the node. Secondly, a new Extender service is added, which provides a POST localhost:47515 / extender interface. This interface receives a request body containing GPU information for the Pod to be scheduled (e.g., request quantity of 0.5, virtualization mode VGPU / mps, affinity label, etc.), a list of pre-selected nodes (including node names and GPU IDs), and a list of containers already allocated to the node (including GPU allocation details). The scheduling logic first filters nodes that do not have vGPU scheduling enabled, then iterates through each physical GPU card on the node, checking if the sum of the allocated quantity and the new request quantity is ≤ 1, whether the affinity label is compatible, and whether the virtualization mode is compatible. Existing GPU cards that meet the conditions are reused first, and finally, the target node and GPU ID or error information are returned. Finally, during the WCS container creation phase, the vgpu resource value needs to be declared in the Pod's resources.requests field, and the allocated GPU card number list, isolation mode, and scheduling affinity label are injected into the Pod's Annotation / Label using gpu-ids, gpu-share-mode, and gpu-affinity keys, respectively.

[0067] Thus, this application's embodiment combines a dual matching mechanism of virtualization mode and business affinity tags to achieve multiple technical values: it reduces GPU resource fragmentation waste and improves single-card resource utilization by prioritizing the reuse of remaining resources of non-idle GPUs, while avoiding resource fragmentation problems caused by frequent occupation of idle GPUs; at the same time, the consistency verification of virtualization mode ensures technical compatibility when different containers share GPUs, and the affinity tag matching related to business line ownership achieves business-level resource isolation, effectively preventing contention and fault propagation caused by different business containers sharing resources, ultimately achieving a balance between "efficient resource utilization" and "stable business operation," solving the dual pain points of resource waste and chaotic sharing in traditional scheduling.

[0068] Based on the first and second embodiments of this application, in this third embodiment, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. In addition, the GPU demand information further includes a first service identifier, which represents the business service affiliation of the container to be scheduled; Step A20 may include: Step A201: When the first virtualization mode is virtual GPU mode, determine whether the first service identifier and the second service identifier of the mounted container are consistent. Step A202: If the first service identifier and the second service identifier are inconsistent, determine that the second GPU meets the GPU sharing requirements of the container to be scheduled.

[0069] It should be noted that the service identifier of a container represents the business service it belongs to. Containers from different business lines may belong to the same service. Specifically, the business service could be a real-time loan amount calculation service, a credit card bill installment interest rate calculation service, etc. Containers serving the same service but from different business lines indicate that the same business service (such as the "risk control rule engine service") is reused by different business lines. The service identifier of the containers is the same, but the business lines they belong to are different (such as serving the "personal consumer loan business line" and the "micro and small enterprise operating loan business line" respectively). Although these containers are from different business lines, they are different instances of the same business service because their service functions and resource consumption characteristics are consistent. Therefore, they must comply with the rule of "prohibiting shared GPU cards".

[0070] The service identifier of the container to be scheduled is referred to as the first service identifier. When the first virtualization mode is Virtual GPU (VGPU) mode, the priority of the container's use of GPU resources is "isolation priority," meaning the container's business requirement is stability. In this case, it is necessary to determine whether the first service identifier and the service identifier of the mounted container (hereinafter referred to as the second service identifier for distinction) are consistent. If the first and second service identifiers are consistent, it means that the container to be scheduled and the mounted container belong to the same service category, and resource contention may occur when using GPU resources, thus affecting the data processing stability of the container to be scheduled. Therefore, the container to be scheduled and the mounted container are not allowed to share a GPU card; that is, the second GPU does not meet the GPU sharing requirement of the container to be scheduled. If the first and second service identifiers are inconsistent, it is considered that the container to be scheduled and the mounted container belong to different services, and resource contention will not occur. The second GPU meets the GPU sharing requirement of the container to be scheduled.

[0071] In this embodiment, the GPU scheduling rules include an in-service distribution system. In-service distribution scheduling follows fixed default rules and does not allow user configuration: For containers using VGPU mode, the system automatically enables an in-service anti-affinity mechanism, that is, it determines the service based on the service identifier, prohibiting multiple Pods of the same business service (even if they belong to different business lines) from sharing the same physical GPU card, thereby avoiding stability issues caused by resource contention among multiple instances within the same service; For containers using MPS mode, this in-service anti-affinity rule is disabled by default, allowing multiple Pods of the same business service to share the same physical GPU card, in order to adapt to the core requirement of improving concurrency performance through CUDA Context integration in MPS mode.

[0072] In this embodiment, the GPU requirement information further includes a first user identifier, and step A20 may include: Step A203: When the first virtualization mode is a multi-process service mode, determine whether the first user identifier and the second user identifier of the mounted container are consistent. Step A204: If the first user identifier and the second user identifier are consistent, determine that the second GPU meets the GPU sharing requirements of the container to be scheduled.

[0073] It should be noted that the user identifier of a container refers to the unique identifier of the user to whom the core running process of that container belongs, i.e., the UID. The user identifier is used to distinguish processes of different users and is the core permission verification basis for implementing GPU sharing in MPS (Multi-Process Service) mode.

[0074] The user identifier of the container to be scheduled is referred to as the first user identifier for distinction. In the case that the first virtualization mode is a multi-process service mode, it is determined whether the first user identifier and the user identifier of the mounted container (hereinafter referred to as the second user identifier for distinction) are consistent. If the first user identifier and the second user identifier are consistent, the container to be scheduled and the mounted container are allowed to share a GPU card, and it is determined that the second GPU meets the GPU sharing requirements of the container to be scheduled; otherwise, the container to be scheduled and the mounted container are not allowed to share a GPU card.

[0075] Thus, this application's embodiments formulate differentiated verification rules for different virtualization modes, achieving multi-dimensional technical optimization. Specifically, in virtual GPU mode, by verifying the inconsistency between the first service identifier of the container to be scheduled and the second service identifier of the container already mounted on the second GPU, multiple instances of the same business service are forced not to share the same GPU card, effectively avoiding stability issues caused by resource contention among instances within the same service, and ensuring business reliability in isolation-first scenarios. In multi-process service mode, by verifying the consistency between the first user identifier of the container to be scheduled and the second user identifier of the container already mounted, it aligns with the technical characteristic of MPS mode that "only the same user process can achieve CUDA Context fusion," avoiding Context fusion failure caused by cross-user process sharing, and reducing switching overhead and improving inference throughput in high-concurrency scenarios through Context fusion. The combination of targeted verification rules for the two modes not only meets the differentiated needs of different businesses for isolation or performance, but also achieves refined and compliant sharing of GPU resources, balancing resource utilization and business operation stability.

[0076] In this embodiment, step S40 may include: Step S401: When the first virtualization mode is virtual GPU mode, the resource isolation control library and limiting parameters of the target GPU are mounted to the container to be scheduled through the first component on the target node. The resource isolation control library is used to isolate and manage the resources of the target GPU, and the limiting parameters include maximum video memory usage and maximum computing power share. It's important to note that during the GPU node's compute isolation activation phase, each GPU node in the new cluster will launch two Daemonsets: one is the existing component vgpu-device-plugin, whose core functions include distributing the libvgpu.so hijacking library for VGPU mode, and also performing device, environment variable, and hostPath mounting operations for containers in both VGPU and MPS modes; the other is the newly added component nvidia-mps-daemon, used to start the nvidia-mps-server server on the node in MPS mode. User container processes can interact with this server to achieve process merging, thereby achieving compute isolation. Specifically, vgpu-device-plugin checks the Pod's configuration information when starting a Pod and enables VGPU or MPS mode by completing different hostPath mounting and environment variable configurations.

[0077] When the first virtualization mode is virtual GPU mode, the first component (vgpu-device-plugin) on the target node first locates the storage path of the libvgpu.so resource isolation control library, and then generates limit parameters by combining the resource request quota of the container to be scheduled (such as 0.5 cards corresponding to 50% computing power and 8GB of video memory). Subsequently, the libvgpu.so library, environment variables related to the limit parameters (such as CUDA_DEVICE_MEMORY_LIMIT_0), and lock file directory ( / tmp / vgpulock) are mounted to the container to ensure that resource isolation can be achieved through the library file after the container starts.

[0078] Step S402: When the first virtualization mode is a multi-process service mode, the process communication pipeline directory and shared memory space of the target GPU are mounted to the container to be scheduled through the second component on the target node. The process communication pipeline directory includes a communication interface between the container to be scheduled and the target node, and the shared memory space is used to store temporary computing data.

[0079] When the first virtualization mode is a multi-process service mode, the second component (mps-daemon) on the target node first identifies the process communication pipe directory ( / run / nvidia / mps / gpu / pipe) and shared memory space ( / run / nvidia / mps / shm) corresponding to the started MPS Server. Then, it mounts these two paths to the container and injects communication-related environment variables (such as CUDA_MPS_PIPE_DIRECTORY), enabling the container to communicate with the MPS Server through the pipe directory and store temporary data through the shared memory space. Finally, the mounting and adaptation of the target GPU and the container to be scheduled are completed.

[0080] For example, the hostPath mount path and environment variable configurations for the two GPU sharing modes, VGPU (isolation priority) and MPS (performance priority), are as follows: VGPU mode requires mounting / hook-path / vgpu / libvgpu.so and / hook-path / vgpu / containers / <id>There are three types of paths: / tmp / vgpulock, with environment variables including NVIDIA_VISIBLE_DEVICES (specifying visible GPU devices), CUDA_DEVICE_MEMORY_LIMIT_0 (memory limit), and CUDA_DEVICE_SM_LIMIT (computing power limit). MPS mode requires mounting the / tmp / nvidia-mps path, with environment variables including NVIDIA_VISIBLE_DEVICES (specifying visible GPU devices), CUDA_MPS_PIPE_DIRECTORY (inter-process communication pipe directory), CUDA_MPS_ACTIVE_THREAD_PERCENTAGE (computing power percentage limit), and CUDA_MPS_PINNED_DEVICE_MEM_LIMIT (fixed memory limit). Through differentiated mounting and environment variable configuration, the computing power isolation requirements of the two modes are achieved respectively.

[0081] In addition, the MPS mounting is explained as follows: MPS clients and servers communicate through pipes and shared memory (shm) shared by the host, therefore hostPath mounting configuration is required. The Daemonset used to deploy the MPS server contains two containers. The initialization container (initContainer) is responsible for mounting the host's / run / nvidia / mps directory to its own / mps path and mounting the shared memory to / mps / shm at startup. The main container (MPSDaemon) is responsible for starting nvidia-cuda-mps-control. When a business process calls CUDA through MPS, the corresponding nvidia-cuda-mps-server will be automatically started, which needs to complete the mounting from host / run / nvidia / mps to its own / mps and from host / run / nvidia / mps / shm to its own / dev / shm. When the business container (MPS Client) starts, vgpu-device-plugin will recognize its MPS mode configuration and return the mount and environment variable configurations through the Allocate interface. Specifically, it will mount the host / run / nvidia / mps / shm to the container / dev / shm, mount the host / run / nvidia / mps / gpu / pipe to the container's path with the same name, and inject three types of environment variables: CUDA_MPS_PIPE_DIRECTORY (specifying the communication pipe directory), CUDA_MPS_ACTIVE_THREAD_PERCENTAGE (the percentage of computing power limit requested by the container), and CUDA_MPS_PINNED_DEVICE_MEM_LIMIT (the number of MB of video memory limit requested by the container).

[0082] Thus, this application's embodiments employ differentiated mounting strategies for different virtualization modes to achieve precise adaptation and performance optimization: In virtual GPU mode, the first component mounts a resource isolation control library and limiting parameters such as maximum video memory usage and maximum computing power share to forcibly manage GPU resource usage, effectively avoiding resource contention between containers and ensuring business stability in isolation-priority scenarios; In multi-process service mode, the second component mounts a process communication pipeline directory (including communication interfaces between containers and target nodes) and shared memory space (used to store temporary computing data) to ensure efficient interaction between containers and the MPS server, reduce data transmission and context switching overhead, and improve inference performance in high-concurrency scenarios. Ultimately, this achieves the dual goals of isolation reliability and performance optimization for GPU sharing in both modes, balancing resource utilization and business operation requirements.

[0083] For example, loan risk model A requires 1.5 GPUs (1 full card + 0.5 shared cards), and model B requires 0.5 GPUs. Since they belong to different business lines, they need to be isolated through affinity. The implementation involves three steps: Step 1 is resource application and configuration. Model A submits two requests: one for a full card resource of 100 vgpu-cores, with annotations in the configuration information indicating gpu-share-mode: "performance"; the other for a shared card resource of 50 vgpu-cores, annotated with gpu-share-mode: "performance" and gpu-affinity: "loan_model_A"; Model B applies for a shared card resource of 50 vgpu-cores, with annotations in the configuration information indicating gpu-share-mode: "isolation" and gpu-affinity: "loan_model_B". Step 2 is GPU card scheduling. When the Extender makes a decision, it allocates an idle GPU card 0 to the entire GPU portion of Model A. After excluding GPU card 0 in the shared portion, it selects an idle GPU card 1, allocates 0.5 share, and records the label and mode. For Model B, it iterates and excludes GPU card 0 that is already exclusively used and GPU card 1 with mismatched affinity labels, and finally allocates an idle GPU card 2. Step 3 is node deployment and isolation settings. Model A (MPS mode) starts mps-daemon on the target node, mounts the / run / nvidia / mps path, and sets environment variables for computing power and memory limits to achieve multi-process sharing of CUDA Context. Model B (VGPU mode) loads vgpu-device-plugin, mounts the libvgpu.so hijacking library, and sets environment variables for computing power and memory limits to achieve forced resource isolation.

[0084] Thus, this embodiment not only increases GPU utilization from 30% to 85% by deploying the 0.5 card shared portion of model A on the same node as the entire card, effectively revitalizing fragmented resources, but also leverages the difference in tags between gpu-affinity=loan_model_A and gpu-affinity=loan_model_B to force model B to be scheduled to GPU card 2, completely avoiding resource competition with model A, and flexibly adapting the mode according to business needs; it also achieves card-level scheduling through the Extender component (such as allocating GPU card 1 to the shared portion of model A), breaking through the limitations of traditional node-level scheduling, strengthening fault isolation by constructing isolation rules with business-specific affinity tags, and supporting the on-demand activation of MPS and VGPU dual modes, balancing resource efficiency and operational stability, forming a deep synergy between technical effects and innovative design.

[0085] This application also provides a GPU scheduling device, please refer to... Figure 7 The GPU scheduling device includes: The determination module 10 is used to determine the GPU requirement information of the container to be scheduled in response to the GPU resource request of the container to be scheduled. Node scheduling module 20 is used to determine candidate nodes based on the GPU demand information, and to select GPUs with remaining available capacity on each candidate node as candidate GPUs. Resource scheduling module 30 is used to determine the target GPU from each of the candidate GPUs based on the GPU demand information; The deployment module 40 is used to deploy the container to be scheduled to the target node where the target GPU is located, and to mount the target GPU to the container to be scheduled.

[0086] Optionally, the GPU demand information includes a first GPU capacity demand, and the resource scheduling module 30 is further configured to: Based on the remaining available capacity of each of the candidate GPUs, the GPU that meets the capacity requirement of the first GPU among the candidate GPUs is selected as the first GPU. Identify the non-idle second GPUs among the first GPUs and determine whether the second GPUs meet the GPU sharing requirements of the container to be scheduled. If the conditions are met, then the second GPU will be used as the target GPU; If the conditions are not met, then a third GPU other than the second GPU is determined from each of the first GPUs, and the target GPU is determined from each of the third GPUs.

[0087] Optionally, the GPU demand information further includes a first virtualization mode and a first affinity label, wherein the first virtualization mode represents the priority requirement of the container to be scheduled for GPU resource utilization, and the first affinity label represents the business line affiliation of the container to be scheduled. The resource scheduling module 30 is also used for: Determine the second virtualization mode and second affinity tag of the container already mounted in the second GPU; If the first virtualization mode and the second virtualization mode are the same, and the first affinity label is the same as the second affinity label, then the second GPU is determined to meet the GPU sharing requirements of the container to be scheduled.

[0088] Optionally, the GPU demand information may further include a first service identifier, which represents the business service affiliation of the container to be scheduled; The resource scheduling module 30 is also used for: When the first virtualization mode is virtual GPU mode, determine whether the first service identifier and the second service identifier of the mounted container are consistent; If the first service identifier and the second service identifier are inconsistent, it is determined that the second GPU meets the GPU sharing requirements of the container to be scheduled.

[0089] Optionally, the GPU demand information further includes a first user identifier, and the resource scheduling module 30 is further configured to: When the first virtualization mode is a multi-process service mode, determine whether the first user identifier and the second user identifier of the mounted container are consistent. If the first user identifier and the second user identifier are the same, it is determined that the second GPU meets the GPU sharing requirements of the container to be scheduled.

[0090] Optionally, the node scheduling module 20 is further configured to: Obtain the first schedulable node in the cluster to which the container to be scheduled belongs, and determine the total remaining GPU capacity of each first node; Based on the total remaining GPU capacity, the nodes in each of the first nodes that meet the first GPU capacity requirements are selected as candidate nodes.

[0091] Optionally, the deployment module 40 is further configured to: When the first virtualization mode is virtual GPU mode, the resource isolation control library and limiting parameters of the target GPU are mounted to the scheduling container through the first component on the target node. The resource isolation control library is used to isolate and manage the resources of the target GPU, and the limiting parameters include maximum video memory usage and maximum computing power share. When the first virtualization mode is a multi-process service mode, the process communication pipeline directory and shared memory space of the target GPU are mounted to the container to be scheduled through the second component on the target node. The process communication pipeline directory includes a communication interface between the container to be scheduled and the target node, and the shared memory space is used to store temporary computing data.

[0092] The GPU scheduling device provided in this application, employing the GPU scheduling method described in the above embodiments, can solve the technical problem of how to improve the utilization rate of GPU resources. Compared with the prior art, the beneficial effects of the GPU scheduling device provided in this application are the same as those of the GPU scheduling method described in the above embodiments, and other technical features in the GPU scheduling device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0093] This application provides a GPU scheduling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the GPU scheduling method in the first embodiment described above.

[0094] The following is for reference. Figure 8 The diagram illustrates a structural schematic of a GPU scheduling device suitable for implementing embodiments of this application. The GPU scheduling device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The GPU scheduling device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0095] like Figure 8 As shown, the GPU scheduling device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory 1002 or a program loaded from storage device 1003 into random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the GPU scheduling device. The processing unit 1001, read-only memory 1002, and random access memory 1004 are interconnected via bus 1005. Input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the GPU scheduling device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows GPU scheduling devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0096] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0097] The GPU scheduling device provided in this application, employing the GPU scheduling method in the above embodiments, can solve the technical problem of how to improve the utilization rate of GPU resources. Compared with the prior art, the beneficial effects of the GPU scheduling device provided in this application are the same as those of the GPU scheduling method provided in the above embodiments, and other technical features in this GPU scheduling device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0098] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0099] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0100] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the GPU scheduling method in the above embodiments.

[0101] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0102] The aforementioned computer-readable storage medium may be included in the GPU scheduling device; or it may exist independently and not be assembled into the GPU scheduling device.

[0103] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a GPU scheduling device, cause the GPU scheduling device to: in response to a GPU resource request from a container to be scheduled, determine the GPU requirement information of the container to be scheduled; determine candidate nodes based on the GPU requirement information, and select GPUs with remaining available capacity on each candidate node as candidate GPUs; determine a target GPU from among the candidate GPUs based on the GPU requirement information; deploy the container to be scheduled to the target node where the target GPU is located, and mount the target GPU to the container to be scheduled.

[0104] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0107] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described GPU scheduling method, thereby solving the technical problem of how to improve the utilization rate of GPU resources. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the GPU scheduling method provided in the above embodiments, and will not be repeated here.

[0108] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the GPU scheduling method described above.

[0109] The computer program product provided in this application can improve the utilization rate of GPU resources. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as the beneficial effects of the GPU scheduling method provided in the above embodiments, and will not be repeated here.

[0110] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.< / id>

Claims

1. A GPU scheduling method, characterized in that, The GPU scheduling method includes: In response to a GPU resource request from a container to be scheduled, determine the GPU requirement information of the container to be scheduled; Based on the GPU demand information, candidate nodes are determined, and GPUs with remaining available capacity on each candidate node are selected as candidate GPUs. The target GPU is determined from each of the candidate GPUs based on the GPU requirement information; The container to be scheduled is deployed to the target node where the target GPU is located, and the target GPU is mounted to the container to be scheduled.

2. The GPU scheduling method as described in claim 1, characterized in that, The GPU requirement information includes a first GPU capacity requirement, and the step of determining the target GPU from the candidate GPUs based on the GPU requirement information includes: Based on the remaining available capacity of each of the candidate GPUs, the GPU that meets the capacity requirement of the first GPU among the candidate GPUs is selected as the first GPU. Identify the non-idle second GPUs among the first GPUs and determine whether the second GPUs meet the GPU sharing requirements of the container to be scheduled. If the conditions are met, then the second GPU will be used as the target GPU; If the conditions are not met, then a third GPU other than the second GPU is determined from each of the first GPUs, and the target GPU is determined from each of the third GPUs.

3. The GPU scheduling method as described in claim 2, characterized in that, The GPU requirement information also includes a first virtualization mode and a first affinity label. The first virtualization mode represents the priority requirement of the container to be scheduled for GPU resource utilization, and the first affinity label represents the business line affiliation of the container to be scheduled. The step of determining whether the second GPU meets the GPU sharing requirements of the container to be scheduled includes: Determine the second virtualization mode and second affinity tag of the container already mounted in the second GPU; If the first virtualization mode and the second virtualization mode are the same, and the first affinity label is the same as the second affinity label, then the second GPU is determined to meet the GPU sharing requirements of the container to be scheduled.

4. The GPU scheduling method as described in claim 3, characterized in that, The GPU demand information also includes a first service identifier, which represents the business service affiliation of the container to be scheduled. The step of determining that the second GPU meets the GPU sharing requirements of the container to be scheduled includes: When the first virtualization mode is virtual GPU mode, determine whether the first service identifier and the second service identifier of the mounted container are consistent; If the first service identifier and the second service identifier are inconsistent, it is determined that the second GPU meets the GPU sharing requirements of the container to be scheduled.

5. The GPU scheduling method as described in claim 3, characterized in that, The GPU requirement information also includes a first user identifier, and the step of determining that the second GPU meets the GPU sharing requirement of the container to be scheduled includes: When the first virtualization mode is a multi-process service mode, determine whether the first user identifier and the second user identifier of the mounted container are consistent. If the first user identifier and the second user identifier are the same, it is determined that the second GPU meets the GPU sharing requirements of the container to be scheduled.

6. The GPU scheduling method as described in claim 2, characterized in that, The step of determining candidate nodes based on the GPU demand information includes: Obtain the first schedulable node in the cluster to which the container to be scheduled belongs, and determine the total remaining GPU capacity of each first node; Based on the total remaining GPU capacity, the nodes in each of the first nodes that meet the first GPU capacity requirements are selected as candidate nodes.

7. The GPU scheduling method as described in claim 3, characterized in that, The step of mounting the target GPU to the container to be scheduled includes: When the first virtualization mode is virtual GPU mode, the resource isolation control library and limiting parameters of the target GPU are mounted to the scheduling container through the first component on the target node. The resource isolation control library is used to isolate and manage the resources of the target GPU, and the limiting parameters include maximum video memory usage and maximum computing power share. When the first virtualization mode is a multi-process service mode, the process communication pipeline directory and shared memory space of the target GPU are mounted to the container to be scheduled through the second component on the target node. The process communication pipeline directory includes a communication interface between the container to be scheduled and the target node, and the shared memory space is used to store temporary computing data.

8. A GPU scheduling device, characterized in that, The GPU scheduling device includes: The determination module is used to determine the GPU requirement information of the container to be scheduled in response to the GPU resource request of the container to be scheduled. The node scheduling module is used to determine candidate nodes based on the GPU demand information, and to select GPUs with remaining available capacity on each candidate node as candidate GPUs. The resource scheduling module is used to determine the target GPU from each of the candidate GPUs based on the GPU demand information; The deployment module is used to deploy the container to be scheduled to the target node where the target GPU is located, and to mount the target GPU to the container to be scheduled.

9. A GPU scheduling device, characterized in that, The GPU scheduling device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the GPU scheduling method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the GPU scheduling method as described in any one of claims 1 to 7.

Images