Kubernetes-based GPU resource fragment intelligent scheduling system and method

Abstract

The invention discloses a Kubernetes-based GPU resource fragment intelligent scheduling system and method, and relates to the technical field of artificial intelligence platform infrastructures. In order to solve the problems of GPU resource fragmentation, high migration risk and the like caused by scheduling strategy defects and lack of an automation mechanism in Kubernetes, the method comprises the following steps: intelligently identifying and determining migration candidate objects by monitoring a cluster state and utilizing a simulation scheduling algorithm; then, a GPU defragmentation task is created and persisted, and the task is used as a carrier to drive a non-inductive migration process of first capacity expansion and then capacity reduction; according to the process, a healthy Pod copy is created at a new node through directional scheduling, after the new copy is ready, the flow of an old copy is smoothly removed, then a non-ready old copy is accurately removed through a capacity reduction mechanism in Kubernetes, and finally task termination and resource integration are achieved. Therefore, the resource utilization rate is effectively improved and the continuity and stability of the service are ensured through the automatic and intelligent management of the GPU fragments and the refined migration process.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence platform infrastructure technology, and more specifically, to a Kubernetes-based intelligent scheduling system and method for GPU resource fragmentation. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence (AI), machine learning (ML), and high-performance computing (HPC), hardware accelerators, represented by graphics processing units (GPUs), have become an indispensable core resource supporting such computationally intensive applications. In order to achieve efficient and elastic management and scheduling of these expensive and important hardware resources, the industry has widely adopted container orchestration platforms, represented by Kubernetes, to build cloud-native AI infrastructure.

[0003] In the standard Kubernetes architecture, a Device Plugin framework is typically used to manage physical devices such as GPUs, allowing users to declare their GPU resource requirements in the Pod specifications. Kubernetes' default scheduler, when scheduling a Pod, seeks a node capable of fulfilling all its resource requests for deployment. However, in large-scale, multi-tenant, and highly dynamic production environments, this standard management and scheduling model leads to an increasingly prominent GPU resource fragmentation problem. Specifically, although the total amount of idle GPU resources in the cluster may be sufficient to meet the needs of new tasks, these idle resources are scattered across different physical nodes. This results in no single node being able to fulfill a task request requiring multiple GPUs, causing task scheduling failures and severely limiting the overall resource utilization and throughput of the cluster.

[0004] Faced with this fragmentation dilemma, the existing Kubernetes-based management system has exposed many inherent flaws.

[0005] First, its scheduling strategy itself lacks foresight and optimization awareness. The core goal of Kubernetes' default scheduler is to find an available node, rather than the optimal one, and it tends to distribute workloads to achieve a certain degree of CPU and memory load balancing. However, for indivisible, exclusive resources like GPUs, this strategy is precisely the root cause of continuous and deliberate fragmentation.

[0006] Secondly, once GPU resource fragmentation occurs, the current Kubernetes ecosystem lacks a native, automated defragmentation mechanism. Solving this problem often relies on high-risk manual intervention by platform operations personnel. This involves manually identifying migrateable workloads and performing a series of complex and error-prone operations such as eviction, scaling down, and resizing to free up Pods to merge idle resources. This process is not only inefficient and slow, but also highly likely to cause interruptions to online services.

[0007] More importantly, existing migration methods are often destructive and coarse-grained. Simply deleting and rebuilding a Pod can directly cause the service to become unavailable for a short period of time. If shared configurations such as affinity rules of the workload (such as Deployment) are modified to achieve precise migration of a single Pod, it will trigger global and unexpected Pod rolling updates, producing side effects that affect the whole system. This intrusive operation also temporarily pollutes the application's declarative configuration.

[0008] Finally, there is a lack of intelligence in the decision-making level of defragmentation. In a complex cluster environment, how to scientifically select the Pod with the lowest migration cost and the highest defragmentation benefit as the migration target? Existing technologies do not provide an intelligent decision engine that can simulate, predict the migration effect and conduct a comprehensive evaluation, which often leads to poor defragmentation results. In fact, it may even create new fragments on another node while defragmenting one node.

[0009] Therefore, an optimized intelligent scheduling scheme for GPU resource fragments is needed. Summary of the Invention

[0010] To overcome the limitations of existing technologies, according to one aspect of this application, a Kubernetes-based intelligent scheduling method for GPU resource fragmentation is provided, comprising: Based on real-time cluster status monitoring and fragmentation risk identification, migration candidates are obtained. Based on the source workload definition, the migration candidate objects are initialized and persisted with migration tasks to obtain active GPU defragmentation tasks (CRs). The system expands capacity based on the active GPU defragmentation task CR and selectively schedules new Pod replicas to obtain the new Pods that have been scheduled. Perform new replica health checks and traffic switching preparations on the scheduled new Pod to obtain a ready new Pod; Smoothly remove traffic from old Pod instances and isolate them from old replicas to obtain non-ready old Pods; Perform precise scaling down, resource release, and task termination on non-ready old Pods to obtain terminated GPU defragmentation tasks.

[0011] According to another aspect of this application, a Kubernetes-based intelligent scheduling system for GPU resource fragments is also provided, which is used to execute the Kubernetes-based intelligent scheduling method for GPU resource fragments as described above.

[0012] Compared with existing technologies, the Kubernetes-based GPU resource fragment intelligent scheduling system and method provided in this application is based on the core technical concept of achieving intelligent governance of GPU resources through a closed-loop, state machine-based task orchestration process.

[0013] This method first monitors the real-time status of the cluster through a resident GPU defragmentation controller and uses an intelligent algorithm that simulates rescheduling to evaluate the migration benefits of running Pods, thereby identifying and selecting migration candidates with the lowest defragmentation cost and the best optimization effect. Once the migration target is determined, the method does not use a simple destructive eviction, but transforms it into a persistent, traceable custom resource (CR) task. This task serves as the carrier to drive a refined migration process with controllable impact on business operations. This process, through deep collaboration with a customized GPU application scheduler, first safely "expands" a new healthy replica on the predetermined target node. In this step, the scheduler can identify the Pods associated with the migration task and force them to run on the designated node, ensuring the accurate achievement of the migration intention. After the new replica is fully ready and can handle business traffic, the method smoothly removes traffic from the old Pod instance to be removed by updating the native Kubernetes Readiness Gate mechanism, safely isolating it without affecting service. Finally, this method cleverly utilizes Kubernetes' built-in strategy of prioritizing the deletion of non-ready Pods during scaling down. By restoring the number of replicas of the workload to its original value, the deletion of isolated old replicas can be precisely triggered, thereby releasing the GPU resources they occupy. The entire resource cleanup and consolidation is completed without modifying the workload configuration and without the user's awareness.

[0014] Through the implementation of the above technical solutions, the present invention has achieved significant beneficial effects.

[0015] First, it enables automated and proactive management of GPU resource fragmentation, continuously integrating GPU resources on physical nodes, significantly improving the actual availability and utilization of GPUs in the cluster, thereby effectively solving the scheduling failure problem caused by resource fragmentation.

[0016] Secondly, this invention combines customized scheduling strategies with a defragmentation process, forming a collaborative optimization loop of "prevention + governance." This ensures that during daily scheduling, pods are placed compactly to reduce fragmentation, and defragmentation tasks are executed efficiently and accurately when necessary. Crucially, this method, through an atomic operation process of scaling up and then scaling down, combined with smooth traffic migration and clever use of Kubernetes' native mechanisms, truly achieves a refined, seamless Pod migration with controllable impact on business operations. This ensures business continuity and stability while avoiding intrusive modifications to application configurations.

[0017] Finally, the intelligent decision-making mechanism based on simulation scheduling ensures that every fragmentation operation is based on data evaluation, beneficial and efficient, avoiding the negative benefits and risks of blind migration. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 This is a schematic diagram of task initialization in the Kubernetes-based intelligent scheduling method for GPU resource fragments according to an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the scaling phase in the Kubernetes-based GPU resource fragment intelligent scheduling method according to an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of the scaling-down phase in the Kubernetes-based intelligent scheduling method for GPU resource fragments according to an embodiment of this application.

[0022] Figure 4 This is a flowchart of a Kubernetes-based intelligent scheduling method for GPU resource fragments according to an embodiment of this application.

[0023] Figure 5 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0025] Therefore, in view of the technical deficiencies revealed in the background art, this application proposes a Kubernetes-based intelligent scheduling system and method for GPU resource fragmentation. Figure 1 This is a schematic diagram of task initialization in the Kubernetes-based intelligent scheduling method for GPU resource fragments according to an embodiment of this application. Figure 2 This is a schematic diagram of the scaling phase in the Kubernetes-based GPU resource fragment intelligent scheduling method according to an embodiment of this application. Figure 3 This is a schematic diagram of the scaling-down phase in the Kubernetes-based intelligent scheduling method for GPU resource fragments according to an embodiment of this application. Figure 4 The flowchart below shows a Kubernetes-based intelligent scheduling method for GPU resource fragments according to an embodiment of this application. like Figures 1 to 4 As shown, the Kubernetes-based GPU resource fragmentation intelligent scheduling method includes the following steps: S110, monitoring the cluster status and identifying fragmentation risks based on real-time cluster status to obtain migration candidates; S120, initializing and persisting migration tasks for the migration candidates based on the source workload definition to obtain active GPU defragmentation tasks (CRs); S130, scaling up and scheduling new Pod replicas based on the active GPU defragmentation tasks (CRs) to obtain scheduled new Pods; S140, performing new replica health checks and traffic switching preparations on the scheduled new Pods to obtain ready new Pods; S150, performing smooth traffic removal and old replica isolation on old Pod instances to obtain non-ready old Pods; and S160, performing precise scaling down, resource release, and task termination on non-ready old Pods to obtain terminated GPU defragmentation tasks.

[0026] In step S110, cluster status monitoring and fragmentation risk identification are performed based on real-time cluster status to obtain migration candidates. That is, based on the real-time acquired cluster status, continuous monitoring of the cluster status and intelligent identification of fragmentation risks are performed, and suitable migration candidates are selected accordingly. This optimization decision-making step proactively and periodically scans the cluster through a built-in intelligent detection and decision-making mechanism. It can automatically identify Pod instances that, although currently running normally, have a fragmenting impact on the overall cluster resource layout. By using data simulation and benefit evaluation at this stage, this step can accurately select an optimal migration target, thereby transforming fragmentation remediation from a passive, reactive remedial measure into an automated, proactive optimization behavior, providing clear and high-value execution input for the subsequent truly business-insensitive Pod migration process.

[0027] like Figure 1 and Figure 4 As shown in this embodiment, the specific process of monitoring cluster status and identifying fragmentation risks based on real-time cluster status to obtain migration candidates begins with obtaining a complete real-time cluster status. In one specific implementation, this real-time cluster status may include a list of nodes and the GPU usage of each node, as well as a list of all running Pods in the cluster and their specification definitions. Based on this status snapshot, the method first filters out nodes from the node list that have allocated more than zero GPUs, forming a GPU node set. This aims to focus the analysis on the computing nodes that actually bear the GPU workload, thus excluding interference from irrelevant nodes. Next, based on this GPU node set, it filters out Pod instances from the list of running Pods that run on the GPU node set and declare GPU resource requests in their specifications, thereby obtaining a candidate Pod set.

[0028] After obtaining the candidate Pod set, the method sorts the candidate Pod set to obtain a sorted list of GPU Pods. The sorting strategy here follows the principle of minimizing the cost of reorganization. A specific implementation can be to scan all nodes containing GPU applications in order of the number of GPUs requested by the Pods running on them (e.g., in 1, 2, 4, etc.). Since Pods requesting fewer GPUs are more likely to cause resource fragmentation on the node, and their migration and reorganization costs are relatively lower, Pods requesting fewer GPUs will be placed at the top of the sorted list.

[0029] Subsequently, this method enters the core simulation and evaluation phase, which involves performing simulated scheduling calculations on each candidate Pod in the sorted GPU Pod list to obtain the corresponding simulated scheduling results. This process specifically includes: extracting the first candidate Pod sequentially from the sorted GPU Pod list; then, extracting the source node information of the candidate Pod and the amount of GPU resources requested by the Pod itself; next, the method does not directly operate on the real cluster, but generates a hypothetical cluster state in memory based on the real-time cluster state. A key operation in this hypothetical state is marking the GPU resources originally belonging to this candidate Pod on the source node as available, simulating the scenario where the resources occupied by the Pod are released after migration; then, the candidate Pod and this hypothetical cluster state are input into the system's simulation scheduler to obtain a simulated scheduling target node. In this implementation, the simulation scheduler can operate using a compact scheduling strategy. Specifically, when processing the candidate Pod and the hypothetical cluster state, it selects a node from the hypothetical cluster state with the fewest remaining GPU resources that can still satisfy the candidate Pod's resource requests, and uses this node as the target node for this simulated scheduling. Finally, the simulated scheduling target node and the candidate Pod are encapsulated to form the simulated scheduling result of this iteration.

[0030] After obtaining the simulation scheduling results, the process proceeds to the final decision-making stage. This method evaluates the migration benefits and makes a final decision based on the simulation scheduling results and the GPU status of the source node to determine the migration candidates. The benefit evaluation here includes a precise judgment of the optimization effect: if, after the simulation scheduling calculation, the candidate Pod is rescheduled back to its current node, or if the simulated migration operation does not significantly change the resource utilization of the source and target nodes (e.g., the total number of remaining GPUs on both nodes remains unchanged), then the system determines that this migration does not significantly optimize the overall GPU utilization of the cluster, and the Pod does not need to be migrated. Conversely, if the simulated migration can optimize resource allocation, then the simulated migration is considered beneficial. Once the system determines that a simulated migration is beneficial, it packages the relevant information of the first candidate Pod into a migration candidate object and outputs it. At the same time, the entire scanning and identification process stops in this round to ensure the atomicity of the operation and avoid concurrent conflicts. If it determines that there is no optimization benefit, it discards the result of this simulation and continues to take the next candidate Pod from the sorted list of GPU Pods, repeating the above simulation scheduling and evaluation process until a migration candidate object that meets the conditions is found or the entire list is traversed.

[0031] It is worth mentioning that, in this embodiment, the real-time cluster status refers to a complete snapshot of all entities and their statuses related to GPU resources in the entire Kubernetes cluster at a specific point in time, including physical nodes and their GPU allocation, the definition of all running Pods and their resource requests, etc.; the migration candidate is a structured data object that encapsulates the final decision of the analysis and evaluation in this stage, clearly specifying which specific Pod instance should be migrated and the target node to which it is to be migrated; and the compact scheduling strategy is a scheduling idea aimed at maximizing the utilization of node resources. It tends to deploy newly scheduled Pods to nodes that are already partially used as much as possible, thereby reserving completely idle nodes with a large amount of continuous resources.

[0032] In a specific implementation example, assume a cluster contains Node-A (total 8 GPUs, 1 used, 7 remaining), Node-B (total 8 GPUs, 7 used, 1 remaining), and Node-C (total 8 GPUs, completely idle). Currently, Node-A is running an instance Pod-alpha requesting 1 GPU. When performing this step, Pod-alpha is prioritized for simulation because it requests fewer GPUs and is at the top of the sorted list. Subsequently, the system generates a hypothetical cluster state where all 8 GPUs of Node-A are marked as available. When the simulation scheduler selects a new node for Pod-alpha requesting 1 GPU based on a compact scheduling strategy, it will choose Node-B, which has the fewest remaining resources, as the target node for simulation scheduling, between Node-B (with 1 remaining GPU) and Node-C (with 8 remaining GPUs). Finally, the method evaluates the benefits of this migration: before the migration, node resources were severely fragmented (both Node-A and Node-B were underutilized); after the migration, Node-A's GPU resources were fully integrated (8 GPUs remaining), while Node-B was fully filled (0 GPUs remaining). Since this simulation was deemed beneficial, the system generates a migration candidate to migrate Pod-alpha from Node-A to Node-B, as the final output of this step.

[0033] In step S120, the migration candidate object is initialized and persisted based on the source workload definition to obtain an active GPU defragmentation task (CR). That is, after determining the migration candidate object, the entire intelligent scheduling method enters the stage of initializing and persisting the migration task for that candidate object based on the source workload definition, aiming to obtain an active GPU defragmentation task custom resource (CR). In this way, the transient scheduling decision generated in memory in the previous stage is transformed into a persistent, stateful task entity that can be understood and collaboratively executed by all automated components in the cluster.

[0034] In Kubernetes' declarative architecture, directly imposing commands on Pods is difficult to track and carries risks. Therefore, this solution creates a custom resource (CR) to describe and track the complete lifecycle of each GPU defragmentation task. This approach not only provides a robust, queryable source of status and parameters for all subsequent precise operations (such as scaling up, targeted scheduling, and scaling down), but also makes the entire complex migration process monitorable, manageable, and fault-tolerant. By encapsulating migration intents as an API object, the complex multi-stage orchestration process is transformed into a standard coordination process following the Kubernetes controller pattern, thereby significantly improving the system's robustness and scalability.

[0035] In this embodiment, the process of initializing and persisting migration tasks for the migration candidate object based on the source workload definition to obtain an active GPU defragmentation task (CR) includes the following steps. First, the system aggregates migration context data for the migration candidate object generated in the previous step to obtain a more enriched migration context. This means that the system not only uses the migration candidate object itself, but also actively queries the Kubernetes API Server based on information such as the Pod name and its associated workload reference to obtain the complete source workload definition associated with the Pod, as well as the specific GPU state snapshots of the source and target nodes at the time of decision-making. This scattered information is then aggregated into a rich data structure containing the context required for all subsequent operations.

[0036] After aggregating complete context information, the system synthesizes GPU defragmentation task resources based on this enriched migration context and the definition of the source workload, resulting in a task CR synthesized in memory. This process takes place in the controller's memory, strictly following the predefined GPU defragmentation task CRD (Custom Resource Definition) specification. A CR object is instantiated, and the various data from the enriched migration context are meticulously populated into the corresponding fields of the object. This information to be populated is very specific and may include: a reference to the workload being operated on (workloadRef); the name of the specific Pod instance selected to be migrated (podName); the predetermined target node (predeterminedTargetNode) calculated by the simulated scheduling; a snapshot of the GPU state of the source and target nodes at the time of decision, for subsequent safety verification by the scheduler; the original number of replicas of the target workload, which is crucial for subsequent precise scaling down; and the task timeout, used to control the execution time of the entire task and prevent the task from being suspended indefinitely due to anomalies.

[0037] Once the synthesized task CR is fully constructed in memory, the system performs the final step: persisting and activating the lifecycle of the synthesized task CR to obtain an active GPU defragmentation task CR. Specifically, this involves first serializing the synthesized task CR into an object, for example, converting it to JSON format to obtain a serialized CR representation. Then, the controller uses this serialized CR as the request body to send a standard create request to the Kubernetes API Server. After receiving and successfully processing this request, the API Server persists the object to its backend database and returns a complete object containing system-assigned metadata (such as a unique UID and version number). At this point, the task CR is no longer a temporary in-memory object but an active, real API resource existing in the cluster, marking the official entry of the migration process into the executable phase. Its initial state will be set to Pending.

[0038] It is worth mentioning that, in the embodiments of this application, the source workload definition refers to the complete declarative description of the parent controller (such as Deployment, StatefulSet, etc.) to which the migrated Pod belongs. It defines key behaviors such as the Pod's generation template and the expected number of replicas. The GPU defragmentation task CR (Custom Resource) is the core data structure and state machine carrier of this invention. It is a user-defined API resource type that is extended to the Kubernetes API through CRD and used to manage the entire defragmentation task in a declarative manner. An active CR specifically refers to a CR instance that has been accepted and persisted by the Kubernetes API Server and can be detected by the controller in the system through the standard watch mechanism for its creation, update, and deletion events, so that it can react to its state changes.

[0039] In a specific example, the system has generated a migration candidate object, specifying that Pod-alpha will be migrated from Node-A to Node-B. Now, the controller will execute this step. First, it finds the Deployment to which Pod-alpha belongs through its ownerReference and reads that its original replica count is 1. Simultaneously, it records a snapshot of the current state of Node-A (7 GPUs remaining) and Node-B (1 GPU remaining). Then, the controller constructs a GPU defragmentation task (CR) object in memory, pointing the workloadRef to the Deployment, setting the podName to the instance name of Pod-alpha, setting the predictedTargetNode to Node-B, setting originalReplicaCount to 1, and filling in the node state snapshot and the default timeout. Finally, it submits this constructed CR object to the Kubernetes API Server. After the API Server successfully creates the resource, a new GPUDefragTask object named defrag-task-xxxx appears in the cluster, its status now Pending, awaiting processing by the controller in the next stage.

[0040] In step S130, the system expands capacity based on the active GPU defragmentation task CR and directs the scheduling of new Pod replicas to obtain a new scheduled Pod. That is, after successfully creating and activating a GPU defragmentation task CR, the system expands capacity based on the active GPU defragmentation task CR and directs the scheduling of new Pod replicas to obtain a new scheduled Pod. This initiates the actual Pod migration process in a way that has a controllable impact on business operations.

[0041] To achieve seamless migration of online services, this application's technical solution replaces the destructive operation of simply deleting the old Pod and creating a new one. Instead, it adopts a more elegant strategy of scaling up first and then scaling down. This step is the scaling-up phase of this strategy, aiming to safely create a new Pod replica in parallel without interfering with existing service Pods. Furthermore, it's not enough to simply create a new replica; it must also ensure that this new replica is precisely scheduled to the predetermined target node determined through intelligent computation in the first phase. Therefore, this step is designed to add a temporary replica to the target workload without loss, and utilizes an innovative scheduling coordination mechanism to ensure that this newly created replica bypasses conventional scheduling logic and is deterministically placed in the designated location after security verification, laying a solid foundation for subsequent traffic switching and removal of the old replica.

[0042] like Figure 2 As shown in this embodiment, the process is completed through a multi-party interaction involving the GPU defragmentation controller and the GPU application scheduler, and specifically includes the following steps. First, the controller performs workload scaling and identifier injection. Specifically, when the GPU defragmentation controller detects an active GPU defragmentation task CR with a pending status, it immediately reads the workload reference and original replica count from the task. Then, the controller sends an update request to the Kubernetes API Server, incrementing the replica count of the workload by one from its original value, thus triggering the initial scaling action. Simultaneously, to accurately identify and track the Pod created by the scaling, the controller injects a unique identifier (e.g., a special annotation whose value can be the unique identifier UID of the task CR) into the updated workload's Pod template. After these two modifications, an augmented workload definition is obtained, and the controller updates the task CR's status to ScalingUpNewReplice to record progress, thereby generating an updated GPU defragmentation task CR.

[0043] Next, Kubernetes' native mechanisms come into play. The system's built-in workload controller (such as the Deployment Controller) detects a mismatch between the expected number of replicas in the augmented workload definition and the actual number of replicas currently running. This triggers its native Pod replica instantiation logic, creating a new Pod object based on the augmented Pod template. During this process, the unique identifier injected in the previous step is fully inherited into the metadata of this new Pod. This new Pod, in a Pending state and carrying a tracking identifier, is a traceable, pending-scheduling Pod.

[0044] Finally, there's the deterministic scheduling and binding based on task collaboration, handled by a dedicated GPU application scheduler. When a traceable Pod enters the scheduling queue, the GPU application scheduler checks its metadata and identifies it as a special Pod associated with a GPU defragmentation task based on its unique identifier. At this point, the scheduler doesn't perform its usual node filtering and scoring logic. Instead, based on this identifier, it queries the active GPU defragmentation task CR and extracts the pre-determined target node and the node state snapshot recorded at the time of decision-making. This is a crucial safety verification step: the scheduler checks the current actual resource status of the target node to determine if it matches the snapshot taken when the task was created or still meets the resource requirements of the new Pod. If the verification passes, the scheduler directly binds the traceable Pod to the target node; if the verification fails, it means the target node's state has changed unexpectedly and cannot accept the new Pod, the scheduling will fail, and the task will be placed in a failed state. Once the binding is successful, the node name field of the Pod object will be filled in, and its state will change to a new Pod that has been scheduled, waiting for the Kubelet process on the target node to complete the subsequent startup process.

[0045] In a specific example, the system already has an active GPU defragmentation task CR for Pod-alpha, in a pending state, specifying the target node Node-B and the original replica count of 1. After the controller detects this task, it sends a request to the API Server to change the replica count of the Deployment to which Pod-alpha belongs to 2. The Kubernetes Deployment controller then creates a new Pod instance, Pod-alpha-new. At this point, the GPU application scheduler receives Pod-alpha-new for scheduling. Upon inspection, it finds that this Pod has a special task identifier, so it doesn't compare the scheduling scores with Node-B, but directly locks the target—Node-B—from the associated task CR. The scheduler then checks if Node-B still has at least one GPU available (verified against the task snapshot). Once resource availability is confirmed, it directly binds Pod-alpha-new to Node-B. Thus, the old Pod-alpha is still running on Node-A, while simultaneously, an identical new Pod (Pod-alpha-new) has been scheduled as a new Pod, explicitly set to run on Node-B, completing the first phase of scaling.

[0046] In step S140, a new replica health check and traffic switching preparation are performed on the scheduled new Pod to obtain a ready new Pod. That is, after the new Pod replica is successfully scheduled to the target node, the intelligent scheduling method proceeds to the stage of performing a new replica health check and traffic switching preparation on the scheduled new Pod. The purpose is to ensure that before any disposal operations are performed on the old Pod, the newly created Pod replica has not only been successfully started, but its internal application has also been fully initialized, entering a ready state capable of healthily and stably handling business traffic. It should be understood that in complex application environments, a Pod needs to go through a series of processes from being scheduled to being truly available, including image pulling, container creation, application startup, and health checks. If the old replica is removed hastily before the new replica is fully ready, it will directly lead to service interruption and capacity fluctuations. To this end, this step establishes a reliable verification mechanism, relying on strict health status verification and comprehensive timeout monitoring, to ensure that the entire migration process is only allowed to proceed to the next step when the new copy is 100% confirmed to be able to take over the work of the old copy. This paves the way for subsequent zero-downtime traffic switching and the safe retirement of the old copy.

[0047] In this embodiment, the process of performing health checks and traffic switching preparation on a scheduled new Pod to obtain a ready new Pod includes the following steps. The process begins with the controller continuously monitoring the scheduled new Pod generated in the previous step. Specifically, the controller periodically queries the status fields of this new Pod instance, focusing not only on whether the Pod is in the running stage, but also on the more precise array of status conditions (status.conditions). Within this array, the controller specifically looks for the condition of type "Ready" and waits for the status field of that condition to become true. In Kubernetes, a Pod's ready condition being true means that all containers within the Pod have passed their respective configured readiness probes, a prerequisite for the Pod to be added to the service backend endpoint list and begin receiving traffic. Parallel to status monitoring is strict timeout monitoring. Throughout the waiting process for the new replica to become ready, the controller reads the preset task timeout in seconds from the associated GPU defragmentation task (CR) and simultaneously monitors the elapsed time. If the new replica is not ready within the timeout period, the controller will immediately stop waiting and trigger the rollback process, while marking the CR (Replica Ready) task status as Timeout. This is to prevent the migration task from being suspended indefinitely due to a Pod startup error. When all checks pass successfully, that is, when the controller successfully observes the new replica's readiness status becoming true within the timeout period, the health check step is successfully completed. At this point, the controller will immediately send an update request to the Kubernetes API Server, updating the CR status phase of the GPU defragmentation task corresponding to this migration from ScalingUpNewReplica to its next state, NewReplicaReady. This status update signifies the successful completion of the scaling and preparation phases and serves as a signal to unlock permissions for the next phase of smooth traffic removal and isolation from the old replica.

[0048] In a specific example, a new Pod, Pod-alpha-new, has been scheduled to Node-B. The GPU defragmentation controller begins monitoring its status. At initial time T0, the readiness condition of Pod-alpha-new is false. The controller continues to wait. At T0+35 seconds, the application inside Pod-alpha-new successfully starts and completes its health check. Kubelet then updates the Pod status, inverting the readiness condition to true. In the next monitoring cycle, the GPU defragmentation controller captures this change, confirming that Pod-alpha-new has become a ready new Pod. It then immediately updates the associated defrag-task-xxxx task CR, setting its state stage to NewReplicaReady.

[0049] In step S150, the old Pod instance undergoes smooth traffic removal and isolation from the old replica to obtain a non-ready old Pod. That is, after a new Pod replica is confirmed to be fully healthy and ready, the old Pod instance undergoes smooth traffic removal and isolation from the old replica to ultimately obtain a non-ready old Pod. This clever use of Kubernetes internal mechanisms safely and gracefully removes the dynamically migrated "old" service instance from the active service pool, isolating the old Pod instance selected as the migration target but still providing services, without interrupting any ongoing service requests or affecting the service quality for new user requests. The Kubernetes traffic management layer (such as Service, Ingress Controller) is notified in a controlled manner to stop forwarding any new network traffic to this specific old Pod instance, while giving it sufficient time to process already received or en route requests. This process not only ensures the continuity of user service experience, but more importantly, it paves the way for scaling down operations driven by Kubernetes native mechanisms by marking the old Pod as an inactive but still "alive" state.

[0050] In this embodiment, this step is specifically implemented by the GPU defragmentation controller performing a targeted API operation on a Kubernetes function called a Readiness Gate. After the new replica is ready, the controller selects the old Pod instance to be defragmented (i.e., the instance specified by the podName field in the task CR) and updates its Readiness Gate to make it appear as a non-ready state. Specifically, the controller first obtains the precisely locked name of the old Pod instance from the GPU defragmentation task CR. Next, the controller needs to ensure that this old Pod contains a specific Readiness Gate defined by the system in its definition. If the original definition of the Pod does not have one, the controller dynamically adds one via a PATCH request, for example, a Readiness Gate named "chaos.expert.com / defrag-in-progress". Then, the controller updates the state of this old Pod instance again, specifically by adding or updating a condition with the same name as the aforementioned Readiness Gate in its state condition array and setting its state field to false. Because Kubernetes' internal logic defines that for a Pod to be ultimately considered "ready," it must not only pass all its own readiness probes but also satisfy all its defined readiness gate conditions. When the controller sets our custom readiness gate condition status to false, even if the application inside the old Pod is still healthy, Kubernetes will still determine its overall readiness status as false. Once the Pod's readiness status becomes false, Kubernetes' traffic management components—primarily the EndpointSliceController—will immediately respond by removing the Pod's IP address from the list of backend endpoints corresponding to its service. This ensures that all new service requests will no longer be dispatched to the old Pod instance, thus achieving a smooth removal of traffic. After this operation, the GPU defragmentation controller will update the status of the task CR to "Draining OldReplica," an update that signifies that the old instance has been successfully isolated from the service plane.

[0051] It's worth noting that in this embodiment, the "old Pod instance" refers to the original Pod instance running on the source node (i.e., the node with resource fragmentation) and selected as the migration target at the start of the defragmentation process. The Readiness Gate is an advanced feature of Kubernetes Pods that allows external controllers to inject additional custom judgment conditions into the Pod's readiness state, and is a key technology for achieving precise traffic control in this step. A "non-ready old Pod" refers to a Pod instance whose internal containers are still running normally, but whose overall readiness state is artificially set to false, thus isolating it from the service network and preventing it from receiving new traffic. This is a logical state, not a physical destruction.

[0052] In a specific example, after the new replica Pod-alpha-new (located on Node-B) is fully ready, the controller reads the target old Pod instance named Pod-alpha (located on Node-A) from the task CR. The controller injects a readinessGate: {conditionType: "chaos.expert.com / defrag-in-progress"} into the definition of Pod-alpha. Then, it immediately updates the status of Pod-alpha, adding a new condition: {type: "chaos.expert.com / defrag-in-progress", status: "False"}. Kubernetes detects this update, and even though Pod-alpha's own health checks still pass, its overall readiness status is immediately set to false because it fails to meet this new readiness gate. Next, the EndpointSlice controller removes Pod-alpha's IP address from the list of backend endpoints for its associated service. At this point, all new requests to that service will only flow to the ready Pod-alpha-new, while Pod-alpha no longer receives new traffic, quietly awaiting the final scaling-down instruction, becoming a non-ready old Pod.

[0053] In step S160, the non-ready old Pods undergo precise scaling down, resource release, and task termination to obtain a terminated GPU defragmentation task. That is, after the old Pod instance is successfully isolated from service traffic and placed in a non-ready state, precise scaling down, resource release, and task termination are performed on the non-ready old Pod to ultimately obtain a terminated GPU defragmentation task. This step is the closed-loop action of the entire scaling-up-then-scaling strategy. It completes the final release of resources in a highly deterministic manner that conforms to cloud-native best practices. At this time, the cluster is in a temporary overprovisioned state with one more replica than normal. This step aims to restore the number of replicas of the workload to its original normal state and ensure that only the old Pods we want to delete are precisely deleted, rather than randomly deleting or mistakenly deleting newly started Pods. Therefore, the core purpose of this step is to leverage the built-in controller behavior of Kubernetes, combined with the groundwork laid in the previous step, to achieve precise removal of specific Pod instances and bring the entire defragmentation task's lifecycle to a close.

[0054] like Figure 3As shown, this step is implemented by the GPU defragmentation controller making another simple API call to trigger a chain reaction. After the old Pod traffic is drained, the controller restores the replica count field of the target workload to its original number. Specifically, the controller reads the original replica count recorded in the GPU defragmentation task CR, and then sends an update request to the Kubernetes API Server for the target workload, modifying its replica count (spec.replicas) value from the current "original replica count + 1" back to the initial "original replica count". Once this scaling-down command is issued, the Kubernetes native workload controller (such as the Deployment controller) will immediately detect this change, finding that the number of currently running Pods is one more than the expected number, thus initiating its built-in, standard scaling-down logic. The effectiveness of this mechanism depends on the scaling-down strategy of the Kubernetes core controller, which prioritizes deleting Pods that do not meet the readiness conditions when performing scaling-down. Since we set the specific old Pod instance to a non-ready state (Ready=False) through the ready gate in the previous step, while the newly created Pod replica was fully ready (Ready=True), the system will automatically and accurately select the target to be deleted when the scaling-down logic starts. Kubernetes follows a graceful termination process when deleting the Pod, giving the Pod process time to complete its cleanup. After issuing the scaling-down command, the GPU defragmentation controller will continue to monitor the lifecycle of the old Pod until it is confirmed that it has been completely deleted and the GPU resources it occupied have been released on the node. Once everything is confirmed, the controller will update the GPU defragmentation task CR one last time, setting its status to a final state, such as Successed. If any error occurs during this process or a preceding step fails, the task status may be marked as Failed or Timeout. At this point, the lifecycle of the GPU defragmentation task is complete.

[0055] In this embodiment, precise scaling down refers to ensuring that only, and necessarily pre-selected, specific Pod instances are deleted from multiple Pod replicas. This precision is entirely based on the utilization of Kubernetes' native behavior and controller scheduling policies. Resource release is the physical result of scaling down; that is, the CPU, memory, and especially exclusive GPU card resources occupied by the terminated Pod are returned to the node, making them available for scheduling and use by new Pods. A terminated GPU defragmentation task is a CR instance that represents the complete end of its task lifecycle. It is in a terminated state such as success, failure, or timeout. The controller will no longer coordinate any operations on it, but it will be retained in the system as a historical record that can be audited and traced.

[0056] In a specific example, in the previous stage, the old Pod-alpha (located on Node-A) was declared not ready, and the system was running two replicas: the healthy Pod-alpha-new and the not ready Pod-alpha. At this point, the GPU defragmentation controller reads from the task CR that the original replica count is 1, so it changes the replica count of the Deployment to which Pod-alpha belongs from 2 back to 1. Upon receiving this instruction, the Kubernetes Deployment controller needs to delete a Pod. Faced with two options—the ready Pod-alpha-new and the not ready Pod-alpha—based on its internal scaling-down priority rules, it will unhesitatingly choose to delete the latter, i.e., Pod-alpha. Pod-alpha begins graceful termination, the one GPU it occupied on Node-A is released, and the fragmented resources on Node-A are fully consolidated. After the controller confirms that Pod-alpha has disappeared from the cluster, it updates the status of the defrag-task-xxxx task to "Succeeded." Thus, a successful defragmentation operation is declared complete, resulting in a terminated GPU defragmentation task.

[0057] According to another aspect of this application, a Kubernetes-based intelligent scheduling system for GPU resource fragments is also provided, which is used to execute the Kubernetes-based intelligent scheduling method for GPU resource fragments as described above.

[0058] It is worth mentioning that the Kubernetes-based intelligent scheduling system for GPU resource fragmentation according to the embodiments of this application can be implemented in various terminal devices, such as Kubernetes-based intelligent scheduling servers for GPU resource fragmentation. In one example, the Kubernetes-based intelligent scheduling system for GPU resource fragmentation according to the embodiments of this application can be integrated into the terminal device as a software module and / or hardware module. For example, the Kubernetes-based intelligent scheduling system for GPU resource fragmentation can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the Kubernetes-based intelligent scheduling system for GPU resource fragmentation can also be one of many hardware modules of the terminal device.

[0059] Alternatively, in another example, the Kubernetes-based GPU resource fragmentation intelligent scheduling system and the terminal device can also be separate devices, and the Kubernetes-based GPU resource fragmentation intelligent scheduling system can connect to the terminal device via wired and / or wireless networks and transmit interactive information in accordance with an agreed data format.

[0060] Below, for reference Figure 5 This describes an electronic device according to embodiments of the present application.

[0061] Figure 5 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0062] like Figure 5 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0063] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0064] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the Kubernetes-based GPU resource fragmentation intelligent scheduling method and / or other desired functions described in the various embodiments of this application above. The computer-readable storage medium may also store various content such as migration candidates and real-time cluster status.

[0065] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0066] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0067] The output device 14 can output various information to the outside, including completed GPU defragmentation tasks. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices.

[0068] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0069] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the Kubernetes-based GPU resource fragmentation intelligent scheduling method according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0070] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0071] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the Kubernetes-based GPU resource fragment intelligent scheduling method according to various embodiments of this application as described in the "Exemplary Methods" section above.

[0072] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications made by those skilled in the art to the technical solution of the present invention by means of equivalent substitution or equivalent transformation without departing from all the technical content described in the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for intelligent scheduling of GPU resource fragments based on Kubernetes, characterized in that, Includes the following steps: Based on real-time cluster status monitoring and fragmentation risk identification, migration candidates are obtained. Based on the source workload definition, the migration candidate objects are initialized and persisted with migration tasks to obtain active GPU defragmentation tasks (CRs). The system expands capacity based on the active GPU defragmentation task CR and selectively schedules new Pod replicas to obtain the new Pods that have been scheduled. Perform new replica health checks and traffic switching preparations on the scheduled new Pod to obtain a ready new Pod; Smoothly remove traffic from old Pod instances and isolate them from old replicas to obtain non-ready old Pods; Perform precise scaling down, resource release, and task termination on non-ready old Pods to obtain terminated GPU defragmentation tasks.

2. The Kubernetes-based intelligent scheduling method for GPU resource fragmentation according to claim 1, characterized in that, The real-time cluster status includes a list of nodes and their GPU usage, as well as a list of running Pods and their Pod specifications.

3. The Kubernetes-based intelligent scheduling method for GPU resource fragments according to claim 2, characterized in that, Cluster status monitoring and fragmentation risk identification based on real-time cluster status are used to obtain migration candidates, including: Filter the node list to obtain a set of GPU nodes by selecting nodes whose GPU usage has a greater than zero number of allocated GPUs; Based on the GPU node set, Pod instances that meet preset conditions are filtered from the list of running Pods to obtain a candidate Pod set; The candidate Pod set is sorted to obtain a sorted list of GPU Pods; Simulated scheduling calculations are performed on each candidate Pod in the sorted GPU Pod list to obtain simulated scheduling results; Based on the simulation scheduling results and the source node GPU status, the migration benefit is evaluated and a final decision is made to obtain the migration candidates.

4. The Kubernetes-based intelligent scheduling method for GPU resource fragments according to claim 3, characterized in that, Simulated scheduling calculations are performed on each candidate Pod in the sorted GPU Pod list to obtain simulated scheduling results, including: Extract the first candidate Pod from the sorted list of GPU Pods; Extract the source node of the first candidate Pod and the number of GPU resources it requests; Based on the real-time cluster state, a hypothetical cluster state is generated, wherein, in the hypothetical cluster state, the GPU resources on the source node belonging to the first candidate Pod are marked as available; The first candidate Pod and the hypothetical cluster state are input into the simulation scheduler to obtain the simulated scheduling target node; The target node and the first candidate Pod are encapsulated to obtain the simulation scheduling result.

5. The Kubernetes-based intelligent scheduling method for GPU resource fragments according to claim 4, characterized in that, Inputting the first candidate Pod and the hypothetical cluster state into the simulation scheduler to obtain the simulated scheduling target node includes: The simulated scheduler uses a compact scheduling strategy to process the first candidate Pod and the hypothetical cluster state to select the node with the fewest remaining GPU resources from the hypothetical cluster state that can satisfy the request of the first candidate Pod as the simulated scheduling target node.

6. The Kubernetes-based intelligent scheduling method for GPU resource fragments according to claim 1, characterized in that, Based on the source workload definition, the migration candidate objects are initialized and persisted with migration tasks to obtain active GPU defragmentation tasks (CRs), including: The migration candidate objects are subjected to migration context data aggregation to obtain enriched migration contexts; Based on the enriched migration context and source workload definition, GPU defragmentation task resources are synthesized to obtain the synthesized task CR; The synthesized task CR is persisted and its lifecycle is activated to obtain an active GPU defragmentation task CR.

7. The Kubernetes-based intelligent scheduling method for GPU resource fragments according to claim 6, characterized in that, The synthesized task CR is persisted and its lifecycle is activated to obtain an active GPU defragmentation task CR, including: The synthesized task CR is object serialized to obtain a serialized synthesized task CR; Using the serialized and synthesized task CR as the request body, a creation request is sent to the Kubernetes API Server to obtain the active GPU defragmentation task CR.

8. The Kubernetes-based intelligent scheduling method for GPU resource fragments according to claim 1, characterized in that, Scaling is performed based on the active GPU defragmentation task CR, and new Pod replicas are scheduled in a targeted manner to obtain the new Pods that have been scheduled, including: Workload expansion and identifier injection are performed based on the active GPU defragmentation task CR to obtain an augmented workload definition and an updated GPU defragmentation task CR. Based on the augmented workload definition, instantiate native Pod replicas to obtain traceable Pods to be scheduled; Perform deterministic scheduling and binding based on task coordination on traceable Pods to obtain the new Pods that have been scheduled.

9. A GPU resource fragmentation intelligent scheduling system based on Kubernetes, characterized in that, The Kubernetes-based GPU resource fragment intelligent scheduling system is used to execute the Kubernetes-based GPU resource fragment intelligent scheduling method as described in any one of claims 1 to 8.

Images