Job scheduling method and system
By constructing a global resource view and job queue status, and dynamically switching node strategies, the cluster resource scheduler supports multiple granularities, improving resource utilization and reducing operational complexity, thus solving the problem of resource idleness and job queuing coexisting.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGSU XCMG STATE KEY LAB TECH CO LTD
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-14
AI Technical Summary
In existing technologies, cluster resource schedulers cannot support multiple resource allocation granularities simultaneously, resulting in low resource utilization and high operational complexity, and failing to meet scheduling needs of various different granularities.
By constructing a global resource view and job queue status, and dynamically switching node working strategies, the same cluster can simultaneously support two resource allocation granularities: nodes and resources. By utilizing resource usage status such as CPU utilization, memory usage, GPU memory, and I/O load, node resource allocation can be optimized.
It improves resource utilization efficiency, resolves the contradiction between resource idleness and job queuing, reduces operation and maintenance complexity, and avoids the technical burden of maintaining multiple clusters.
Smart Images
Figure CN121858291A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a job scheduling method and system, belonging to the field of high-performance computing cluster resource management technology. Background Technology
[0002] Because the computing power of a single computer is limited, clusters are typically used to handle large-scale computational tasks. A cluster is a supercomputer composed of multiple computers interconnected by a high-speed network. Multiple users request computing resources on demand by sharing the cluster, thereby completing their computing tasks. Although the cluster can allocate available computing resources according to user requests, its resources are also finite and cannot meet the resource needs of all users at all times. Therefore, the cluster needs to use a job scheduler to rationally schedule and allocate computing resources. This ensures full utilization of resources and improves efficiency, while also guaranteeing fair use among users and preventing situations where resources are occupied by a few users for extended periods while others have access to them. Currently, several job schedulers are used in this context, such as Slurm Workload Manager (SLURM, Slurm is an open-source job scheduling system), Platform LSF (LSF), Portable Batch System (PBS), and Univa Grid Engine (UGE).
[0003] Although there are many types of job schedulers, their main functions are largely the same: solving the problem of resource scheduling and allocation. A typical cluster can range in size from a single node to tens of thousands of nodes. To facilitate the management of these nodes, a common practice is to group them into partitions or queues, each consisting of one or more nodes. The job scheduler is responsible for allocating the computing resources within a queue to the jobs submitted to that queue according to a predetermined scheduling strategy. Once a job has acquired computing resources, it can begin running and output its results.
[0004] Existing job scheduling methods, for the same cluster, either choose node-level scheduling or resource-level scheduling. If multiple clusters need to be deployed to meet various scheduling requirements, each cluster must choose a different scheduling strategy. The resources of different clusters cannot communicate with each other, require independent maintenance, and have low resource utilization, affecting the actual usage effect. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a job scheduling method and system that breaks the limitation of the single scheduling strategy, enables the same cluster to support two resource allocation granularities at the same time, and can determine and switch the working strategy of each node according to the global resource view and job queue status, thereby optimizing the allocation of node resources. In this application, the node strategy can be switched according to the actual working situation, which improves resource utilization efficiency, solves the contradiction between resource idleness and job queuing, reduces operation and maintenance complexity, and avoids the technical burden of maintaining multiple clusters.
[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution: In a first aspect, the present invention provides a job scheduling method, comprising: Collect the resource usage status of each node and build a global resource view based on the resource usage status of each node; Obtain the job queue status and determine the working strategy for each node based on the global resource view and the job queue status; Obtain the job status and current strategy of each node; wherein, the job strategy and the current strategy both include node mode and resource mode; Based on the job status of each node, the current strategy of each node is switched to the working strategy to complete job scheduling.
[0007] Furthermore, the resource usage status includes CPU utilization, memory usage, GPU memory usage, and I / O load.
[0008] Furthermore, determining the working strategy for each node based on the global resource view and job queue status includes: Obtain the total number of nodes, and calculate the node-based scheduling policy pressure value for each node based on the job queue status and the total number of nodes; Calculate the resource-based scheduling policy pressure value for each node based on the global resource view; The node-based scheduling policy pressure value of each node is compared with the corresponding resource-based scheduling policy pressure value, and the working policy of each node is determined based on the comparison result.
[0009] Furthermore, determining the working strategy of each node based on the comparison results includes: If the node's node-based scheduling policy pressure value is greater than the corresponding resource-based scheduling policy pressure value, then the node's working policy is determined to be node mode. If the node's node-based scheduling policy pressure value is less than the corresponding resource-based scheduling policy pressure value, then the node's working policy is determined to be resource mode. If the node's node-based scheduling policy pressure value equals the corresponding resource-based scheduling policy pressure value, then the task ends.
[0010] Furthermore, the operational status of each node includes both a working status and an idle status.
[0011] Furthermore, the step of switching the current strategy of each node to a working strategy based on the working status of each node includes: If a node's job status is idle, immediately switch the node's current policy to a working policy. If a node's job status is active, wait for the node's job status to change to idle before switching the node's current policy to active policy.
[0012] Furthermore, switching the current policy of each node to the working policy includes: If the node's current strategy is the same as the working strategy, the work ends; If the node's current strategy is node mode and the node's working strategy is resource mode, then the node's resources are isolated, a resource isolation sandbox is created, and the node's resources are divided into allocable fragments based on the resource isolation sandbox. If the node's current policy is resource mode and the node's working policy is node mode, then isolate the node's resources, destroy the resource isolation sandbox, and release the node's resources.
[0013] In a second aspect, the present invention provides a job scheduling system based on the job scheduling method described in the first aspect, including a resource coordinator, wherein the resource coordinator integrates a dynamic partition manager, and each node includes a node agent; The resource coordinator is used to collect the resource usage status of each node, construct a global resource view based on the resource usage status of each node, obtain the job queue status, and determine the working strategy of each node based on the global resource view and the job queue status. The node agent is used to obtain the node's job status and current policy; and to switch the node's current policy to the working policy based on the node's job status. The dynamic partition manager is used to run multiple virtual partitions within a node; each virtual partition corresponds to a different working strategy.
[0014] Furthermore, the resource coordinator collects the resource usage status of each node through the Slurm API, sysfs, and nvidia-smi.
[0015] Furthermore, the dynamic partition manager includes a dynamic partition update interface.
[0016] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: This job scheduling method breaks the limitation of a single scheduling strategy, enabling the same cluster to support two resource allocation granularities simultaneously. It can also determine and switch the working strategy of each node based on the global resource view and job queue status, thereby optimizing the allocation of node resources. In this application, the node strategy can be switched according to the actual working situation, improving resource utilization efficiency, resolving the contradiction between resource idleness and job queuing, reducing operation and maintenance complexity, and avoiding the technical burden of maintaining multiple clusters. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating a job scheduling method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the process of switching the current policy of a node to the working policy according to an embodiment of the present invention. Detailed Implementation
[0018] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.
[0019] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Example 1:
[0020] like Figures 1-2 As shown, the present invention provides a job scheduling method, including: Collect the resource usage status of each node and build a global resource view based on the resource usage status of each node; Obtain the job queue status and determine the working strategy for each node based on the global resource view and the job queue status; Obtain the job status and current strategy of each node; wherein, the job strategy and the current strategy both include node mode and resource mode; Based on the job status of each node, the current strategy of each node is switched to the working strategy to complete job scheduling.
[0021] Specifically, the resource coordinator collects the resource utilization of each node at a rate of seconds using tools such as the Slurm API (Slurm's interface), sysfs (a memory-based virtual file system), and nvidia-smi (NVIDIA's system management interface) to build a global resource view. When the node switching process is completed, the node's registration status is verified to determine whether the switching process was successful. If the process of switching each node's current policy to the working policy fails, a rollback operation is performed, and the resource coordinator is notified of the failure. Figure 2 Strategy 1 and Strategy 2 are two of the node mode and resource mode, respectively.
[0022] In this embodiment, the resource usage status includes CPU utilization, memory usage, GPU memory usage, and I / O load.
[0023] In this embodiment, determining the working strategy of each node based on the global resource view and job queue status includes: Obtain the total number of nodes, and calculate the node-based scheduling policy pressure value for each node based on the job queue status and the total number of nodes; Calculate the resource-based scheduling policy pressure value for each node based on the global resource view; The node-based scheduling policy pressure value of each node is compared with the corresponding resource-based scheduling policy pressure value, and the working policy of each node is determined based on the comparison result.
[0024] The process of determining the working strategy of each node based on the comparison results includes: If the node's node-based scheduling policy pressure value is greater than the corresponding resource-based scheduling policy pressure value, then the node's working policy is determined to be node mode. If the node's node-based scheduling policy pressure value is less than the corresponding resource-based scheduling policy pressure value, then the node's working policy is determined to be resource mode. If the node's node-based scheduling policy pressure value equals the corresponding resource-based scheduling policy pressure value, then the task ends.
[0025] The operational status of each node includes both working status and idle status.
[0026] The step of switching the current policy of each node to the working policy based on the working status of each node includes: If a node's job status is idle, immediately switch the node's current policy to a working policy. If a node's job status is active, wait for the node's job status to change to idle before switching the node's current policy to active policy.
[0027] Specifically, the policy switching conditions are triggered based on the global resource view and job queue status: If the node's node-based scheduling policy pressure value is greater than the corresponding resource-based scheduling policy pressure value, then the node's working policy is determined to be node mode. If the node's node-based scheduling policy pressure value is less than the corresponding resource-based scheduling policy pressure value, then the node's working policy is determined to be resource mode. If a node currently has no running jobs, its job status is idle, allowing for immediate job switching. If a node has running jobs, the job status is "working," and immediate job switching is not allowed. The node is then marked as having a delayed switchover. Generate policy switching conditions, including node ID, working policy, and whether there is a delay switch.
[0028] In this embodiment, switching the current policy of each node to the working policy includes: If the node's current strategy is the same as the working strategy, the work ends; If the node's current strategy is node mode and the node's working strategy is resource mode, then the node's resources are isolated, a resource isolation sandbox is created, and the node's resources are divided into allocable fragments based on the resource isolation sandbox. If the node's current policy is resource mode and the node's working policy is node mode, then isolate the node's resources, destroy the resource isolation sandbox, and release the node's resources.
[0029] Specifically, if a node's agent receives a policy switching condition and there is no delay flag, the policy switch is executed immediately: First, a new resource control group is created using Linux cgroups (the control groups in the Linux system) to isolate unused resources. If switching from node mode to resource mode, a resource isolation sandbox is started, dividing node resources (CPU cores, memory, GPU) into allocatable fragments and registering them as members of the resource mode virtual partition with the dynamic partition manager. If switching from resource mode to node mode, the resource isolation sandbox is destroyed, cgroups are released, and the entire node is registered as a member of the node mode virtual partition. The dynamic update interface of the dynamic partition manager is called, and the dynamic partition manager updates the resource mapping table to complete the virtual partition migration. Finally, the dynamic partition manager reports the switch result to the resource coordinator.
[0030] If the node's agent receives the policy switching condition and a delay flag exists, the policy switching will be performed after the node's job has finished: First, the node agent uses the cgroups delegation controller mechanism to ensure that job resource boundaries are not affected by policy switching; and marks the resources for executing jobs to prevent them from being suspended by the new policy; the new policy only applies to subsequently submitted jobs, while historical jobs continue to run according to the original policy until completion.
[0031] Meanwhile, the resource coordinator continuously monitors the resource utilization and job response time of the nodes after the switch, and, combined with user preference factors, periodically calls the method steps of this application to achieve adaptive optimization of the strategy switch.
[0032] This application breaks the limitation of a single scheduling strategy, enabling the same cluster to support two resource allocation granularities simultaneously. It can also determine and switch the working strategy of each node based on the global resource view and job queue status, thereby optimizing the allocation of node resources. In this application, the node strategy can be switched according to the actual working situation, improving resource utilization efficiency, resolving the contradiction between resource idleness and job queuing, reducing operation and maintenance complexity, and avoiding the technical burden of maintaining multiple clusters. Example 2:
[0033] This invention provides a job scheduling system based on the job scheduling method described in Embodiment 1, including a resource coordinator, wherein the resource coordinator integrates a dynamic partition manager, and each node includes a node agent; The resource coordinator is used to collect the resource usage status of each node, construct a global resource view based on the resource usage status of each node, obtain the job queue status, and determine the working strategy of each node based on the global resource view and the job queue status. The node agent is used to obtain the node's job status and current policy; and to switch the node's current policy to the working policy based on the node's job status. The dynamic partition manager is used to run multiple virtual partitions within a node; each virtual partition corresponds to a different working strategy.
[0034] In this embodiment, the resource coordinator collects the resource usage status of each node through the Slurm API, sysfs, and nvidia-smi, and the dynamic partition manager includes a dynamic partition update interface.
[0035] Specifically, the Resource Orchestrator (RO) is deployed as an independent daemon on at least one physical or virtual server independent of the management node. Through the Slurm API, the slurm.conf interface (Slurm's core configuration file), and custom RPCs (Remote Procedure Calls) and slurmctld (Slurm's core management daemon), it collects real-time data on the resource usage status of each node in the cluster (CPU utilization, memory usage, GPU memory, I / O load, etc.) to build a global resource view. Using a pressure-based policy switching decision algorithm, it dynamically determines the optimal scheduling policy (i.e., working policy) for each node, generates a policy switching command, and sends the command to the target node's Node Agent (NA). After the node policy switching is complete, it synchronously updates the Slurm logical partition configuration, achieving dynamic switching of node policies.
[0036] The Node Agent (NA) runs as a lightweight user-space process on each compute node, deployed in isolation from slurmd (the core daemon process in Slurm that runs on compute nodes). It has fault tolerance and self-recovery capabilities. It receives policy switching instructions from the RO, performs local policy switching operations, and reports the switch confirmation to the RO. The specific operations of local policy switching include: dynamically adjusting the Linux cgroups configuration to isolate node resources; creating a target policy sandbox through namespace isolation technology to realize logical changes in resource modes; and calling the Dynamic Partition Update (DPU) interface of the dynamic partition manager to logically migrate the node from the original partition to the target partition, thereby avoiding restarting the Slurmd daemon process.
[0037] The Dynamic Partition Manager (DPM) is integrated within the RO (Remote Operating System) and maintains a resource mapping table that dynamically maps physical nodes to logical virtual partitions. Each virtual partition corresponds to a scheduling policy. By extending Slurm's plugin mechanism, the cluster can accept DPM partition member change requests at runtime without restarting the service, ensuring that the job scheduler always assigns jobs based on the latest partition configuration. Two sets of partition instances run simultaneously within the management node. The first set of partition instances is configured with a node-mode policy and manages node-based virtual partitions through the first configuration file slurm.conf.1. The second set of partition instances is configured with a resource-mode policy and manages resource-based virtual partitions through the second configuration file slurm.conf.2. Both sets of partition instances share the same SlurmDBD database but use isolated table prefixes to avoid transaction conflicts.
[0038] Based on the traditional single-management-end architecture, this invention innovatively proposes a logically isolated dual-management-end service architecture, based on which a high-performance computing scheduling system that supports seamless coexistence of dual policies for nodes and resources is realized.
[0039] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0040] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0041] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0042] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0043] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A job scheduling method, characterized in that, include: Collect the resource usage status of each node and build a global resource view based on the resource usage status of each node; Obtain the job queue status and determine the working strategy for each node based on the global resource view and the job queue status; Obtain the job status and current strategy of each node; wherein, the job strategy and the current strategy both include node mode and resource mode; Based on the job status of each node, the current strategy of each node is switched to the working strategy to complete job scheduling.
2. The job scheduling method according to claim 1, characterized in that, The resource usage status includes CPU utilization, memory usage, GPU memory usage, and I / O load.
3. The job scheduling method according to claim 1, characterized in that, The process of determining the working strategy for each node based on the global resource view and job queue status includes: Obtain the total number of nodes, and calculate the node-based scheduling policy pressure value for each node based on the job queue status and the total number of nodes; Calculate the resource-based scheduling policy pressure value for each node based on the global resource view; The node-based scheduling policy pressure value of each node is compared with the corresponding resource-based scheduling policy pressure value, and the working policy of each node is determined based on the comparison result.
4. The job scheduling method according to claim 3, characterized in that, The process of determining the working strategy of each node based on the comparison results includes: If the node's node-based scheduling policy pressure value is greater than the corresponding resource-based scheduling policy pressure value, then the node's working policy is determined to be node mode. If the node's node-based scheduling policy pressure value is less than the corresponding resource-based scheduling policy pressure value, then the node's working policy is determined to be resource mode. If the node's node-based scheduling policy pressure value equals the corresponding resource-based scheduling policy pressure value, then the task ends.
5. The job scheduling method according to claim 4, characterized in that, The operational status of each node includes both working status and idle status.
6. The job scheduling method according to claim 5, characterized in that, The step of switching the current policy of each node to the working policy based on the working status of each node includes: If a node's job status is idle, immediately switch the node's current policy to a working policy. If a node's job status is active, wait for the node's job status to change to idle before switching the node's current policy to active policy.
7. The job scheduling method according to claim 1, characterized in that, The step of switching the current policy of each node to the working policy includes: If the node's current strategy is the same as the working strategy, the work ends; If the node's current strategy is node mode and the node's working strategy is resource mode, then the node's resources are isolated, a resource isolation sandbox is created, and the node's resources are divided into allocable fragments based on the resource isolation sandbox. If the node's current policy is resource mode and the node's working policy is node mode, then isolate the node's resources, destroy the resource isolation sandbox, and release the node's resources.
8. A job scheduling system, based on the job scheduling method according to any one of claims 1 to 7, characterized in that, It includes a resource coordinator, which integrates a dynamic partition manager, and each node includes a node agent; The resource coordinator is used to collect the resource usage status of each node, construct a global resource view based on the resource usage status of each node, obtain the job queue status, and determine the working strategy of each node based on the global resource view and the job queue status. The node agent is used to obtain the node's job status and current policy; and to switch the node's current policy to the working policy based on the node's job status. The dynamic partition manager is used to run multiple virtual partitions within a node; each virtual partition corresponds to a different working strategy.
9. The job scheduling system according to claim 8, characterized in that, The resource coordinator collects the resource usage status of each node through the Slurm API, sysfs, and nvidia-smi.
10. The job scheduling system according to claim 8, characterized in that, The dynamic partition manager includes a dynamic partition update interface.