Slurm cluster servitization dynamic management method and system, medium and product

Abstract

The invention provides a Slurm cluster servitization dynamic management method and system, a medium and a product, and the method comprises the steps: submitting a custom resource CRD based on Kubernetes Operator, and defining the configuration information of a Slurm cluster; the controller schedules and adjusts the computing nodes according to the scheduling strategy, automatically generates a configuration file required by the Slurm cluster, and deploys the Slurm service; a user submits a job through the Slurm client, and knows the state of the job through the Slurm monitoring service; and the controller monitors the change of the custom resource CRD. According to the invention, a service dynamic management method for the Slurm cluster is introduced through the Kubernetes Operator, and the technical problem that it is difficult for a user to construct and manage the Slurm cluster efficiently and autonomously in a cloud multi-user environment in the existing Slurm cluster is solved.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a Slurm cluster service-oriented dynamic management method and system, medium and product. Background Art

[0002] With the rapid development of cloud computing technology, more and more enterprises and scientific research institutions are gradually migrating high-performance computing (HPC) clusters and distributed computing platforms to cloud environments to take full advantage of the cloud platform's elastic expansion, on-demand use, and resource sharing. Especially in industries with intensive large-scale scientific research computing and complex computing tasks, such as life sciences, meteorological simulation, engineering simulation, and artificial intelligence, cloud-based HPC clusters can not only significantly reduce hardware and operation and maintenance costs, but also quickly adapt to the diverse needs of users.

[0003] In a cloud environment, dynamic resource management, flexible scheduling, and multi-user isolation are particularly important. In order to meet the computing needs of different teams, projects, and departments, the cloud platform needs to be able to provide efficient resource scheduling and task management capabilities to ensure that computing resources can be allocated on demand and used independently by different users, and dynamically adjusted in a timely manner when resource demand fluctuates. However, as the business of enterprises grows and the scale of scientific research expands, a single cluster environment is often unable to meet the growing multi-tenant needs and dynamic computing tasks. Therefore, how to efficiently manage distributed computing resources through a cloud platform, ensure maximum resource utilization, and ensure resource isolation and independent operation of multiple users has become a core issue in cloud HPC cluster management.

[0004] Currently, Slurm, as an open source workload management system, has been widely used in task scheduling and resource management of HPC clusters and distributed systems.

[0005] Although Slurm, as an open source and efficient workload management tool, performs well in scheduling and managing large-scale computing resources, the existing Slurm deployment and management model still has many shortcomings in multi-user, multi-cluster and cloud scenarios, which are specifically manifested in the following aspects: (1) Lack of automatic expansion and reduction capabilities: In the traditional Slurm deployment mode, resource management and allocation are usually static. Administrators need to manually expand or reduce computing nodes when the cluster size changes, such as adding computing nodes during peak loads and reducing nodes to save resources when the load is light. In a cloud environment, the user's computing task load may be highly volatile. Manual resource management is not only inefficient, but also prone to resource waste or insufficient computing resources. The existing Slurm scheduling mechanism cannot automatically allocate resources according to load changes, and it is difficult to meet the requirements of on-demand use and elastic expansion under the cloud platform.

[0006] (2) Insufficient multi-tenant support and isolation: Slurm's traditional cluster management method is to provide a shared computing resource pool for one or more users, and lacks native support for multi-tenant environments. In cloud scenarios, different users, projects, or teams need to independently schedule tasks and use resources in the same cluster environment, which requires the system to be able to isolate resources from different users to prevent resource contention and task interference between different users. Although the traditional Slurm architecture can limit resource usage by manually setting job priorities and quotas, this method relies on manual intervention by administrators and is difficult to efficiently respond to the dynamic needs of large-scale, multi-user environments.

[0007] (3) Cluster management is complex and lacks automation capabilities: Traditional Slurm cluster deployment and management usually requires cluster administrators to manually configure and schedule. Although this method is relatively effective in small-scale clusters, manual management and monitoring of multiple clusters becomes increasingly complex in large-scale, complex cluster environments. For example, when a node or service fails, the administrator needs to manually diagnose the problem and perform repair operations, lacking automatic fault tolerance and self-healing capabilities. For multi-cluster environments in cloud platforms, administrators need to manage multiple Slurm clusters at the same time, which increases management difficulty and reduces the scalability of the system.

[0008] In summary, with the emergence of multi-user, multi-tenant cloud application scenarios, traditional Slurm cluster management faces some significant challenges and shortcomings. When facing large-scale, multi-cluster environments, it is difficult to effectively respond to the challenges of dynamic resource demands and task scheduling, and it is difficult to efficiently manage cluster expansion, node scheduling and fault recovery. At the same time, job scheduling is also relatively rigid, and it is difficult to flexibly respond to the needs of multi-tenant resource isolation and heterogeneous resources. Summary of the invention

[0009] The purpose of the present invention is to provide a Slurm cluster service-oriented dynamic management method and system, medium, and product, and to introduce a Slurm cluster service-oriented dynamic management method through Kubernetes Operator to solve the technical problem that it is difficult for users to efficiently and independently build and manage Slurm clusters in a cloud-based multi-user environment.

[0010] As a mature container orchestration platform, Kubernetes has powerful automated management capabilities, and Kubernetes Operator can extend the Kubernetes API to automatically manage complex applications. Combining the automation features of Kubernetes, using Operator to dynamically manage multiple Slurm clusters in a cloud environment helps improve resource utilization and cluster management efficiency.

[0011] Based on this, the technical solution of the present invention is as follows: A Slurm cluster service-oriented dynamic management method, comprising: S1. Submit custom resource CRD based on KubernetesOperator to define the configuration information of Slurm cluster; S2. Implement a custom controller for custom resource CRD based on Kubernetes Operator, the controller schedules and adjusts computing nodes according to the scheduling strategy, automatically generates configuration files required by the Slurm cluster, and deploys Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service, and Slurm database service; wherein the Slurm control service, Slurm execution node service, Slurm client, and Slurm database service are connected to PVC persistent storage; S3,Users submit jobs through the Slurm client and learn the job status through the Slurm monitoring service; S4, the controller monitors the changes of the custom resource CRD and automatically performs expansion, reduction, fault recovery and monitoring operations of the nodes of the Slurm cluster; S5. When the Slurm cluster is no longer needed, request to delete the entire cluster by deleting the custom resource CRD.

[0012] Furthermore, the configuration information in step S1 includes the number of nodes, resource requirements, and scheduling strategies of the Slurm cluster.

[0013] Furthermore, the scheduling and adjustment method of the computing nodes in step S2 includes: S201, node resource filtering, preliminarily screening available computing nodes according to the resource requirements in the custom resource CRD; S202, after selecting suitable computing nodes, prioritize the computing nodes according to the sorting rules; S203, after the priority is sorted, the node label is set to indicate whether the computing node is occupied, so as to realize exclusive node allocation; S204. When the node resources in the cluster are insufficient to meet the resource demand, node expansion is automatically triggered according to the task load. When the resource demand decreases, the computing nodes are automatically reduced.

[0014] Furthermore, in step S2, the controller deploys Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service, and Slurm database service in a Kubernetes Deployment manner.

[0015] On the other hand, the present invention also proposes a Slurm cluster service-oriented dynamic management system, comprising: Control module: As the core management component, it is responsible for the dynamic management of the Slurm cluster. By monitoring the changes of custom resource CRDs, it automatically performs cluster expansion, reduction, fault recovery and monitoring operations; k8s scheduling module: responsible for deploying and managing Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service, and Slurm database service using the native scheduling function of Kubernetes; Slurm dedicated scheduling module: responsible for scheduling and adjusting computing nodes according to the resource requirements and scheduling policies of the custom resource CRD; The k8s scheduling module and the Slurm dedicated scheduling module are controlled and managed by the control module.

[0016] Furthermore, the custom resource CRD defines configuration information of the Slurm cluster, and the configuration information includes the number of nodes, resource requirements, and scheduling strategy of the Slurm cluster.

[0017] Furthermore, the k8s scheduling module deploys Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service, and Slurm database service in a Kubernetes Deployment manner, wherein the Slurm control service, Slurm execution node service, Slurm client, and Slurm database service are connected to PVC persistent storage.

[0018] Furthermore, the scheduling and adjustment of computing nodes by the Slurm dedicated scheduling module include: node resource filtering, preliminary screening of available computing nodes according to resource requirements in the custom resource CRD; after screening out suitable computing nodes, prioritizing the computing nodes according to the sorting rules; after priority sorting, setting node labels to indicate whether the computing nodes are occupied, thereby achieving exclusive node allocation; when the node resources in the cluster are insufficient to meet the resource requirements, node expansion is automatically triggered according to the task load, and when the resource requirements are reduced, the computing nodes are automatically reduced.

[0019] The present invention also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned Slurm cluster service-oriented dynamic management method.

[0020] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the above-mentioned Slurm cluster service-oriented dynamic management method is implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention improves the automation level of Slurm cluster management: Through the Kubernetes Operator, the full life cycle management of the Slurm cluster is realized, including the automatic creation, expansion, reduction, configuration and destruction of the cluster, which reduces manual operations, greatly improves management efficiency and reduces the complexity of operation and maintenance.

[0022] (2) The present invention optimizes resource utilization: The present invention adopts dynamic resource scheduling and node exclusive allocation strategy to ensure that tasks exclusively occupy node resources according to actual needs and avoid resource contention. It also dynamically adjusts resource allocation according to load demand through automatic expansion and reduction mechanism, thereby maximizing resource utilization and avoiding unnecessary waste.

[0023] (3) The present invention improves the reliability and fault tolerance of the cluster: through the sharing mechanism of persistent storage, the task scheduling information and job history data of the Slurm cluster can be persisted. Even when a node fails, it can be automatically restored through the fault self-healing mechanism, thus ensuring the continuity of the cluster and the security of the data.

[0024] (4) The present invention enhances multi-tenant support: The present invention supports resource isolation and cluster management in a multi-tenant environment. Each user can independently create and manage their Slurm cluster, and resources are isolated from each other to avoid resource contention and security issues. This makes the system particularly suitable for multi-user scenarios in cloud environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the deployment process of the Slurm cluster service-oriented dynamic management according to an embodiment of the present invention; Figure 2 It is a schematic diagram of system function modules of an embodiment of the present invention. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0027] First, the abbreviations and key terms involved in the present invention are explained.

[0028] Kubernetes: k8s for short, Kubernetes is an open source container orchestration platform for automating the deployment, expansion and management of containerized applications. It can help users manage container clusters across multiple hosts and achieve automatic deployment, expansion, load balancing and recovery of containers.

[0029] Kubernetes Operator: Kubernetes Operator is a software extension mechanism that helps users manage complex stateful applications. It can automatically manage applications in Kubernetes and handle their lifecycles, such as deployment, expansion, and recovery, by defining custom resources (CRD, Custom Resource Definitions) and custom controllers.

[0030] PVC: PVC is the abbreviation of Persistent Volume Claim, which refers to the persistent volume claim in Kubernetes. PVC allows users to request persistent storage resources (Persistent Volume, PV) and keep data persistent during the life cycle of the container to avoid data loss due to container destruction.

[0031] Slurm: Slurm (Simple Linux Utility for Resource Management) is an open source, flexible, and efficient workload management system for large-scale clusters and high-performance computing (HPC) environments. Slurm provides resource management and task scheduling capabilities for allocating computing resources in a cluster and executing jobs.

[0032] Slurmctld: Slurmctld is the Slurm control service, responsible for managing and coordinating resource allocation, task scheduling, and job queues for the entire Slurm cluster. It is the core component of the Slurm system, responsible for receiving job requests submitted by users and assigning them to available computing nodes in the cluster.

[0033] Slurmd: Slurmd is the Slurm execution node service, responsible for receiving and executing tasks assigned by Slurmctld on specific computing nodes. Each computing node in the Slurm cluster runs a Slurmd process, which is used to actually process jobs and report status.

[0034] Slurmdbd: Slurmdbd is the Slurm database service, which is used to store the historical information of cluster jobs in the database. It is responsible for interacting with the database and saving information such as job status, resources used, and execution time for subsequent statistics and analysis.

[0035] The present invention aims to provide a solution that can automatically manage the life cycle of a Slurm cluster, including automatic expansion, reduction, task scheduling and fault recovery of the cluster, thereby improving the resource utilization of the cluster and the flexibility of the system, reducing manual intervention, and optimizing the management efficiency of the cluster under the cloud platform.

[0036] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0037] The deployment process of the Slurm cluster service-oriented dynamic management based on Kubernetes Operator proposed in the present invention is as follows: Figure 1 As shown, including: ① The user submits the cluster configuration through the custom resource CRD.

[0038] Users submit custom resource CRDs through the Kubernetes interface or API to define the configuration information of the Slurm cluster they need. The configuration information includes the number of cluster nodes, resource requirements (such as CPU, memory, storage, GPU, etc.), and task scheduling strategies. These custom configurations submitted by users through CRDs will serve as the basis for Slurm cluster deployment and resource management.

[0039] ② Adjust nodes according to node scheduling strategy.

[0040] After receiving the CRD configuration submitted by the user, the Kubernetes Operator implements a custom controller for the custom resource CRD, which automatically schedules and adjusts the computing nodes according to the resource configuration. The specific steps of the scheduling strategy are as follows: ●Node resource filtering: First, the Operator will preliminarily screen the available nodes based on the resource requirements defined by the user. Specifically, the Operator will check which computing nodes in the cluster can meet the required resource requests (such as the number of CPU cores, memory size, number of GPUs, etc.), and exclude nodes that do not meet the requirements; ● Priority sorting: After selecting suitable computing nodes, the Operator prioritizes the nodes according to the sorting rules. The sorting rules usually include the current load, health status, network latency, etc. of the node. Nodes with lower loads are given priority to balance the resource usage of the cluster. The current load of the node is measured based on the CPU and memory usage. The health status of the node is determined by continuously monitored indicators, such as CPU temperature, hard disk status, etc. The priority calculation formula is as follows: ; in: : Comprehensive score of node i; :The weights of the three factors satisfy ; : The load factor of node i is calculated by CPU and memory, and the value is: Max(cpu usage, memory usage); : The health factor of node i reflects the overall health status of the node and is calculated based on the hardware status of the node: . It is the value of each health parameter, such as CPU temperature, hard disk status, etc. : Network delay factor of node i, network delay between the node and the control node; ● Exclusive node allocation: After priority sorting, check the node label to confirm whether it is occupied. If no occupied label is checked, add a label to the node to ensure that Slurmd can be exclusively allocated to a computing node. This means that when a cluster's Slurmd is scheduled to a node, the node's resources will not be used by other clusters until the cluster releases them; ●Node expansion and reduction: If the node resources in the cluster are insufficient to meet the task requirements, node expansion is automatically triggered according to the task load, and more computing nodes are dynamically added to the cluster. When the task is completed and the resource requirements are reduced, the computing nodes are automatically reduced to release the resources that are no longer needed to improve the overall resource utilization efficiency.

[0041] ③Generate configuration file.

[0042] The controller reads the user-defined Slurm cluster configuration information and automatically generates the configuration files required by the Slurm cluster. These configuration files include specific parameter settings for control services, computing services, and database services. The generated configuration files provide guidance for each service of the cluster, including cluster resource allocation rules, task scheduling strategies, storage paths, etc.

[0043] ④Deploy the Slurm control service Slurmctld.

[0044] The controller deploys the Slurm control service Slurmctld in Kubernetes Deployment mode according to the generated configuration file. As the core control service of the Slurm cluster, Slurmctld is responsible for handling key functions such as task scheduling and resource allocation. In addition, Slurmctld is connected to the PVC persistent storage to save user data.

[0045] ⑤Deploy the Slurm execution node service Slurmd.

[0046] After the Slurm control service is deployed, the controller first obtains the list of computing nodes to run the Slurmd service, and then deploys the Slurm execution node service Slurmd on each node in the form of Kubernetes Deployment. These services run on each computing node in the cluster and are responsible for executing the tasks received from Slurmctld. At the same time, the Slurmd service shares storage resources with the PVC persistent storage to ensure that intermediate data, calculation logs, etc. during job execution can be saved. Persistent storage can also be used to store input and output data of large computing tasks to avoid data loss due to node failure.

[0047] ⑥Deploy the Slurm client.

[0048] The controller deploys the Slurm client in the form of Kubernetes Deployment through the k8s scheduling module. Users can submit job tasks, query task status and monitor the operation of the cluster through the client. The Slurm client is also connected to the PVC persistent storage to save user-defined task data.

[0049] ⑦Deploy Slurm monitoring service.

[0050] The controller deploys the Slurm monitoring service as a Kubernetes Deployment to continuously monitor the health status and resource usage of the cluster. The monitoring service is responsible for collecting the running status of nodes, task execution, and system resource consumption, and generating real-time reports. When a node or service fails, the monitoring service can automatically trigger the self-healing mechanism through Kubernetes to restart the failed node or service to ensure stable operation of the cluster.

[0051] ⑧Deploy the Slurm database service Slurmdbd.

[0052] The controller block deploys the Slurm database service Slurmdbd as a Kubernetes Deployment and connects to the PVC persistent storage. The Slurmdbd service is responsible for storing the execution history, resource usage, and other related information of all jobs in the cluster. By sharing storage with PVC, users can access job history data at any time for subsequent analysis and optimization of computing tasks.

[0053] The following describes in detail the steps and implementation process for the three operations of creating a cluster, updating a cluster (adding and deleting execution nodes), and deleting a cluster.

[0054] Create a Slurm cluster: Step 1: The user submits a custom resource CRD to define the cluster configuration. The user submits a custom resource CRD, which includes the configuration parameters of the Slurm cluster, through the Kubernetes API or command-line tools (such as kubectl). The user needs to specify the number of control nodes and execution nodes, the resource requirements of each node (such as CPU, memory, GPU, etc.), and the scheduling strategy. For example, the user can create a yaml file to define a Slurm cluster with 1 control node (Slurmctld) and 3 execution nodes (Slurmd). The resource requirements are as follows: each execution node needs to be configured with 1 gpu, and the control service is configured with 2 cpu cores and 8GB of memory.

[0055] Step 2: Operator defines the controller to receive and parse CRD and perform scheduling related operations. The controller monitors the CRD resource changes in the cluster. After receiving the CRD submitted by the user, it starts to parse the configuration, extracts the number of control nodes and execution nodes required by the Slurm cluster and their resource requirements from the configuration, schedules the nodes, and automatically generates a configuration file for the cluster.

[0056] Step 3: Deploy the Slurm control service Slurmctld. The controller first automatically deploys the Slurm control service Slurmctld according to the configuration defined in the CRD. The resources of the control service are allocated according to user requirements, and Slurmctld is connected to the PVC persistent storage to save user-defined data.

[0057] Step 4: Deploy Slurm execution service Slurmd. After the control node is deployed, the controller will automatically deploy multiple execution service Slurmd. These nodes are responsible for executing Slurm job tasks. The resources of each service are allocated according to the user's configuration file and connected to the PVC persistent storage to save the intermediate data and logs of the job.

[0058] Step 5: Deploy the Slurm client. After the controller and execution nodes are deployed, the controller deploys and configures the Slurm client, through which users can submit jobs, query job status, and monitor cluster operations. The Slurm client is also connected to the PVC persistent storage to ensure that all user-submitted job queues and job history data can be saved and managed to ensure that the status and history of tasks can be consistent across different nodes in the cluster.

[0059] Step 6: Deploy Slurm monitoring service. The controller deploys the monitoring service of the Slurm cluster, which is responsible for monitoring the cluster's resource usage, node health status, task execution, etc. Through the monitoring service, users can understand the running status of the cluster in real time.

[0060] Step 7: Deploy the Slurm database service Slurmdbd. The controller automatically deploys the Slurm database service Slurmdbd, which is responsible for recording the cluster's job history, resource usage records, etc.

[0061] Step 8: The cluster is ready and users can start submitting jobs. Once the control nodes, execution nodes, clients, monitoring services, and database services are all deployed, the entire cluster is ready and users can submit jobs through the Slurm client and learn about the job execution status through the monitoring service.

[0062] Update the Slurm cluster (add or delete execution services): Step 1: The user modifies the CRD to add or delete execution nodes. The user wants to add or delete execution nodes in the Slurm cluster according to changes in task load. This can be achieved by updating the replicas field in the CRD. For example, if the user needs to increase the number of execution nodes from 3 to 5, the user only needs to modify the yaml file created previously and update the number of execution nodes in the CRD configuration from 3 to 5.

[0063] Step 2: The controller monitors the CRD update. The controller will continue to monitor the changes of CRD resources. When it detects that the user has updated the cluster configuration (such as changing the number of execution nodes), it will automatically trigger the update operation.

[0064] Step 3: Automatically expand or reduce nodes. Expansion scenario: If the user increases the number of execution nodes (such as from 3 to 5), the controller will automatically deploy new execution nodes Slurmd according to the updated configuration. The new nodes will automatically connect to the Slurm control service Slurmctld, join the existing cluster, and start executing tasks. Reduction scenario: If the user reduces the number of execution nodes, the controller will remove the redundant nodes from the cluster and release the related computing resources. The controller will also ensure that the tasks running on these nodes have been completed or migrated to other nodes to avoid data loss.

[0065] Step 4: The cluster runs smoothly. After the node adjustment is completed, the cluster will continue to run smoothly, the new node will be added to the task scheduling pool, and the deleted node will be completely removed from the cluster. Cluster resources are dynamically adjusted according to the current load demand to ensure efficient operation.

[0066] Deleting a Slurm cluster: Step 1: The user submits a request to delete the cluster. When the user no longer needs to use the Slurm cluster, he can request to delete the entire cluster by deleting the CRD. The user can delete the CRD through the Kubernetes API or command line tools (such as kubectl).

[0067] Step 2: The controller receives the deletion request. When the controller detects that the CRD is deleted, it automatically triggers the cluster deletion operation.

[0068] Step 3: Shut down and remove the Slurm control service and execution service. The controller will first shut down the Slurm control service Slurmctld and stop the cluster's task scheduling. Then, the controller will gradually shut down all the execution services Slurmd. During the shutdown process, the controller will ensure that the tasks running on the execution nodes are properly terminated or migrated to prevent data loss or task failure.

[0069] Step 4: Shut down and remove Slurmdbd and monitoring related services.

[0070] Step 5: Release the persistent storage resource PVC. Once all node services are shut down, the controller will release the PVC persistent storage resource used by the cluster.

[0071] Step 6: The cluster is deleted. When all services and storage resources are released, the life cycle of the entire Slurm cluster ends. The user does not need to perform any additional operations, and the system will automatically clean up and release all resources.

[0072] Through the above method, the present invention achieves the following technical effects: (1) Quickly build a Slurm cluster: Users only need to call a simple interface to complete the creation of a Slurm cluster in a short time. By defining custom resources (CRDs), users can specify the number of computing nodes in the cluster, the configuration of control services, and the task scheduling strategy, and the Kubernetes Operator automatically completes the cluster construction and resource configuration according to user needs, greatly simplifying the traditional cumbersome manual deployment process. This automated deployment method allows users to quickly and flexibly adjust the cluster size according to task requirements without relying on cluster administrators to perform complex configurations or wait for long debugging. Users can instantly create or modify cluster configurations according to their own workload requirements, thereby improving resource utilization and computing efficiency.

[0073] (2) Autonomous control and resource isolation: In a multi-tenant cloud environment, resource isolation and autonomous control are important safeguards to ensure that different users do not interfere with each other. The present invention ensures that each user can independently manage their Slurm cluster and computing tasks through Kubernetes' multi-tenant support and custom scheduling strategies. Each user's Slurm cluster is deployed in an independent Kubernetes namespace, and uses custom resource CRDs to define their own cluster configurations and task scheduling strategies. The Kubernetes Operator ensures that each tenant's computing tasks run on its exclusive computing nodes, avoiding resource contention and task interference. At the same time, through the Operator's automated management, users can adjust task priorities and resource allocation strategies at any time, enjoy complete autonomous control, and reduce dependence on cluster administrators.

[0074] (3) Automation of cluster lifecycle management: The present invention introduces the automated lifecycle management capability of the Slurm cluster through the Kubernetes Operator, covering the full lifecycle operations of the cluster, such as creation, operation, expansion, reduction, and destruction. In a multi-tenant environment, each user's cluster requirements and computing tasks are different. The present invention can automatically manage the lifecycle of each user's cluster, ensure the continuous availability of the cluster, and automatically make adjustments when resource requirements change. Through the monitoring and fault-tolerance mechanism of Kubernetes, when a computing node fails or there is a shortage of resources, the system will automatically trigger the repair process or expand new computing nodes to ensure the stable operation of the task and the high availability of the cluster. This full lifecycle automated management capability of the cluster greatly reduces the management burden in a multi-tenant environment and improves the flexibility and stability of the cluster.

[0075] The present invention also proposes a Slurm cluster service dynamic management system based on Kubernetes Operator to implement the above process, which includes three core modules: Figure 2 shown.

[0076] Control module: As the core management component of the system, it is responsible for the dynamic management of the entire cluster. It automatically expands, shrinks, recovers from failures, and monitors the cluster by monitoring the changes in custom resource CRDs, ensuring the stable operation of the Slurm cluster in a dynamic environment.

[0077] k8s scheduling module: responsible for the deployment and management of Slurm control and support services, including Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service, and Slurm database service; using Kubernetes' native scheduling function to deploy and manage these core services to ensure their efficient operation on the Kubernetes platform.

[0078] Slurm dedicated scheduling module: one of the core components for efficient resource allocation and task scheduling. This module is responsible for the scheduling and adjustment of Slurm cluster computing nodes, and dynamically optimizes resource usage by combining user resource requests (such as CPU, memory, GPU, etc.) in custom resource CRDs. In order to ensure maximum resource utilization and efficient execution of tasks, the scheduling module adopts a series of filtering and priority sorting strategies to ensure that tasks can exclusively allocate computing nodes.

[0079] like Figure 2 As shown, the control module of the Slurm cluster service-oriented dynamic management system controls and manages the k8s scheduling module and the Slurm dedicated scheduling module, implements the Slurm cluster service-oriented dynamic management method, and has the same technical effect as the Slurm cluster service-oriented dynamic management method.

[0080] The above-mentioned embodiments are only preferred implementations of the present invention, and are only used to help understand the method and core ideas of the present application. The protection scope of the present invention is not limited to the above-mentioned embodiments. All technical solutions under the idea of ​​the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A Slurm cluster service-oriented dynamic management method, characterized in that: include: S1. Submit custom resource CRD based on KubernetesOperator to define the configuration information of Slurm cluster; S2. Implement a custom controller for custom resource CRD based on Kubernetes Operator, the controller schedules and adjusts computing nodes according to the scheduling strategy, automatically generates configuration files required by the Slurm cluster, and deploys Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service, and Slurm database service; wherein the Slurm control service, Slurm execution node service, Slurm client, and Slurm database service are connected to PVC persistent storage; S3,Users submit jobs through the Slurm client and learn the job status through the Slurm monitoring service; S4, the controller monitors the changes of the custom resource CRD and automatically performs expansion, reduction, fault recovery and monitoring operations of the nodes of the Slurm cluster; S5. When the Slurm cluster is no longer needed, request to delete the entire cluster by deleting the custom resource CRD.

2. The Slurm cluster service dynamic management method according to claim 1, characterized in that: The configuration information in step S1 includes the number of nodes, resource requirements, and scheduling strategies of the Slurm cluster.

3. The Slurm cluster service dynamic management method according to claim 1, characterized in that: The scheduling and adjustment method of the computing nodes in step S2 includes: S201, node resource filtering, preliminarily screening available computing nodes according to the resource requirements in the custom resource CRD; S202, after selecting suitable computing nodes, prioritize the computing nodes according to the sorting rules; S203, after the priority is sorted, the node label is set to indicate whether the computing node is occupied, so as to realize exclusive node allocation; S204. When the node resources in the cluster are insufficient to meet the resource demand, node expansion is automatically triggered according to the task load. When the resource demand decreases, the computing nodes are automatically reduced.

4. The Slurm cluster service dynamic management method according to claim 1, characterized in that: In step S2, the controller deploys Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service, and Slurm database service in a Kubernetes Deployment manner.

5. A Slurm cluster service dynamic management system, characterized in that: include: Control module: As the core management component, it is responsible for the dynamic management of the Slurm cluster. By monitoring the changes of custom resource CRDs, it automatically performs cluster expansion, reduction, fault recovery and monitoring operations; k8s scheduling module: responsible for deploying and managing Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service, and Slurm database service using the native scheduling function of Kubernetes; Slurm dedicated scheduling module: responsible for scheduling and adjusting computing nodes according to the resource requirements and scheduling policies of the custom resource CRD; The k8s scheduling module and the Slurm dedicated scheduling module are controlled and managed by the control module.

6. The Slurm cluster service dynamic management system according to claim 5, characterized in that: The custom resource CRD defines the configuration information of the Slurm cluster, and the configuration information includes the number of nodes, resource requirements, and scheduling strategy of the Slurm cluster.

7. The Slurm cluster service dynamic management system according to claim 5, characterized in that: The k8s scheduling module deploys Slurm control service, Slurm execution node service, Slurm client, Slurm monitoring service and Slurm database service in a Kubernetes Deployment manner, wherein the Slurm control service, Slurm execution node service, Slurm client and Slurm database service are connected to PVC persistent storage.

8. The Slurm cluster service dynamic management system according to claim 5, characterized in that: The scheduling and adjustment of computing nodes by the Slurm dedicated scheduling module include: node resource filtering, preliminary screening of available computing nodes according to the resource requirements in the custom resource CRD; after screening out suitable computing nodes, prioritizing the computing nodes according to the sorting rules; after the priority sorting, setting the node label to indicate whether the computing node is occupied to achieve node exclusive allocation; when the node resources in the cluster are insufficient to meet the resource requirements, automatically triggering node expansion according to the task load, and automatically reducing the computing nodes when the resource requirements decrease.

9. A computer-readable storage medium storing a computer program, characterized in that: The computer program is used to execute the Slurm cluster service-oriented dynamic management method as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the Slurm cluster service-oriented dynamic management method as described in any one of claims 1 to 4 is implemented.

Images