SLURM-based out-of-cloud user management method and system

Abstract

The application relates to the technical field of multi-user scheduling and management, and provides a SLURM-based external cloud user management method and system. The method comprises the following steps: analyzing the operating system type of an external user system; when the user system is not a LINUX system, reserving the business logic of the user system and the communication mode of the user system and a SLURM scheduler; when there is a requirement of importing user information, extracting single-user information based on the communication mode, importing the business logic into a cluster to obtain a user group through the single-user information after eliminating user permissions; when there is no requirement of importing user information, importing the business logic into the cluster to obtain the user group through preset information; automatically adding the user group to the SLURM scheduler through a mapping relationship, and synchronizing the user information in the user group; initializing permissions according to the group user information to obtain a management platform. The application has strong expandability, is compatible with different system types, supports multi-terminal modification, and does not affect data consistency.

Description

Technical Field

[0001] This invention relates to the field of multi-user scheduling and management technology, and in particular to a SLURM-based method and system for managing external cloud users. Background Technology

[0002] Currently, a significant portion of mainstream high-performance computing (HPC) products use SLURM as a scheduler for resource management and job scheduling, and they are usually equipped with a web-based visual management platform to reduce the learning curve for users.

[0003] The need for large-scale user management is a core part of HPC product use cases. It connects to the platform-side operation permissions and visualization functions, and to the two user systems of the operating system (LINUX, WINDOWS SERVER, etc.) and SLURM. However, existing user management solutions for HPC products using SLURM have many defects, including: data inconsistency and permission conflict risks caused by complex user systems, difficulties in connecting to external user systems, and lack of platform-side permission control and scalability.

[0004] Specifically, to address multi-node synchronization and information management issues for users, lightweight directory protocols such as LDAP and NIS are typically used. The advantages of this approach are simple configuration, ease of maintenance, and high availability. However, it has significant limitations in areas such as user information synchronization, integration with existing user systems of external clients, and user resource / permission restrictions. Furthermore, since resource monitoring is based on the SLURM user system, it is not interoperable with the operating system's user system. This means that after adding operating system users, SLURM users still need to be added and configured individually.

[0005] Existing solutions typically map the two, which solves some user information synchronization issues, but lacks scalability when accessing platforms. Furthermore, it is only compatible with the Linux user system. Even though SLURM is almost always deployed on Linux, there are still many application scenarios that are implemented through jump servers, so this solution has poor versatility. Summary of the Invention

[0006] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a SLURM-based method and system for managing external cloud users.

[0007] This invention provides a SLURM-based method for managing externally connected cloud users, comprising:

[0008] S1: Obtain the user system to be externalized, and analyze to obtain the operating system type of the user system;

[0009] S2: When the operating system type is not LINUX, retain the business logic of the user system and obtain the communication method between the user system and the pre-accessed SLURM scheduler;

[0010] S3: Determine whether the user system has a need to import user information;

[0011] If there is a need to import user information, extract individual user information based on the communication method, and import the business logic of the user system into the cluster to obtain a user group by removing the individual user information after removing user permissions.

[0012] If there is no need to import user information, the business logic of the user system is imported into the cluster to obtain user groups through preset information;

[0013] S4: Pre-set mapping relationship, automatically add the user group to the pre-accessed SLURM scheduler through the mapping relationship, and synchronize the user information in the user group to obtain group user information;

[0014] S5: Initialize permissions based on the group user information, obtain the management platform, and complete the management of the external user system.

[0015] According to the SLURM-based cloud-connectable user management method provided by the present invention, in step S2, when the operating system is a LINUX system, the external configuration of the user system is imported into the cluster.

[0016] According to the SLURM-based cloud user management method provided by the present invention, the permission initialization in step S5 includes preset administrators and adding platform-side permissions.

[0017] According to the present invention, a cloud-connectable user management method based on SLURM is provided, wherein the pre-connected SLURM scheduler is linked to the LDAP protocol.

[0018] According to the SLURM-based cloud-connectable user management method provided by the present invention, step S4 further includes:

[0019] S41: Configure the pre-connected SLURM scheduler.

[0020] According to the SLURM-based cloud-connectable user management method provided by the present invention, step S41 further includes:

[0021] S411: Import the LDAP configuration of the pre-accessed SLURM scheduler so that each node in the cluster recognizes the LDAP entries;

[0022] S412: Import the LDAP configuration of the user group so that the user information in the user group is maintained through LDAP entries;

[0023] S413: Modify nsswitch.conf in the pre-access SLURM scheduler.

[0024] According to the SLURM-based cloud-connectable user management method provided by the present invention, the LDAP configuration in step S411 includes authconfig, system-auth, and password-auth.

[0025] According to the SLURM-based cloud-connectable user management method provided by the present invention, step S5 further includes:

[0026] S51: Connect the management platform to the visualization management and operation platform of HPC products.

[0027] According to the SLURM-based cloud-connectable user management method provided by the present invention, the management permissions of the management platform in step S5 include creating users, deleting users, updating users, and querying users.

[0028] The present invention also provides a SLURM-based external cloud user management system for executing a SLURM-based external cloud user management method as described in any of the preceding claims, comprising:

[0029] Preprocessing module: used to obtain the user system to be externalized, and also used to analyze and obtain the operating system type of the user system;

[0030] Logical communication module: used to retain the business logic of the user system and obtain the communication method between the user system and the pre-accessed SLURM scheduler;

[0031] Group building module: used to extract single user information based on the communication method, and import the business logic of the user system into the cluster to obtain a user group by removing the single user information after removing user permissions; also used to import the business logic of the user system into the cluster to obtain a user group by using preset information.

[0032] Mapping access module: used to pre-configure mapping relationships, and automatically add the user group to the pre-accessed SLURM scheduler through the mapping relationships;

[0033] Management platform module: used to synchronize user information in the user group, obtain group user information, initialize permissions based on the group user information, and obtain the management platform.

[0034] This invention provides a SLURM-based externally connectable cloud user management method and system. It proposes a multi-level externally connectable cloud user management scheme based on the SLURM scheduler, which is used in HPC scenarios and links with protocols such as LDAP / NIS. It is more flexible in use, has better performance, strong scalability and low coupling, and has high practicality.

[0035] This invention enables multi-level integrated user management, centralizing users managed via LDAP / NIS and SLURM in LINUX onto a management platform. Secondly, the management solution supports external existing user systems and provides scalability for platform-side permission management. It also supports multi-terminal modifications; modifications made through the management platform's external interface, through the machine where SLURM resides, and through external user systems are equivalent and synchronous, without affecting data consistency or system availability.

[0036] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a SLURM-based cloud-connectable user management method provided by the present invention;

[0039] Figure 2 This is a schematic diagram of an externally connectable cloud user management system based on SLURM provided by the present invention.

[0040] Figure label:

[0041] 100. Preprocessing module; 200. Logical communication module; 300. Group construction module; 400. Mapping access module; 500. Management platform module. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0043] In the description of the embodiments of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0044] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0045] In embodiments of the present invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0046] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0047] To better understand this invention, the following explanations are provided for the technical terms used in this invention.

[0048] High-performance computing (HPC) is a field of computing, sometimes referred to as "supercomputing," that aims to solve scientific, engineering, and commercial problems that require processing large amounts of data, complex models, or computationally intensive tasks by utilizing massive computing resources.

[0049] SLURM (Simple Linux Utility for Resource Management) is an open-source, highly scalable job scheduling system primarily used for managing and allocating resources in computing clusters. It is widely used in scientific computing, high-performance computing (HPC), and large-scale data processing.

[0050] LDAP (Lightweight Directory Access Protocol) is an open standard protocol for accessing and managing distributed directory services. It provides a unified way to find, modify, and verify directory information stored on a network. LDAP is commonly used for Linux-based identity authentication and authorization within organizations, such as employee and organizational structure, user accounts, email addresses, and other information.

[0051] Network Information Service (NIS) is a service used to manage user and host information in distributed systems. Originally developed by Sun Microsystems, it is widely used in UNIX and UNIX-like systems. NIS aims to provide a centralized way to manage user accounts, passwords, groups, hostnames, and other information within a network.

[0052] Existing user management solutions using the HPC product SLURM suffer from problems such as data inconsistency and permission conflicts due to complex user systems, difficulties in integrating with external user systems, and a lack of platform-side permission control and scalability.

[0053] This invention addresses the shortcomings of existing solutions by proposing a multi-level, externally accessible cloud user management solution for HPC scenarios. This solution is based on the SLURM scheduler and its user system, and integrates protocols such as LDAP / NIS. This invention not only allows for flexible access to external user systems but also achieves integrated multi-level user management while providing access control capabilities for a visualized HPC management platform.

[0054] The following is combined with Figures 1 to 2 Embodiments of the present invention are described.

[0055] This invention provides a SLURM-based method for managing externally connected cloud users, comprising:

[0056] S1: Obtain the user system to be externalized, and analyze to obtain the operating system type of the user system;

[0057] Furthermore, we first analyze the user systems that need to be connected and the types of operating systems they are based on. The most commonly used is the LDAP user system based on Linux.

[0058] S2: When the operating system type is not LINUX, retain the business logic of the user system and obtain the communication method between the user system and the pre-accessed SLURM scheduler;

[0059] In step S2, when the operating system is a LINUX system, the external configuration of the user system is imported into the cluster.

[0060] The pre-connected SLURM scheduler is linked to the LDAP protocol.

[0061] Furthermore, based on the operating system type obtained in step S1, if the external user system needs to be based on Linux, then the external configuration is directly imported into the cluster, meaning the external user system is migrated to the machine where SLURM resides without any loss. If the external user system needs to be based on a different operating system than Linux, then the communication method between the user system and the machine where SLURM resides needs to be obtained, and this method is used to extract and transmit user information. The external user system still needs to be retained.

[0062] S3: Determine whether the user system has a need to import user information;

[0063] If there is a need to import user information, extract individual user information based on the communication method, and import the business logic of the user system into the cluster to obtain a user group by removing the individual user information after removing user permissions.

[0064] If there is no need to import user information, the business logic of the user system is imported into the cluster to obtain user groups through preset information;

[0065] Furthermore, if user information needs to be imported, remove unnecessary information such as user permissions (to prevent conflicts with user permissions on the management platform), leaving only basic information, and import it into the cluster as LINUX users and user groups; if user information is not imported and you simply want to continue using the previous user management logic, you can use the preset information to add users / user groups.

[0066] S4: Pre-set mapping relationship, automatically add the user group to the pre-accessed SLURM scheduler through the mapping relationship, and synchronize the user information in the user group to obtain group user information;

[0067] In some embodiments, the system can be built based on a standard Linux image, version number CentOS 7.6. The process of creating the system image and installing basic dependencies will not be described in detail here. The embodiments assume that the customer already has their own unified authentication system based on LDAP.

[0068] Step S4 further includes:

[0069] S41: Configure the pre-connected SLURM scheduler.

[0070] Step S41 further includes:

[0071] S411: Import the LDAP configuration of the pre-accessed SLURM scheduler so that each node in the cluster recognizes the LDAP entries;

[0072] S412: Import the LDAP configuration of the user group so that the user information in the user group is maintained through LDAP entries;

[0073] S413: Modify nsswitch.conf in the pre-access SLURM scheduler.

[0074] The LDAP configuration in step S411 includes authconfig, system-auth, and password-auth.

[0075] Furthermore, installing SLURM requires some special configuration. This involves importing the customer's LDAP configuration, including authconfig, system-auth, and password-auth. The main purpose is to enable all nodes in the cluster to recognize entries in LDAP as operating system users. Then, nsswitch.conf is modified to ensure the operating system prioritizes recognizing LDAP entries as operating system users, preventing non-standard operations from creating users with the same name. Next, LDAP entry information from external systems is imported, and the operating system automatically recognizes these entries as operating system users; user information is maintained through LDAP entries. Then, Linux user information is added to the SLURM user system according to preset mappings. The LDAP entry information is imported into the management platform's database. Finally, permission information is initialized, including but not limited to: setting administrators, granting resource operation permissions, and user management permissions.

[0076] S5: Initialize permissions based on the group user information, obtain the management platform, and complete the management of the external user system.

[0077] The permission initialization in step S5 includes setting a preset administrator and adding platform-side permissions.

[0078] Step S5 further includes:

[0079] S51: Connect the management platform to the visualization management and operation platform of HPC products.

[0080] The following describes the process of accessing the visual management and operation platform in some other embodiments.

[0081] In this embodiment, the HPC product is equipped with a visual management platform and needs to be connected to the management platform. The main steps are as follows:

[0082] First, you need to integrate the user management interface, which requires connecting to the management platform's interfaces for creating, deleting, modifying, querying, disabling, and enabling / disabling user functions.

[0083] Secondly, access the platform-side permission system. Platform-side permissions such as user login, job submission, and cluster information query can be represented in a specific form and stored in the permission field reserved by the management platform.

[0084] Finally, the component responsible for SLURM resource monitoring and management is integrated, allowing scheduler-related permissions to be modified directly through the management platform's preset interfaces. For example, to limit a user to submit only n jobs, the UpdateUser interface can be called and relevant fields provided.

[0085] In step S5, the management permissions of the management platform include creating users, deleting users, updating users, and querying users.

[0086] The following describes the usage process of the user management platform in some other embodiments.

[0087] This embodiment describes CRUD operations performed using a user management platform.

[0088] First, select entry points including the platform side, Linux side, SLURM side, and management platform side to make modifications on multiple ends. The operation methods of these multiple entry points depend on the operating system version and type. That is, modifying through the external interface of the management platform, modifying through the machine where SLURM is located, and modifying through the external user system are equivalent and synchronous, and do not affect data consistency and system availability.

[0089] Creating a user requires specifying a username and password. First, the system checks if the user has permission to create a user. Then, the module stores the relevant information in the local database and returns a success message, sending a user creation command to Linux. Next, Linux constructs an LDAP entry based on the user information and adds it, synchronizing the user creation to all nodes in the cluster. Finally, the user is added to the SLURM user system according to the mapping. LDAP only contains basic information (such as email, phone number, home directory address), while the management platform has detailed information (such as interface permissions, whether it has been deleted, whether it has been disabled, and organizational structure).

[0090] Deleting a user requires specifying the username and whether to retain the home directory. This embodiment implements a soft deletion. Upon receiving a request, the management platform first deletes the user in the local database using a keyword and returns success, while simultaneously sending a command to Linux. Linux will determine whether to delete the home directory based on the input parameters (saving the previous output), and will prohibit the user from logging in and revoke all permissions. The LDAP entry remains unchanged, while the SLURM side deletes the user to prevent them from accessing SLURM remotely.

[0091] To update a user, you need to specify the username. It supports updating the user's basic information and permission information. If it is a platform-side permission information update, you only need to make changes in the local database of the management platform. If it is a basic information update, it also needs to be synchronized to the underlying LDAP entry modification. If it is a scheduler-side permission update, you can directly call the LINUX command to make changes.

[0092] The user query function supports querying a specific user, all users, and deleted users, and returns paginated data types.

[0093] The present invention also provides a SLURM-based external cloud user management system for executing a SLURM-based external cloud user management method as described in any of the preceding claims, comprising:

[0094] Preprocessing module 100: used to obtain the user system to be externalized, and also used to analyze and obtain the operating system type of the user system;

[0095] Logic communication module 200: used to retain the business logic of the user system and obtain the communication method between the user system and the pre-accessed SLURM scheduler;

[0096] Group building module 300: is used to extract single user information based on the communication method, and import the business logic of the user system into the cluster to obtain a user group by removing the single user information after removing user permissions; it is also used to import the business logic of the user system into the cluster to obtain a user group by using preset information.

[0097] Mapping access module 400: used to pre-set mapping relationships, and automatically add the user group to the pre-accessed SLURM scheduler through the mapping relationships;

[0098] Management platform module 500: used to synchronize user information in the user group, obtain group user information, initialize permissions based on the group user information, and obtain the management platform.

[0099] This invention provides a SLURM-based user management method and system that can be connected to an external cloud, enabling multi-level user integrated management and centralizing the management of LINUX (such as users managed through LDAP / NIS) and SLURM users on a management platform.

[0100] Compared to existing technologies, this system not only uses protocols such as LDAP to manage users, but also has a self-developed management platform to coordinate multiple user systems. This eliminates the need for cross-layer queries to LINUX or SLURM and reduces the workload of adding user information at the LINUX and SLURM layers. Secondly, it simplifies the amount of information that LINUX needs to maintain, reducing the risk of data inconsistency. Finally, customers only need to focus on the user system of the management platform, and the underlying user system is automatically synchronized, reducing the learning cost and improving the user experience.

[0101] In addition, the management solution provided by this invention supports external existing user systems and provides scalability for platform-side permission management.

[0102] Compared to existing technologies, where visualization platforms and user management systems are typically deployed in tandem and both rely on SLURM and the operating system, our management platform can be deployed independently while exposing standardized interfaces. This provides scalability for platform-side permission management and allows for seamless integration with well-developed visualization management platforms.

[0103] Furthermore, existing technologies can only extract information from existing user systems and import it one by one, while this technical solution can more easily connect to external user systems based on different operating systems and support integration with existing user management business logic.

[0104] Furthermore, the management method provided by this invention supports multi-terminal modification. Modifications made through the external interface of the management platform, through the machine where SLURM is located, and through the external user system are equivalent and synchronous, without affecting data consistency and system availability.

[0105] Existing technologies have already addressed data consistency issues by mapping fields across different user systems. However, their drawback is that modifications can only be made on one end, and user information synchronization is limited to the operating system itself. Integrating the user system into a visual management platform requires additional business logic to maintain the data; otherwise, every user-related business process would have to be routed to the underlying database, significantly impacting performance.

[0106] This invention provides a management platform that automatically synchronizes user operations from multiple terminals, offering multiple entry points for users with different scenarios and usage habits, and possessing higher usability, modifiability, and scalability.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A SLURM-based method for managing externally connected cloud users, characterized in that, include: S1: Obtain the user system to be externalized, and analyze to obtain the operating system type of the user system; S2: When the operating system type is not LINUX, retain the business logic of the user system and obtain the communication method between the user system and the pre-accessed SLURM scheduler; S3: Determine whether the user system has a need to import user information; If there is a need to import user information, extract individual user information based on the communication method, and import the business logic of the user system into the cluster to obtain a user group by removing the individual user information after removing user permissions. If there is no need to import user information, the business logic of the user system is imported into the cluster to obtain user groups through preset information; S4: Pre-set mapping relationship, automatically add the user group to the pre-accessed SLURM scheduler through the mapping relationship, and synchronize the user information in the user group to obtain group user information; S5: Initialize permissions based on the group user information, obtain the management platform, and complete the management of the external user system.

2. The SLURM-based cloud-connectable user management method according to claim 1, characterized in that, In step S2, when the operating system is a LINUX system, the external configuration of the user system is imported into the cluster.

3. The SLURM-based cloud-connectable user management method according to claim 1, characterized in that, The permission initialization in step S5 includes setting a preset administrator and adding platform-side permissions.

4. The SLURM-based cloud-connectable user management method according to claim 1, characterized in that, The pre-connected SLURM scheduler is linked to the LDAP protocol.

5. The SLURM-based cloud-connectable user management method according to claim 4, characterized in that, Step S4 also includes: S41: Configure the pre-connected SLURM scheduler.

6. The SLURM-based cloud-connectable user management method according to claim 5, characterized in that, Step S41 further includes: S411: Import the LDAP configuration of the pre-accessed SLURM scheduler so that each node in the cluster recognizes the LDAP entries; S412: Import the LDAP configuration of the user group so that the user information in the user group is maintained through LDAP entries; S413: Modify nsswitch.conf in the pre-access SLURM scheduler.

7. The SLURM-based cloud-connectable user management method according to claim 6, characterized in that, The LDAP configuration in step S411 includes authconfig, system-auth, and password-auth.

8. The SLURM-based cloud-connectable user management method according to claim 1, characterized in that, Step S5 also includes: S51: Connect the management platform to the visualization management and operation platform of HPC products.

9. The SLURM-based cloud-connectable user management method according to claim 1, characterized in that, The management permissions of the management platform in step S5 include creating users, deleting users, updating users, and querying users.

10. A SLURM-based external cloud user management system, used to execute the SLURM-based external cloud user management method as described in any one of claims 1 to 9, characterized in that, include: Preprocessing module: used to obtain the user system to be externalized, and also used to analyze and obtain the operating system type of the user system; Logical communication module: used to retain the business logic of the user system and obtain the communication method between the user system and the pre-accessed SLURM scheduler; Group building module: used to extract single user information based on the communication method, and import the business logic of the user system into the cluster to obtain a user group by removing the single user information after removing user permissions; also used to import the business logic of the user system into the cluster to obtain a user group by using preset information. Mapping access module: used to pre-configure mapping relationships, and automatically add the user group to the pre-accessed SLURM scheduler through the mapping relationships; Management platform module: used to synchronize user information in the user group, obtain group user information, initialize permissions based on the group user information, and obtain the management platform.

Images