HPC system and task processing method based on HPC system
By introducing content management platform, authentication module and HPC control module into the HPC system, multi-user sharing and data isolation are realized, solving the problem that existing HPC systems cannot maximize resource utilization and data isolation, ensuring user privacy and data Security and calculates the user's resource occupancy and expenses.
Patent Information
- Application Number
- CN202411111625.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-14
AI Technical Summary
The existing HPC system cannot achieve data isolation between multiple different users, resulting in the inability to maximize the utilization of resources and is expensive and difficult to bear by users with limited conditions.
An HPC system was designed, including a content management platform, authentication module and HPC control module, which received user requests through the operation interface, performed user authentication and task authentication, ensured data isolation, and was in the scheduler, storage management unit and user management unit. Perform operations to provide a high-performance computing environment shared by multiple users.
Data isolation between multiple users is realized, HPC system resources are fully utilized, and the privacy and data security of each user are protected, while calculating resource occupation and cost information of each user.
Smart Images

Figure CN119089511A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a HPC system and a task processing method based on the HPC system. Background Art
[0002] High Performance Computing (HPC) is a technology that uses powerful processor clusters working in parallel to process massive multidimensional data sets (big data) and solve complex problems at extremely high speeds. HPC systems usually include a computer cluster (i.e., an HPC cluster) consisting of multiple networked high-speed computer servers, which runs multiple tasks simultaneously on multiple computing nodes, achieving large-scale parallel computing and thus achieving faster running speeds.
[0003] However, current HPC systems are unable to achieve data isolation between multiple different users and are generally only available to a single user. On the one hand, this fails to maximize resource utilization, and on the other hand, it is difficult for some users with limited conditions to independently bear the cost of the HPC system. Summary of the invention
[0004] In view of the above-mentioned deficiencies of the prior art, an object of the present invention is to provide an HPC system and a task processing method based on the system, so as to support multiple users to use the HPC system and realize data isolation between different users.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] In a first aspect, the present invention provides an HPC system, including a content management platform, an authentication module and an HPC control module, wherein the content management platform provides an operation interface, wherein:
[0007] The content management platform is used to receive a login request submitted by a user through the operation interface, and send a user authentication request to the authentication module based on the login request, so that the authentication module verifies the user authentication request, and when the verification is successful, sends an authorization token to the content management platform;
[0008] The content management platform is also used to receive a task request submitted by a user through the operation interface, and send the authorization token and the task request to the HPC control module. The HPC control module is used to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request and feeds back the verification result to the HPC control module. When the verification is successful, the HPC control module is used to execute an operation corresponding to the task request.
[0009] Further, the HPC control module includes a controller and a scheduler management unit, a storage management unit and a user management unit connected to the controller;
[0010] The scheduling management unit is connected to a plurality of schedulers and is used to manage the task scheduling of the plurality of schedulers;
[0011] The storage management unit is used for performing data storage management;
[0012] The user management unit is used to perform user management;
[0013] The controller is used to control the scheduler management unit, the storage management unit or the user management unit to perform corresponding operations according to the task request.
[0014] Furthermore, the content management platform is also used to receive a user creation request submitted by a user through the operation interface, and obtain an authorization token corresponding to the user information carried in the user creation request from the authentication module, and then send the user creation request and the authorization token to the HPC control module;
[0015] The HPC control module is also used to send the user creation request and the authorization token to the authentication module for verification. When the verification is successful, the HPC control module is also used to call the content management platform to create a user and register the user in the scheduler management unit and the storage management unit.
[0016] Furthermore, the system also includes a database connected to the content management platform, and the content management platform is further configured to store the user information carried in the user creation request in the database when receiving the user creation request.
[0017] Furthermore, the control module is also used to obtain the storage resources and computing resources occupied by each user, and calculate the cost information corresponding to each user based on the storage resources and computing resources.
[0018] Furthermore, after executing the operation corresponding to the task request, the HPC control module is further used to feed back the execution result through the operation interface provided by the content management platform.
[0019] Furthermore, the system also includes a login node connected to the control module, wherein the login node is deployed with a console-agent service.
[0020] Furthermore, the content management platform is connected to the database via a conductor module.
[0021] Furthermore, the scheduler includes a slurm scheduler and / or an openpbs scheduler.
[0022] In a second aspect, the present invention provides a task processing method based on the aforementioned HPC system, comprising:
[0023] When the content management platform receives a login request submitted by a user through the operation interface, the content management platform sends a user authentication request to the authentication module based on the login request, so that the authentication module verifies the user authentication request, and when the verification is successful, sends an authorization token to the content management platform;
[0024] When the content management platform receives a task request submitted by a user through the operation interface, the authorization token and the task request are sent to the HPC control module. The HPC control module is used to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request. When the verification is successful, the HPC control module is used to execute an operation corresponding to the task request.
[0025] By adopting the above technical solution, the present invention has the following beneficial effects:
[0026] By adopting the above-mentioned content management platform, authentication module and HPC control module, the present invention can provide the HPC system to multiple users and ensure data isolation between different users, so as to effectively protect the privacy and data security of each user when the HPC system is fully used. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 Schematic diagram of the structure of the HPC system in Example 1 of the present invention;
[0028] Figure 2 Schematic diagram of the structure of the HPC system in Example 2 of the present invention. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0030] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms "a", "the" and "the" used in the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0031] Example 1
[0032] This embodiment provides a HPC system, such as Figure 1 As shown, it includes CMP (Content Management Platform), authentication module, HPC control module and database. Among them, the HPC control module specifically includes a controller and a scheduler management unit, a storage management unit and a user management unit connected to the controller.
[0033] In this embodiment, the scheduler management unit is connected to several schedulers. Each scheduler is used to split the task to be executed into multiple subtasks, and then schedule different computing nodes to execute each subtask, and summarize the execution results; and the scheduling management unit is used to manage the task scheduling operations of each scheduler. It should be noted that this embodiment does not impose any restrictions on the number and type of schedulers. Specifically, 1, 2, 3 or more schedulers can be used, and the type of scheduler can be selected from any one or more schedulers such as slurm, openpbs, etc.
[0034] In this embodiment, the storage management unit is connected to the memory for data storage management. The memory used in this embodiment may be one or more of lustre, BGFS, CephFS and other memory, and this embodiment does not impose any specific restrictions on this. The stored data in the memory includes but is not limited to task-related data, such as temporary task data, task results, etc.).
[0035] In this embodiment, the user management unit is connected to the user storage system for user management (including user creation, addition, deletion, and modification, user group management, etc.). The user storage system used in this embodiment may be an openldap system or other systems.
[0036] In this embodiment, the controller is preferably implemented using ForceControl, which is pre-configured with various scheduler agents. The scheduler management module, storage management module, and user management module can call the corresponding scheduler agent when executing the corresponding task to perform task scheduling and return the scheduling result. ForceControl plays a connecting role in this embodiment: for the upper layer, it provides API services for CMP; for the lower layer, it provides a variety of drivers that can drive different memories and schedulers.
[0037] In this embodiment, the content management platform CMP can provide users with a good operation interface, so that different users can establish and submit user creation requests, login requests and task requests through the operation interface.
[0038] When the content management platform receives a user creation request submitted by a user through the operation interface (the request carries user information such as user name, password, identity, etc.), it stores the user information carried in the request in the database, and sends the user creation request to the authentication module, so that the authentication module generates a corresponding authorization token based on the user information carried in the user creation request, and feeds the authorization token back to the content management platform. Then, the content management platform sends the received user creation request and the corresponding authorization token to the HPC control module, and the HPC control module sends the user creation request and the authorization token to the authentication module for verification. When the verification authorization token is legal, the verification is successful. At this time, the HPC control module calls the content management platform to create a user (based on user information). After the creation is completed, the user is registered in the scheduler management unit and the storage management unit, so that the user can access the underlying memory and tasks in the scheduler after registration.
[0039] After creating the user, when the content management platform receives a login request (carrying user information such as user name, password, and identity) submitted by the user through the operation interface, it generates a user authentication request carrying the user information based on the login request, and sends the user authentication request to the authentication module to verify the user authentication request through the authentication module. When the authentication module already has an authorization token corresponding to the user information in the request, the verification is successful, and the authorization token corresponding to the user information is fed back to the content management platform, that is, the user login is completed. At this time, the private directory corresponding to the user can be displayed on the operation interface.
[0040] After the user logs in, when the content management platform receives a task request submitted by the user through the operation interface (the request carries task information such as task type and task identifier), it sends the aforementioned authorization token and task request to the HPC control module; the HPC control module then generates a task authentication request (carrying a token) based on the authorization token and the task request, and sends the task authentication request to the authentication module, so that the authentication module verifies the task authentication request and feeds back the verification result to the HPC control module. When the verification is successful (the user corresponding to the token has the authority to perform the corresponding task), the HPC control module performs the operation corresponding to the task request, specifically including: through the controller, controlling the scheduler management unit, the storage management unit or the user management unit to perform the corresponding operation according to the task request.
[0041] After executing the operation corresponding to the task request, the HPC control module is also used to feedback the execution result through the operation interface provided by the content management platform. Users can see the task results in their own private directory through the interface, and the interface lists the files on the host by calling ftp.
[0042] By adopting the above-mentioned content management platform, authentication module and HPC control module, this embodiment can provide the HPC system to multiple users and ensure data isolation between different users, so as to effectively protect the privacy and data security of each user when the HPC system is fully used.
[0043] Since the HPC system of this embodiment can provide services to multiple users and can achieve data isolation for different users, on this basis, the control module of this embodiment can also be used to count the storage resources occupied by each user and the computing resources occupied by each user when performing tasks, and calculate the corresponding cost information of each user based on the storage resources and computing resources.
[0044] For example, the calculation index of storage resources is: the average storage capacity occupied in a predetermined period under the private directory corresponding to the user. The calculation index of computing power resources can include the amount of CPU computing power resources occupied (unit: core hour) and GPU computing power resources (unit: card hour). Among them, the amount of CPU computing power resources is the number of CPU cores occupied during the task running time * running time, and the amount of GPU computing power resources is the number of GPU cards occupied during the task running time * running time.
[0045] In this embodiment, the database is used to store data, such as user information, resource information, billing information, etc. As a preferred solution, a conductor (constructor) is also provided between the CMP and the database to connect to the database based on the resful API, thereby improving security. Conductor is an open source workflow orchestration system that provides a unified API (application programming interface) that enables users to easily manage, orchestrate and monitor complex data tasks and machine learning pipelines.
[0046] In addition, in addition to configuring and submitting task requests, the operation interface of this embodiment can also manage and transfer job files, share data, etc. It is simple and easy to use, reducing the difficulties caused by complex command line operations.
[0047] Example 2
[0048] This embodiment provides another implementation of the HPC system of the present invention. Figure 2 As shown, the difference between this embodiment and embodiment 1 is that a login node connected to the control module is added, thereby providing another user login method besides CMP.
[0049] In this embodiment, the login node is deployed with a console-agent service, which is used to expose the console graphical interface and API. By configuring novnc (novnc is a Web-based VNC client that can directly access the VNC server in the browser) or DCV (DeepCloud VNC, an enhanced version of VNC that provides better performance and security), VNC (Virtual Network Console) service can be implemented for users. After the user logs in through the node, he can write tasks on the node and perform debugging. The node supports a graphical interface.
[0050] Example 3
[0051] This embodiment provides a task processing method based on the aforementioned HPC system, which specifically includes the following steps:
[0052] When the content management platform receives a login request submitted by a user through the operation interface, it sends a user authentication request to the authentication module based on the login request, so that the authentication module verifies the user authentication request. When the verification is successful, it sends an authorization token to the content management platform.
[0053] When the content management platform receives a task request submitted by a user through the operation interface, the authorization token and the task request are sent to the HPC control module. The HPC control module is used to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request. When the verification is successful, the HPC control module is used to perform operations corresponding to the task request, specifically including: through the controller, controlling the scheduler management unit, the storage management unit or the user management unit to perform corresponding operations according to the task request.
[0054] When the HPC control module completes the operation corresponding to the task request, it will feedback the execution result through the operation interface provided by the content management platform. Users can see the task results in their own private directory through the interface, and the interface lists the files on the host by calling ftp.
[0055] Through the method of this embodiment, the HPC system can be provided to multiple users, and data isolation between different users can be guaranteed, so that the privacy and data security of each user can be effectively protected when the HPC system is fully used.
[0056] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.
Claims
1. A HPC system, characterized in that: It includes a content management platform, an authentication module and an HPC control module, and the content management platform provides an operation interface, wherein: The content management platform is used to receive a login request submitted by a user through the operation interface, and send a user authentication request to the authentication module based on the login request, so that the authentication module verifies the user authentication request, and when the verification is successful, sends an authorization token to the content management platform; The content management platform is also used to receive a task request submitted by a user through the operation interface, and send the authorization token and the task request to the HPC control module. The HPC control module is used to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request and feeds back the verification result to the HPC control module. When the verification is successful, the HPC control module is used to execute an operation corresponding to the task request.
2. The HPC system according to claim 1, wherein: The HPC control module includes a controller and a scheduler management unit, a storage management unit and a user management unit connected to the controller; The scheduling management unit is connected to a plurality of schedulers and is used to manage the task scheduling of the plurality of schedulers; The storage management unit is used for performing data storage management; The user management unit is used to perform user management; The controller is used to control the scheduler management unit, the storage management unit or the user management unit to perform corresponding operations according to the task request.
3. The HPC system according to claim 2, characterized in that: The content management platform is also used to receive a user creation request submitted by a user through the operation interface, and obtain an authorization token corresponding to the user information carried in the user creation request from the authentication module, and then send the user creation request and the authorization token to the HPC control module; The HPC control module is also used to send the user creation request and the authorization token to the authentication module for verification. When the verification is successful, the HPC control module is also used to call the content management platform to create a user and register the user in the scheduler management unit and the storage management unit.
4. The HPC system according to claim 3, characterized in that: The HPC system further includes a database connected to the content management platform. The content management platform is further configured to store the user information carried in the user creation request in the database when receiving the user creation request.
5. The HPC system according to claim 4, characterized in that: The HPC control module is also used to obtain the storage resources and computing resources occupied by each user, and calculate the cost information corresponding to each user based on the storage resources and computing resources, and store it in the database.
6. The HPC system according to claim 1, wherein: The HPC system according to claim 1 is characterized in that, after the HPC control module completes the operation corresponding to the task request, it is further used to feedback the execution result through the operation interface provided by the content management platform.
7. The HPC system according to claim 1, wherein: The system further comprises a login node connected to the control module, wherein the login node is deployed with a console-agent service.
8. The HPC system according to claim 1, wherein: The content management platform is connected to the database via a conductor module.
9. The HPC system according to claim 1, wherein: The scheduler includes a slurm scheduler and / or an openpbs scheduler.
10. A task processing method of an HPC system based on any one of claims 1 to 9, comprising: When the content management platform receives a login request submitted by a user through the operation interface, the content management platform sends a user authentication request to the authentication module based on the login request, so that the authentication module verifies the user authentication request, and when the verification is successful, sends an authorization token to the content management platform; When the content management platform receives a task request submitted by a user through the operation interface, the authorization token and the task request are sent to the HPC control module. The HPC control module is used to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request. When the verification is successful, the HPC control module is used to execute an operation corresponding to the task request.
Citation Information
Patent Citations
Agent test and authorization management system and method
CN111212083A
Hypercomputing-based public computing platform
CN115766714A
Broadcasting control service information management system
CN117156220A
Accessing a cloud-based service platform using enterprise application authentication
US20160065555A1
Method for global identity verification
US20240154952A1
