HPC system and task processing method based on HPC system

Abstract

Provided in the present invention are an HPC system and a task processing method based on the HPC system. The system comprises a content management platform, an authentication module and an HPC control module, wherein the content management platform provides an operation interface, and is used for receiving a login request by means of the operation interface, and sending a user authentication request to the authentication module on the basis of the login request, such that the authentication module verifies the user authentication request, and when the verification succeeds, the authentication module sends an authorization token to the content management platform; and the content management platform is further used for receiving a task request by means of the operation interface, and sending the authorization token and the task request to the HPC control module; the HPC control module is used for sending a task authentication request to the authentication module on the basis of the authorization token and the task request, such that the authentication module verifies the task authentication request and feeds back a verification result to the HPC control module; and when the verification succeeds, the HPC control module executes an operation corresponding to the task request. The present invention can support multiple users to use an HPC system and achieve data isolation between different users.

Description

HPC system and task processing method based on the same TECHNICAL FIELD

[0001] The present application relates to the field of computer technology, and in particular to an HPC system and a task processing method based on the same. BACKGROUND

[0002] High Performance Computing (HPC) is a technology that uses a cluster of powerful processors working in parallel to process massive multi-dimensional data sets (big data) and solve complex problems at extremely high speed. An HPC system usually includes a computer cluster (i.e., an HPC cluster) composed of multiple networked high-speed computer servers. Multiple tasks are run simultaneously on multiple computing nodes to achieve large-scale parallel computing, thereby achieving faster running speed.

[0003] However, the current HPC system cannot achieve data isolation between multiple different users and is generally used by a single user. On the one hand, this cannot maximize the utilization of resources, and on the other hand, it is difficult for some users with limited conditions to independently bear the cost of the HPC system. SUMMARY

[0004] In view of the above deficiencies of the prior art, the present application aims to provide an HPC system and a task processing method based on the same to support multiple users using the HPC system while achieving data isolation between different users.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, the present application provides an HPC system, comprising a content management platform, an authentication module and an HPC control module, and the content management platform provides an operation interface, wherein:

[0007] The content management platform is configured to receive a login request submitted by a user through the operation interface, send a user authentication request to the authentication module based on the login request, and cause the authentication module to verify the user authentication request. When the verification is successful, an authorization token is sent to the content management platform.

[0008] The content management platform is further configured to receive a task request submitted by a user through the operation interface, and send the authorization token and the task request to the HPC control module, and the HPC control module is configured to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request and feeds back a verification result to the HPC control module, and when the verification is successful, the HPC control module is configured to perform an operation corresponding to the task request.

[0009] Further, the HPC control module comprises a controller and a scheduler management unit, a storage management unit and a user management unit connected to the controller.

[0010] The scheduling management unit is connected to a plurality of schedulers and is configured to manage task scheduling of the plurality of schedulers.

[0011] The storage management unit is configured to perform data storage management.

[0012] The user management unit is configured to perform user management.

[0013] The controller is configured to control the scheduler management unit, the storage management unit or the user management unit to perform corresponding operations according to the task request.

[0014] Further, the content management platform is further configured to receive a user creation request submitted by a user through the operation interface, and obtain an authorization token corresponding to user information carried in the user creation request from the authentication module, and then send the user creation request and the authorization token to the HPC control module.

[0015] The HPC control module is further configured to send the user creation request and the authorization token to the authentication module for verification, and when the verification is successful, the HPC control module is further configured to call the content management platform to create a user and register the user in the scheduler management unit and the storage management unit.

[0016] Further, the system further comprises a database connected to the content management platform, and the content management platform is further configured to store the user information carried in the user creation request into the database when the user creation request is received.

[0017] Further, the control module is further configured to obtain storage resources and computing power resources occupied by each user, and calculate fee information corresponding to each user based on the storage resources and computing power resources.

[0018] Further, the HPC control module is further configured to feed back an execution result through an operation interface provided by the content management platform after performing an operation corresponding to the task request.

[0019] Further, the system further comprises a login node connected with the control module, wherein the login node is deployed with a console-agent service.

[0020] Further, the content management platform is connected with the database through a conductor module.

[0021] Further, the scheduler comprises a slurm scheduler and / or an openpbs scheduler.

[0022] In a second aspect, the present application provides a task processing method based on the aforementioned HPC system, comprising:

[0023] When the content management platform receives a login request submitted by a user through the operation interface, the content management platform sends a user authentication request to the authentication module based on the login request, so that the authentication module verifies the user authentication request, and when the verification succeeds, the content management platform sends an authorization token to the content management platform.

[0024] When the content management platform receives a task request submitted by a user through the operation interface, the content management platform sends the authorization token and the task request to the HPC control module, and the HPC control module is configured to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request, and when the verification succeeds, the HPC control module is configured to perform an operation corresponding to the task request.

[0025] By adopting the above technical solutions, the present application has the following beneficial effects:

[0026] By adopting the above content management platform, authentication module and HPC control module, the present application can provide the HPC system to multiple users and ensure data isolation between different users, so as to effectively protect the privacy and data security of each user in the case of full use of the HPC system. BRIEF DESCRIPTION OF DRAWINGS

[0027] Fig. 1 is a structural schematic diagram of an HPC system in embodiment 1 of the present application;

[0028] Fig. 2 is a structural schematic diagram of an HPC system in embodiment 2 of the present application. DETAILED DESCRIPTION

[0029] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0030] The terms used in the present application are merely for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms "a", "an" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein means and includes any or all possible combinations of one or more associated listed items. Embodiments

[0031] The present embodiment provides an HPC system, as shown in FIG. 1, which includes a CMP (Content Management Platform), an authentication module, an HPC control module and a database. The HPC control module specifically includes a controller and a scheduler management unit, a storage management unit and a user management unit connected to the controller.

[0032] In the present embodiment, the scheduler management unit is in communication connection with a plurality of schedulers. Each of the schedulers is used to split the to-be-executed task into a plurality of sub-tasks, and then schedule different computing nodes to execute each sub-task and aggregate the execution results. The scheduler management unit is used to manage the task scheduling operations of each scheduler. It should be noted that the present embodiment does not make any limitation on the number and type of schedulers, and specifically, one, two, three or more schedulers can be used, and the type of scheduler can be selected from any one or more of slurm, openpbs and other schedulers.

[0033] In the present embodiment, the storage management unit is in communication connection with the storage for data storage management. The storage used in the present embodiment can be one or more of lustre, BGFS, CephFS and other storage, and the present embodiment does not make any specific limitation thereon. The storage data in the storage includes but is not limited to task-related data such as temporary task data and task results.

[0034] In the present embodiment, the user management unit is connected to a user storage system for user management (including user creation, addition, deletion, modification, user group management, etc.). The user storage system used in the present embodiment can be an openldap system or other system.

[0035] In the embodiment, the controller is preferably implemented by using ForceControl, which is pre-configured with various scheduler agents. The scheduler management module, the storage management module and the user management module can call the corresponding scheduler agent when performing corresponding tasks, so as to perform task scheduling and return a scheduling result. The ForceControl plays a role of connecting the upper layer and the lower layer in the embodiment: for the upper layer, the ForceControl provides API services; for the lower layer, the ForceControl provides various drivers, which can drive different memories and schedulers.

[0036] In the embodiment, the content management platform CMP can provide a good operation interface for users, so that different users can establish and submit a user creation request, a login request and a task request through the operation interface.

[0037] When the content management platform receives a user creation request submitted by a user through the operation interface (the request carries user information such as a username, a password and an identity), the content management platform stores the user information carried in the request in a database, and sends the user creation request to an authentication module, so that the authentication module generates a corresponding authorization token based on the user information carried in the user creation request, and feeds back the authorization token to the content management platform. Then, the content management platform sends the received user creation request and the corresponding authorization token to an HPC control module, and the HPC control module sends the user creation request and the authorization token to the authentication module for verification. When the authorization token is verified to be legal, the verification is successful. At this time, the HPC control module calls the content management platform to create a user (based on the user information), and after the creation is completed, the user is registered in the scheduler management unit and the storage management unit, so that the user can access tasks in the underlying memory and scheduler after the registration.

[0038] After the user is created, when the content management platform receives a login request submitted by a user through the operation interface (carrying user information such as a username, a password and an identity), the content management platform generates a user authentication request carrying the user information based on the login request, and sends the user authentication request to the authentication module for verification. When the authentication module already has an authorization token corresponding to the user information in the request, the verification is successful, and the content management platform is fed back the authorization token corresponding to the user information, that is, the user login is completed. At this time, the private directory corresponding to the user can be displayed on the operation interface.

[0039] After the user logs in, when the content management platform receives a task request submitted by the user through an operation interface (the request carries task information such as a task type and a task identifier), it sends the foregoing authorization token and the task request to the HPC control module; the HPC control module generates a task authentication request (carrying a token) based on the authorization token and the task request, and sends the task authentication request to the authentication module, so that the authentication module verifies the task authentication request and feeds back a verification result to the HPC control module. When the verification is successful (the user corresponding to the token has the permission to perform the corresponding task), the HPC control module performs an operation corresponding to the task request, specifically including: controlling the scheduler management unit, the storage management unit, or the user management unit to perform a corresponding operation according to the task request through the controller.

[0040] After the HPC control module performs the operation corresponding to the task request, it is further configured to feed back an execution result through an operation interface provided by the content management platform, and the user can see the task result under his / her private directory through the interface. The interface lists the files on the host by calling ftp.

[0041] The content management platform, the authentication module, and the HPC control module described above can be used to provide the HPC system for multiple users and ensure data isolation between different users, so that the privacy and data security of each user can be effectively protected while the HPC system is fully used.

[0042] Since the HPC system can provide services for multiple users and can achieve data isolation between different users, the control module can be further configured to count the storage resources occupied by each user and the computing power resources occupied by each user when performing a task, and calculate the cost information corresponding to each user based on the storage resources and the computing power resources.

[0043] For example, the calculation index of the storage resources is the average storage capacity occupied in a private directory corresponding to the user within a predetermined period. The calculation index of the computing power resources can include the CPU computing power resource amount (unit: core hour) and the GPU computing power resource amount (unit: card hour) occupied. The CPU computing power resource amount is the number of CPU cores occupied * running time within the task running time, and the GPU computing power resource amount is the number of GPU cards occupied * running time within the task running time.

[0044] In the embodiment, the database is used to store data such as user information, resource information, billing information, etc. As a preferred solution, a conductor is further arranged between the CMP and the database to connect the database based on resful API, thereby improving security. The conductor is an open-source workfilow orchestration system that provides a unified API (application program interface) to enable users to easily manage, orchestrate and monitor complex data tasks and machine learning pipelines.

[0045] In addition, through the operation interface of the embodiment, in addition to being able to configure and submit task requests, it can also perform operations such as managing and transmitting job files, sharing data, etc., which is simple and easy to use, and reduces the difficulties brought by complex command line operations. Embodiment

[0046] The embodiment provides another implementation of the HPC system of the application. As shown in FIG. 2, the embodiment differs from embodiment 1 in that a login node connected to the control module is added, thereby providing another user login approach in addition to the CMP.

[0047] In the embodiment, the login node is deployed with a console-agent service, which is used to expose a console graphical interface and an api. By configuring novnc (a Web-based VNC client that can directly access a VNC server in a browser) or DCV (DeepCloud VNC, an enhanced version of VNC that provides better performance and security), the console-agent service can provide VNC (Virtual Network Console) services for users. After logging in through the node, the user can write tasks on the node and also perform debugging, and the node supports graphical interface debugging. Embodiment

[0048] The embodiment provides a task processing method based on the foregoing HPC system, and specifically includes the following steps:

[0049] When the content management platform receives a login request submitted by a user through an operation interface, a user authentication request is sent to an authentication module based on the login request, so that the authentication module verifies the user authentication request, and when the verification is successful, an authorization token is sent to the content management platform.

[0050] When the content management platform receives the task request submitted by the user through the operation interface, the authorization token and the task request are sent to the HPC control module, the HPC control module is configured to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request, and when the verification is successful, the HPC control module is configured to perform an operation corresponding to the task request, specifically including: controlling the scheduler management unit, the storage management unit or the user management unit to perform a corresponding operation according to the task request through the controller.

[0051] When the HPC control module performs the operation corresponding to the task request, the execution result is fed back through the operation interface provided by the content management platform, and the user can see the task result in the private directory of the user through the interface, and the interface lists the files on the host by calling the ftp.

[0052] Through the method of the embodiment, the HPC system can be provided to multiple users, and the data isolation between different users is ensured, so that the privacy and data security of each user can be effectively protected in the case that the HPC system is fully used.

[0053] Although the specific embodiments of the present application are described above, those skilled in the art should understand that this is only an example, the protection scope of the present application is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present application, and these changes and modifications all fall within the protection scope of the present application.

Claims

1. An HPC system, characterized by, The HPC system comprises a content management platform, an authentication module and an HPC control module, and the content management platform provides an operation interface, wherein: The content management platform is configured to receive a login request submitted by a user through the operation interface, and send a user authentication request to the authentication module based on the login request, so that the authentication module verifies the user authentication request, and sends an authorized token to the content management platform when the verification is successful; The content management platform is further configured to receive a task request submitted by a user through the operation interface, and send the authorized token and the task request to the HPC control module, and the HPC control module is configured to send a task authentication request to the authentication module based on the authorized token and the task request, so that the authentication module verifies the task authentication request and feeds back a verification result to the HPC control module, and the HPC control module is configured to perform an operation corresponding to the task request when the verification is successful.

2. The HPC system of claim 1, wherein, The HPC control module comprises a controller and a scheduler management unit, a storage management unit and a user management unit connected to the controller; The scheduler management unit is connected to a plurality of schedulers and is configured to manage task scheduling of the plurality of schedulers; The storage management unit is configured to perform data storage management; The user management unit is configured to perform user management; The controller is configured to control the scheduler management unit, the storage management unit or the user management unit to perform a corresponding operation according to the task request.

3. The HPC system of claim 2, wherein, The content management platform is further configured to receive a user creation request submitted by a user through the operation interface, and obtain an authorized token corresponding to user information carried in the user creation request from the authentication module, and then send the user creation request and the authorized token to the HPC control module; The HPC control module is further configured to send the user creation request and the authorized token to the authentication module for verification, and when the verification is successful, the HPC control module is further configured to call the content management platform to create a user and register the user in the scheduler management unit and the storage management unit.

4. The HPC system of claim 3, wherein, The HPC system further comprises a database connected to the content management platform, and the content management platform is further configured to store user information carried in the user creation request into the database when the user creation request is received.

5. The HPC system of claim 4, wherein, The HPC control module is further configured to obtain storage resources and computing power resources occupied by each user, and calculate fee information corresponding to each user based on the storage resources and the computing power resources, and store the fee information into the database.

6. The HPC system of claim 1, wherein, The HPC system of claim 1, wherein the HPC control module is further configured to feed back an execution result through the operation interface provided by the content management platform after performing the operation corresponding to the task request.

7. The HPC system of claim 1, wherein, The system further comprises a login node connected to the control module, wherein the login node is deployed with a console-agent service.

8. The HPC system of claim 1, wherein, The content management platform is connected with the database through a conductor module.

9. The HPC system of claim 1, wherein, The scheduler comprises a slurm scheduler and / or an openpbs scheduler. 10.A task processing method based on the HPC system according to any one of the preceding claims 1-9, comprising: When the content management platform receives a login request submitted by a user through the operation interface, a user authentication request is sent to the authentication module based on the login request, so that the authentication module verifies the user authentication request, and when the verification is successful, an authorization token is sent to the content management platform; When the content management platform receives a task request submitted by a user through the operation interface, the authorization token and the task request are sent to the HPC control module, and the HPC control module is configured to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module verifies the task authentication request, and when the verification is successful, the HPC control module is configured to perform an operation corresponding to the task request.

Images