HPC system and task processing method

Abstract

Provided in the present invention are an HPC system and a task processing method. The HPC system comprises a content management platform, an authentication module, a database, and an HPC control module, wherein the content management platform is used for receiving a login request, and sending a user authentication request to the authentication module, such that the authentication module performs verification, and when the verification succeeds, the authentication module sends an authorization token to the content management platform; the content management platform is further used for receiving a task request, and sending the authorization token and the task request to the HPC control module; the HPC control module is used for sending a task authentication request to the authentication module on the basis of the authorization token and the task request, such that the authentication module performs verification; and when the verification succeeds, the HPC control module is used for determining whether the task request satisfies a quota requirement and whether the amount of required resources does not exceed the difference between a total amount of valid resources purchased by a user and the amount of used resources, and if both the conditions are met, the HPC control module executing an operation corresponding to the task request. The present invention can provide a leasable HPC service similar to public cloud services.

Description

HPC System and Task Processing Methods Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an HPC system and a task processing method. Background Technology

[0002] High-performance computing (HPC) is a technology that uses powerful processor clusters working in parallel to process massive multidimensional datasets (big data) and solve complex problems at extremely high speeds. HPC systems typically consist of computer clusters (i.e., HPC clusters) composed of multiple networked high-speed computer servers, running multiple tasks simultaneously on multiple computing nodes to achieve large-scale parallel computing and thus achieve faster running speeds.

[0003] However, current HPC systems cannot achieve data isolation between multiple different users and are generally only available to a single user. This not only fails to maximize resource utilization, but also makes it difficult for some users with limited resources to independently bear the cost of HPC systems. Therefore, it is hoped that HPC systems can provide rental services similar to public cloud services. Summary of the Invention

[0004] To address the shortcomings of the existing technology, the present invention aims to provide an HPC system task processing method to support multi-user use of the HPC system and provide leaseable HPC services.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An HPC system includes a content management platform, an authentication module, a database, and an HPC control module, wherein the content management platform provides a user interface, and:

[0007] The database is used to store data, which includes user information for each user, the total amount of valid resources purchased by each user, the amount of resources used by each user, and the quota requirements for each user.

[0008] The content management platform is used to receive login requests carrying user information submitted by users through the operation interface, and send user authentication requests to the authentication module based on the login requests, so that the authentication module can verify the user authentication requests. When the verification is successful, an authorization token is sent to the content management platform.

[0009] The content management platform is also used to receive task requests submitted by users through the operation interface, and send the authorization token and the task request to the HPC control module. The HPC control module is used to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module can verify the task authentication request and return the verification result to the HPC control module. When the verification is successful, the HPC control module is used to determine whether the task request meets the quota requirements corresponding to the user and whether the amount of resources required by the task request does not exceed the difference between the total amount of effective resources purchased by the user and the amount of resources used by the user. If both are true, the task corresponding to the task request is executed.

[0010] Furthermore, the amount of resources used by each user is statistically analyzed by the HPC control module and stored in the database. The resources include storage resources and computing resources, and the computing resources include GPU resource packages and CPU resource packages.

[0011] Furthermore, the quota requirements for each user include: the maximum number of GPUs a user can use, the maximum number of GPUs for a single task, the maximum number of instances for a single task, the maximum number of tasks to run, and the allowed runtime for a single task.

[0012] Furthermore, the HPC control module includes a controller and a scheduler management unit, a storage management unit, and a user management unit connected to the controller;

[0013] The scheduling management unit is connected to several schedulers and is used to manage the task scheduling of the several schedulers. Each scheduler is used to split the task to be executed into multiple sub-tasks, and then schedule different computing nodes to execute each sub-task, and summarize the execution results.

[0014] The storage management unit is used for data storage management;

[0015] The user management unit is used for user management;

[0016] The controller is used to control the scheduler management unit, storage management unit, or user management unit to perform corresponding operations based on the task request.

[0017] Furthermore, the content management platform is also used to receive user creation requests submitted by users through the operation interface, obtain an authorization token corresponding to the user information carried in the user creation request from the authentication module, and then send the user creation request and the authorization token to the HPC control module.

[0018] The HPC control module is also used to send the user creation request and the authorization token to the authentication module for verification. When the verification is successful, the HPC control module is also used to call the content management platform to create the user and register the user in the scheduler management unit and the storage management unit.

[0019] Furthermore, the content management platform is also used to store the user information carried in the user creation request into the database when it receives the user creation request.

[0020] Furthermore, after executing the task corresponding to the task request, the HPC control module is also used to provide feedback on the execution result through the operation interface provided by the content management platform.

[0021] Furthermore, the system also includes a login node connected to the control module, wherein the login node is equipped with a console-agent service.

[0022] Furthermore, the content management platform is connected to the database through the conductor module; the scheduler includes the Slurm scheduler and / or the OpenPB scheduler.

[0023] Secondly, the present invention provides a task processing method based on the aforementioned HPC system, comprising:

[0024] When the content management platform receives a login request submitted by a user through the operation interface, it sends a user authentication request to the authentication module based on the login request, so that the authentication module can verify the user authentication request. When the verification is successful, it sends an authorization token to the content management platform.

[0025] When the content management platform receives a task request submitted by a user through the operation interface, it sends the authorization token and the task request to the HPC control module. The HPC control module sends a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module can verify the task authentication request. When the verification is successful, the HPC control module determines whether the task request meets the user's quota requirements and whether the amount of resources required by the task request does not exceed the difference between the total amount of valid resources purchased by the user and the amount of resources already used by the user. If both are true, the task corresponding to the task request is executed.

[0026] By adopting the above technical solution, the present invention has the following beneficial effects:

[0027] By employing the aforementioned content management platform, authentication module, database, and HPC control module, this invention enables the HPC system to be leased to multiple users while ensuring data isolation between different users. This effectively protects the privacy and data security of each user while ensuring the full utilization of the HPC system. Attached Figure Description

[0028] Figure 1 is a schematic diagram of the HPC system in Embodiment 1 of the present invention;

[0029] Figure 2 is a schematic diagram of the HPC system in Embodiment 2 of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0031] The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. Example

[0032] This embodiment provides an HPC system, as shown in Figure 1, which includes a CMP (Content Management Platform), an authentication module, an HPC control module, and a database. Specifically, the HPC control module includes a controller and a scheduler management unit, a storage management unit, and a user management unit connected to the controller.

[0033] In this embodiment, the database is used to store data, including but not limited to user information corresponding to each user, the total amount of valid resources purchased by each user, the amount of resources used by each user, and the quota requirements corresponding to each user.

[0034] In this embodiment, the scheduler management unit is communicatively connected to several schedulers. Each scheduler is used to break down the task to be executed into multiple subtasks, then schedule different computing nodes to execute each subtask, and summarize the execution results; while the scheduling management unit is used to manage the task scheduling operations of each scheduler. It should be noted that this embodiment does not impose any restrictions on the number and type of schedulers; specifically, one, two, three, or more schedulers can be used, and the type of scheduler can be any one or more of schedulers such as Slurm and OpenPB.

[0035] In this embodiment, the storage management unit is communicatively connected to the memory for data storage management. The memory used in this embodiment can be one or more of Lustre, BGFS, CephFS, etc., and this embodiment does not impose any specific limitations on this. The stored data in the memory includes, but is not limited to, task-related data, such as temporary task data, task results, etc.

[0036] In this embodiment, the user management unit is connected to the user storage system for user management (including user creation, addition, deletion, modification, user group management, etc.). The user storage system used in this embodiment can be an OpenLDAP system or other systems.

[0037] In this embodiment, the controller is preferably implemented using ForceControl, which is pre-configured with various scheduler agents. When executing corresponding tasks, the scheduler management module, storage management module, and user management module can call the corresponding scheduler agent to perform task scheduling and return the scheduling results. In this embodiment, ForceControl plays a bridging role: for the upper layer, it provides API services for CMP; for the lower layer, it provides various drivers that can drive different memories and schedulers.

[0038] In this embodiment, the content management platform (CMP) provides users with a user-friendly interface, enabling different users to create and submit user creation requests, login requests, and task requests through the interface.

[0039] When the content management platform receives a user creation request (carrying user information such as username, password, and identity) through its user interface, it stores the user information in the database and sends the request to the authentication module. The authentication module then generates a corresponding authorization token based on the user information in the request and sends this token back to the content management platform. The content management platform then sends the received user creation request and the corresponding authorization token to the HPC control module. The HPC control module then sends the request and token to the authentication module for verification. If the authorization token is valid, the verification is successful, and the HPC control module calls the content management platform to create the user (based on the user information). After creation, the user is registered in the scheduler management unit and storage management unit, allowing the registered user to access tasks in the underlying storage and scheduler. It should be understood that users only have permission to access tasks in the storage and scheduler corresponding to their identity information.

[0040] After a user is created, when the content management platform receives a login request (carrying user information such as username, password, and identity) submitted by the user through the operation interface, it generates a user authentication request carrying the user information based on the login request and sends the user authentication request to the authentication module for verification. If the authentication module already has an authorization token corresponding to the user information in the request, the verification is successful, and the authorization token corresponding to the user information is returned to the content management platform, thus completing the user login. At this time, the user's private directory can be displayed on the operation interface.

[0041] After a user logs in, when the content management platform receives a task request (carrying task information such as task type and task identifier) ​​submitted by the user through the operation interface, it sends the aforementioned authorization token and task request to the HPC control module. The HPC control module then generates a task authentication request (carrying the token) based on the authorization token and task request, and sends this task authentication request to the authentication module so that the authentication module can verify the task authentication request and send the verification result back to the HPC control module. When the verification is successful (the user corresponding to the token has the permission to execute the corresponding task), the HPC control module determines whether the task request meets the user's corresponding quota requirements and whether the amount of resources required by the task request does not exceed the difference between the total amount of effective resources purchased by the user and the amount of resources already used by the user. If both are true, the task corresponding to the task request is executed. Specifically, this includes controlling the scheduler management unit, storage management unit, or user management unit to execute the corresponding task through the controller according to the task request.

[0042] After completing the operation corresponding to the task request, the HPC control module also provides feedback on the execution results through the operation interface provided by the content management platform. Users can see the task results in their private directory through the interface, which lists the files on the host by calling FTP.

[0043] This embodiment, by employing the aforementioned content management platform, authentication module, and HPC control module, enables the HPC system to be leased to multiple users while ensuring data isolation between different users. This effectively protects the privacy and data security of each user while fully utilizing the HPC system.

[0044] In this embodiment, the amount of resources used by each user is statistically analyzed by the HPC control module and stored in the database. These resources include storage resources and computing resources. The computing resources include GPU resource packages (unit: core time) and CPU resource packages (unit: card time). The CPU resource package usage is the number of CPU cores used multiplied by the running time, and the GPU resource package usage is the number of GPU cards used multiplied by the running time.

[0045] In one feasible approach, GPU resource packages and CPU resource packages do not support expansion or renewal. After a resource package is used up or expires, a new resource package can be purchased. Storage resources support expansion and renewal. Expansion refers to expanding the storage capacity package that is insufficient for a predetermined period (such as daily) to increase the deductible storage capacity for the predetermined period. Renewal refers to renewing the storage capacity package that is insufficient for the predetermined period to extend its usage time.

[0046] In this embodiment, the quota requirements configured for each user include: the maximum number of GPUs a user can use, the maximum number of GPUs per task, the maximum number of instances per task, the maximum number of tasks to run, and the allowed runtime for a single task. Specifically, the maximum number of GPUs a user can use refers to the maximum number of GPUs available to the user for a given GPU resource package; the maximum number of GPUs per task refers to the maximum number of GPUs available when executing a single task; the maximum number of instances per task refers to the maximum number of instances available when executing a single task; the maximum number of tasks to run refers to the maximum number of tasks that can run simultaneously; and the allowed runtime for a single task refers to the maximum duration a single task can run, after which the task will be terminated. In a preferred embodiment, a conductor is also provided between the CMP and the database to connect to the database based on a responsive API, thereby improving security. Conductor is an open-source workflow orchestration system that provides a unified API (Application Programming Interface), enabling users to easily manage, orchestrate, and monitor complex data tasks and machine learning pipelines.

[0047] In addition to configuring and submitting task requests, the user interface of this embodiment can also be used to manage and transfer job files, share data, and perform other operations. It is simple and easy to use, reducing the difficulties caused by complex command-line operations. Example

[0048] This embodiment provides another implementation of the HPC system of the present invention. As shown in Figure 2, the difference between this embodiment and embodiment 1 is that a login node connected to the control module is added, thereby providing an alternative user login method besides CMP.

[0049] In this embodiment, the login node deploys a console-agent service. This service exposes the console graphical interface and API. By configuring Novnc (a web-based VNC client that allows direct access to the VNC server via a browser) or DCV (DeepCloud VNC, an enhanced version of VNC offering better performance and security), VNC (Virtual Network Console) services can be provided to users. After logging in through this node, users can write tasks and perform debugging on it, and the node supports a graphical interface. Example

[0050] This embodiment provides a task processing method based on the aforementioned HPC system, specifically including the following steps:

[0051] When the content management platform receives a login request submitted by a user through the user interface, it sends a user authentication request to the authentication module based on the login request, so that the authentication module can verify the user authentication request. When the verification is successful, it sends an authorization token to the content management platform.

[0052] When the content management platform receives a task request submitted by a user through the operation interface, it sends the authorization token and the task request to the HPC control module. The HPC control module sends a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module can verify the task authentication request. When the verification is successful, the HPC control module determines whether the task request meets the quota requirements corresponding to the user and whether the amount of resources required by the task request does not exceed the difference between the total amount of effective resources purchased by the user and the amount of resources already used by the user. If both are true, the operation corresponding to the task request is executed, specifically including: controlling the scheduler management unit, storage management unit or user management unit to perform corresponding operations through the controller according to the task request.

[0053] After the HPC control module completes the operation corresponding to the task request, it provides feedback on the execution result through the operation interface provided by the content management platform. Users can see the task result in their private directory through the interface, which lists the files on the host by calling FTP.

[0054] The method described in this embodiment can provide an HPC system to multiple users while ensuring data isolation between different users, thereby effectively protecting the privacy and data security of each user while making full use of the HPC system.

[0055] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. An HPC system, characterized in that, It includes a content management platform, an authentication module, a database, and an HPC control module, wherein the content management platform provides a user interface, wherein: The database is used to store data, which includes user information for each user, the total amount of valid resources purchased by each user, the amount of resources used by each user, and the quota requirements for each user. The content management platform is used to receive login requests carrying user information submitted by users through the operation interface, and send user authentication requests to the authentication module based on the login requests, so that the authentication module can verify the user authentication requests. When the verification is successful, an authorization token is sent to the content management platform. The content management platform is also used to receive task requests submitted by users through the operation interface, and send the authorization token and the task request to the HPC control module. The HPC control module is used to send a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module can verify the task authentication request and return the verification result to the HPC control module. When the verification is successful, the HPC control module is used to determine whether the task request meets the quota requirements corresponding to the user and whether the amount of resources required by the task request does not exceed the difference between the total amount of effective resources purchased by the user and the amount of resources already used by the user. If both are true, the operation corresponding to the task request is executed.

2. The HPC system as described in claim 1, characterized in that, The amount of resources used by each user is counted by the HPC control module and stored in the database. The resources include storage resources and computing resources, and the computing resources include GPU resource packages and CPU resource packages.

3. The HPC system as described in claim 2, characterized in that, The quota requirements for each user include: the maximum number of GPUs a user can use, the maximum number of GPUs that can be used in a single task, the maximum number of instances that can be run in a single task, the maximum number of tasks that can be run, and the allowed runtime for a single task.

4. The HPC system as described in claim 1, characterized in that, The HPC control module includes a controller and a scheduler management unit, a storage management unit, and a user management unit connected to the controller. The scheduling management unit is connected to several schedulers and is used to manage the task scheduling of the several schedulers. Each scheduler is used to split the task to be executed into multiple sub-tasks, and then schedule different computing nodes to execute each sub-task, and summarize the execution results. The storage management unit is used for data storage management; The user management unit is used for user management; The controller is used to control the scheduler management unit, storage management unit, or user management unit to perform corresponding operations based on the task request.

5. The HPC system as described in claim 4, characterized in that, The content management platform is also used to receive user creation requests submitted by users through the operation interface, obtain an authorization token corresponding to the user information carried in the user creation request from the authentication module, and then send the user creation request and the authorization token to the HPC control module. The HPC control module is also used to send the user creation request and the authorization token to the authentication module for verification. When the verification is successful, the HPC control module is also used to call the content management platform to create the user and register the user in the scheduler management unit and the storage management unit.

6. The HPC system as described in claim 5, characterized in that, The content management platform is also used to store the user information carried in the user creation request into the database when it receives the user creation request.

7. The HPC system as described in claim 1, characterized in that, After completing the task corresponding to the task request, the HPC control module is also used to provide feedback on the execution result through the operation interface provided by the content management platform.

8. The HPC system as described in claim 1, characterized in that, The system also includes a login node connected to the control module, wherein the login node is equipped with a console-agent service.

9. The HPC system as described in claim 1, characterized in that, The content management platform is connected to the database through the conductor module; the scheduler includes the Slurm scheduler and / or the OpenPB scheduler.

10. A task processing method based on an HPC system according to any one of claims 1-9, comprising: When the content management platform receives a login request submitted by a user through the operation interface, it sends a user authentication request to the authentication module based on the login request, so that the authentication module can verify the user authentication request. When the verification is successful, it sends an authorization token to the content management platform. When the content management platform receives a task request submitted by a user through the operation interface, it sends the authorization token and the task request to the HPC control module. The HPC control module sends a task authentication request to the authentication module based on the authorization token and the task request, so that the authentication module can verify the task authentication request. When the verification is successful, the HPC control module determines whether the task request meets the user's quota requirements and whether the amount of resources required by the task request does not exceed the difference between the total amount of valid resources purchased by the user and the amount of resources already used by the user. If both are true, the operation corresponding to the task request is executed.

Images