Cloud native HPC application processing method and system and medium

Abstract

The invention provides a cloud native HPC application processing method and system and a medium. The method comprises the steps that a K8S cluster is adopted as a basic deployment environment of a high-performance computing HPC application, an application parameter request is generated according to application parameters input by a user interface, and the application parameters comprise to-be-processed HPC application information and a corresponding task starting rule file; generating an HPC client process corresponding to the to-be-processed HPC application according to the application parameter request; according to each HPC client process, generating and starting a corresponding HPC task; and determining a K8S cluster host corresponding to each HPC task, and issuing the HPC tasks, so that the K8S cluster hosts respectively complete the operation of the corresponding HPC tasks. According to the method provided by the invention, the deployment and maintenance cost of the HPC application is reduced, and the efficiency and accuracy of application configuration / updating are improved.

Description

Technical Field

[0001] This application relates to the field of cloud platforms, and particularly to a cloud-native HPC application processing method, system, and medium. Background Art

[0002] High Performance Computing (HPC) is a technical science that utilizes supercomputers to implement parallel computing theories, methods, technologies, and applications. High performance computing has been applied to various fields of computer communication.

[0003] Currently, traditional HPC applications are deployed on physical hosts. Before an HPC application actually runs, a large number of dependent resources need to be installed, and an HPC scheduler, such as Slurm, also needs to be deployed. The entire deployment process is very cumbersome. Taking the WRF application in the meteorological industry as an example, for a user to run the WRF application, they need to deploy the required dependent packages of the WRF application on all physical hosts and deploy the Slurm scheduling cluster at the same time. When running the WRF application, the user needs to use the command line to view the running status of the tasks and also needs to pay attention to the running status of the Slurm cluster. However, this way of deploying HPC applications on physical hosts requires a large amount of time and resources, with both deployment and maintenance costs being relatively high, and the configuration and software update processes require a large amount of human resources, which is not only inefficient but also error-prone.

[0004] Therefore, there is a need for a cloud-native HPC application processing solution that can reduce the deployment and maintenance costs of HPC applications and improve the efficiency and accuracy of configuration / updating. Summary of the Invention

[0005] This application provides a cloud-native HPC application processing method, system, and medium to solve the technical problems of high deployment and maintenance costs of existing HPC applications and poor efficiency and accuracy of configuration / updating.

[0006] In a first aspect, this application provides a cloud-native HPC application processing method, including:

[0007] Using a K8S cluster as the basic deployment environment for high-performance computing HPC applications, generating an application parameter request according to the application parameters input through the user interface, where the application parameters include the HPC application information to be processed and the corresponding task start rule file;

[0008] Generating an HPC client process corresponding to the HPC application to be processed according to the application parameter request;

[0009] Generating and starting a corresponding HPC task according to each HPC client process;

[0010] Determine the K8S cluster hosts corresponding to each HPC task, and issue the HPC tasks, so that each K8S cluster host completes the operation of the corresponding HPC task.

[0011] In a possible implementation manner, the generating an application parameter request according to the application parameters input through the user interface specifically includes:

[0012] Obtain the HPC application image selected by the user, and determine the corresponding HPC application information to be processed and the default template parameter information according to the HPC application image;

[0013] Generate a task start rule file according to the custom parameter information input by the user and the default template parameter information corresponding to the HPC application image, where the task start rule file includes start parameters, the number of tasks, and the number of resources;

[0014] Generate an application parameter request according to the HPC application information to be processed and the task start rule file.

[0015] In a possible implementation manner, the determining the K8S cluster hosts corresponding to each HPC task and issuing the HPC tasks specifically includes:

[0016] Determine the host resources required for each HPC task;

[0017] Determine the K8S cluster hosts corresponding to each HPC task according to the current resource occupancy information of each K8S cluster host and the host resources required for each HPC task;

[0018] Schedule the HPC tasks, and issue each HPC task to its corresponding K8S cluster host, where the HPC tasks include computing tasks and data processing tasks.

[0019] In a possible implementation manner, the method further includes:

[0020] Determine the data disk information required for each HPC task according to the task start rule file, where the data disk information includes the data disk capacity and the data disk type;

[0021] Correspondingly, the issuing each HPC task to its corresponding K8S cluster host includes:

[0022] Issue each HPC task and the data disk information required for each HPC task to its corresponding K8S cluster host.

[0023] In a possible implementation manner, the determining the host resources required for each HPC task specifically includes:

[0024] For each HPC task,

[0025] Determining resource parameters corresponding to the HPC task, the resource parameters including resource sharing information, resource isolation information, task preemption and anti-preemption, task priority and quota, and task-level associated processes;

[0026] Determine the host resources required for the HPC task based on the resource parameters corresponding to the HPC task.

[0027] In one possible implementation, the method further includes:

[0028] Monitor the health status of each HPC task;

[0029] If a target HPC task with an abnormal health status is monitored, the deployment component in the K8S cluster is used to automatically restart the target HPC task and output task alarm information.

[0030] In one possible implementation, the method further includes:

[0031] Monitor the current resource usage of each K8S cluster host;

[0032] If the target host whose current resource occupancy information exceeds the resource occupancy threshold is monitored, the pending HPC task on the target host is migrated to the K8S cluster host whose current resource occupancy information does not exceed the resource occupancy threshold to complete the pending HPC task.

[0033] In a second aspect, the present application provides a cloud-native HPC application processing system, comprising:

[0034] A user interface for obtaining application parameters input by a user, generating an application parameter request based on the application parameters, and sending the parameter request to the HPC client; the application parameters include the HPC application information to be processed and the corresponding task startup rule file;

[0035] The HPC client is configured to generate an HPC client process corresponding to the HPC application to be processed according to the application parameter request; and generate and start a corresponding HPC task according to each HPC client process;

[0036] The scheduler is used to determine the K8S cluster host corresponding to each HPC task and issue the HPC task;

[0037] K8S cluster host, used to complete the operation of corresponding HPC tasks;

[0038] Monitor, used to monitor the health status of each HPC task and the current resource usage of each K8S cluster host;

[0039] High-performance storage for providing the data disks required to run each HPC task;

[0040] High-performance network for providing the network environment required to run each HPC task.

[0041] In a third aspect, the present application provides another cloud-native HPC application processing system, including: a processor, and a memory communicatively connected to the processor;

[0042] The memory stores computer-executable instructions;

[0043] The processor executes the computer-executable instructions stored in the memory to implement the above method.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the above method.

[0045] In a fifth aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0046] The cloud-native HPC application processing method, system and medium provided by the present application can use a K8S cluster as the basic deployment environment for high-performance computing HPC applications, generate an application parameter request according to the application parameters input by the user interface, and the application parameters include the HPC application information to be processed and the corresponding task start rule file; according to the application parameter request, generate an HPC client process corresponding to the HPC application to be processed; according to each HPC client process, generate and start a corresponding HPC task; determine the K8S cluster host corresponding to each HPC task, and issue the HPC task, so that each K8S cluster host completes the operation of the corresponding HPC task. The method of the present application no longer deploys HPC applications on physical hosts, but deploys them on a cloud platform. When a user wants to install or update a certain HPC application, directly select or input the corresponding application parameters on the user interface, and the installation or update of the HPC application can be completed, reducing the deployment and maintenance costs of the HPC application, and this process does not require manual intervention, greatly improving the efficiency and accuracy of HPC application configuration / updating. Further, during the HPC application processing, the HPC client will generate and start a corresponding HPC task according to the application parameter request, and the scheduler will perform task scheduling, issue the HPC task to a suitable K8S cluster host, and dynamically adjust the computing resources according to the actual requirements of the task, which can not only meet the resource requirements but also not waste resources, further reducing the deployment and maintenance costs of the HPC application. Description of the Drawings

[0047] The accompanying drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0048] Figure 1 It is a flowchart of a cloud-native HPC application processing method according to an embodiment of this application;

[0049] Figure 2 It is a schematic structural diagram of a cloud-native HPC application processing system according to an embodiment of this application;

[0050] Figure 3 It is a schematic structural diagram of a cloud-native HPC application processing system according to another embodiment of this application.

[0051] Through the above-mentioned accompanying drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of this application in any way, but to explain the concept of this application to those skilled in the art by referring to specific embodiments. Detailed Description of Specific Embodiments

[0052] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0054] It should also be noted that the cloud-native HPC application processing method, system, and medium of this application can be used in the field of cloud platforms, and can also be used in any field other than the field of cloud platforms, such as the field of HPC applications, etc. The application fields of the cloud-native HPC application processing method, system, and medium of this application are not limited.

[0055] First, the terms involved in this application are explained:

[0056] HPC: Abbreviation for High-Performance Computing, which means high-performance computing.

[0057] WRF: Weather Research and Forecasting, a state-of-the-art atmospheric modeling system designed for meteorological research and numerical weather prediction. It offers numerous options for atmospheric processes and can run on various computing platforms.

[0058] Slurm: Simple Linux Utility for Resource Management, is a widely used open-source scheduler.

[0059] K8s, kubernetes, is an abbreviation formed by replacing the eight characters "ubernete" in the middle of the name with 8. K8s is an open-source tool for managing containerized applications on multiple hosts in a cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient (powerful). Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance.

[0060] Cloud native is a set of cloud technology product systems based on distributed clouds with distributed deployment and unified operation and management, and is built on technologies such as containers, microservices, and DevOps.

[0061] An image is a lightweight, executable independent software package used to package a software running environment and the software developed based on the running environment. It contains all the content required to run a certain software, including code, runtime, libraries, environment variables, and configuration files.

[0062] The cloud-native HPC application processing method, system, and medium can be applied to any scenario that requires the deployment of high-performance computing applications, such as the deployment scenario of WRF applications in the meteorological industry, etc., without any restrictions here.

[0063] Currently, traditional HPC applications are deployed on physical hosts. Before the HPC application actually runs, a large number of dependent resources need to be installed, and at the same time, an HPC scheduler, such as Slurm, needs to be deployed. The entire deployment process is very cumbersome. Taking the WRF application in the meteorological industry as an example, users who want to run the WRF application need to deploy the required dependent packages of the WRF application on all physical hosts and deploy the Slurm scheduling cluster at the same time. When running the WRF application, users need to use the command line to view the running status of tasks and also need to pay attention to the running status of the Slurm cluster.

[0064] However, this way of deploying HPC applications on physical hosts requires a large amount of time and resources, with relatively high deployment and maintenance costs. Moreover, the configuration and software update processes require a large amount of human resources, which is not only inefficient but also error-prone.

[0065] Based on this technical problem, the inventive concept of this application lies in: how to provide a cloud-native HPC application processing method that can reduce the deployment and maintenance costs of HPC applications and improve the efficiency and accuracy of configuration / updating.

[0066] Specifically, a K8S cluster can be used as the basic deployment environment for high-performance computing (HPC) applications. An application parameter request is generated according to the application parameters input through the user interface. The application parameters include the HPC application information to be processed and the corresponding task startup rule file. According to the application parameter request, an HPC client process corresponding to the HPC application to be processed is generated. According to each HPC client process, a corresponding HPC task is generated and started. The K8S cluster host corresponding to each HPC task is determined, and the HPC task is dispatched, so that each K8S cluster host can complete the operation of the corresponding HPC task. The method of this application no longer deploys HPC applications to physical hosts, but deploys them on the cloud platform. When a user wants to install or update a certain HPC application, the corresponding application parameters can be directly selected or input through the user interface, and the installation or update of the HPC application can be completed, reducing the deployment and maintenance costs of the HPC application. Moreover, this process does not require manual intervention, greatly improving the efficiency and accuracy of HPC application configuration / updating. Further, during the HPC application processing, the HPC client will generate and start the corresponding HPC task according to the application parameter request, and the scheduler will perform task scheduling, dispatch the HPC task to the appropriate K8S cluster host, and dynamically adjust the computing resources according to the actual requirements of the task, which can not only meet the resource requirements but also avoid wasting resources, further reducing the deployment and maintenance costs of the HPC application.

[0067] The technical solution of this application and how this technical solution solves the above technical problems will be described in detail below with specific embodiments. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0068] Embodiment 1

[0069] Figure 1 It is a flowchart of the cloud-native HPC application processing method according to an embodiment of this application. In this embodiment, the cloud-native HPC application processing method is described with the execution subject being the cloud-native HPC application processing system. As Figure 1 shown, the cloud-native HPC application processing method may include the following steps:

[0070] S101: Use a K8S cluster as the basic deployment environment for high-performance computing (HPC) applications, and generate an application parameter request according to the application parameters input through the user interface.

[0071] In this embodiment, the application parameters may include the HPC application information to be processed and the corresponding task startup rule file.

[0072] In this embodiment, K8s is used to manage containerized applications on multiple hosts in a cloud platform. Therefore, a K8S cluster is adopted as the basic deployment environment for high-performance computing (HPC) applications. That is, containerization technology is used, which can achieve fast and consistent deployment and update of HPC applications. Originally, it took one week to update an HPC application, but after adopting containerized deployment, the software deployment and update can be completed in only a few hours.

[0073] In a possible implementation manner, generating an application parameter request according to the application parameters input through the user interface in step S101 may include:

[0074] S1011: Obtain the HPC application image selected by the user, and determine the corresponding HPC application information to be processed and the default template parameter information according to the HPC application image.

[0075] S1012: Generate a task startup rule file according to the custom parameter information input by the user and the default template parameter information corresponding to the HPC application image. The task startup rule file includes startup parameters, the number of tasks, and the number of resources.

[0076] S1013: Generate an application parameter request according to the HPC application information to be processed and the task startup rule file.

[0077] In this implementation manner, the user interface of the cloud-native HPC application processing system can be used for user login authentication and provide functions such as HPC cluster management and HPC task management for users.

[0078] In this implementation manner, after the user logs in to the user interface of the cloud-native HPC application processing system, the user can select an HPC application image in the user interface according to the needs, and the cloud-native HPC application processing system can determine the corresponding HPC application information to be processed and the default template parameter information according to the selected HPC application image.

[0079] In this implementation manner, the task startup rule file may be an hpc.yaml file. The HPC application information to be processed may be information such as the number and name of the HPC application image selected by the user that can identify the application image. The default template parameter information may be the default parameters corresponding to the HPC application image preset by the staff, and the user does not need to fill in the information by himself / herself. The user only needs to input the custom parameter information according to his / her own needs, which reduces the probability of parameter information errors and thus ensures the smooth processing of subsequent HPC applications.

[0080] In this embodiment, when traditional HPC applications need to be upgraded or their configurations modified, a large number of configurations need to be manually adjusted. It is necessary to upgrade the dependent environment and uninstall and replace the application programs on all nodes. Since the whole process is complex and there are many places that need to be modified, it takes a lot of time. At the same time, errors may occur due to manual intervention. When the cloud-native HPC application of this application needs to be upgraded, it only needs to repackage and upgrade the application image, and modify the image version in the corresponding task startup rule file to upgrade the HPC applications on all nodes. There is no need for excessive manual intervention in this process, which improves the efficiency and accuracy of HPC application configuration / updating.

[0081] S102: Generate an HPC client process corresponding to the to-be-processed HPC application according to the application parameter request.

[0082] In this embodiment, after the K8S cluster receives the parameter request from the user interface, it will generate an HPC client pod (HPC client process) according to the hpc.yaml file (task startup rule file), and the HPC client pod will start an HPC task (HPC task) to execute the task.

[0083] S103: Generate and start a corresponding HPC task according to each HPC client process.

[0084] In this embodiment, the HPC task is deployed in the form of a job. The task startup rule can be set in the deployed yaml file. The task will continue to run only after the specified number of HPC tasks are successfully started. Otherwise, it will prompt that the task fails.

[0085] In this embodiment, when the kubectl delete -f hpc.yaml command is executed on K8S, the HPC task will be deleted.

[0086] S104: Determine the K8S cluster host corresponding to each HPC task, and issue the HPC task, so that each K8S cluster host completes the operation of the corresponding HPC task.

[0087] In a possible embodiment, determining the K8S cluster host corresponding to each HPC task and issuing the HPC task in step S104 above may include:

[0088] S1041: Determine the host resources required for each HPC task.

[0089] S1042: Determine the K8S cluster host corresponding to each HPC task according to the current resource occupancy information of each K8S cluster host and the host resources required for each HPC task.

[0090] S1043: Schedule the HPC tasks and send each HPC task to its corresponding K8S cluster host. The HPC tasks include computing tasks and data processing tasks.

[0091] In this embodiment, the volcano scheduler in the cloud-native HPC application processing system is responsible for scheduling HPC tasks to different nodes and applying resources for them. To ensure the smooth progress of HPC tasks, the scheduler can determine the K8S cluster host corresponding to each HPC task according to the current resource occupancy information of each K8S cluster host and the host resources required by each HPC task, so that each HPC task can be scheduled to a suitable K8S cluster host to successfully complete each HPC task.

[0092] In a possible implementation, determining the host resources required for each HPC task in the above step S1041 may include: for each HPC task, determining the resource parameters corresponding to the HPC task. The resource parameters include resource sharing information, resource isolation information, task preemption and anti-preemption, the priority and quota of the task, and the associated processes at the task level; determining the host resources required for the HPC task according to the resource parameters corresponding to the HPC task.

[0093] In this embodiment, the volcano scheduler in the cloud-native HPC application processing system can accurately determine the host resources required for an HPC task according to the resource parameters corresponding to each HPC task. Traditional HPC applications are deployed on physical hosts, and when the computing task demand is small, resources will be idle, resulting in resource waste. However, when the computing demand surges and expansion is needed due to insufficient computing power, it takes a lot of time and money to purchase and configure new hardware. The cloud-native HPC application of this application can apply for any amount of resources according to the demand and can dynamically adjust the computing resources according to the actual demand of the task. For example, when the computing demand increases during a certain period, computing nodes can be automatically added, and if the computing demand decreases, computing nodes can be automatically reduced, so that both resource requirements can be met and resources will not be wasted.

[0094] In a possible implementation, the method may further include: determining the data disk information required for each HPC task according to the task start rule file. The data disk information includes the data disk capacity and the data disk type.

[0095] Correspondingly, sending each HPC task to its corresponding K8S cluster host in the above step S1043 may include: sending each HPC task and the data disk information required for each HPC task to its corresponding K8S cluster host.

[0096] In this embodiment, based on the characteristics of HPC applications, HPC tasks may use data disks, and the cloud-native HPC application processing system further includes providing high-performance storage required for each HPC task. Therefore, when each HPC task is sent to its corresponding K8S cluster host, the corresponding data disk information also needs to be sent to ensure the smooth progress of the HPC task, and the data disk information can be determined by the task start rule file.

[0097] In a possible embodiment, the method may further include: monitoring the health status of each HPC task; if a target HPC task with an abnormal health status is monitored, automatically restart the target HPC task using the deployment component in the K8S cluster, and output a task warning message.

[0098] In this embodiment, the Monitor monitor in the cloud-native HPC application processing system can access the K8S cluster through the Restful interface and monitor the health status of each HPC task on it; when an abnormal health status of an HPC task is found, it can pull up or repair the HPC task in time to ensure the normal operation of the task.

[0099] In this embodiment, the Monitor monitor in the cloud-native HPC application processing system can monitor the status of HPC tasks running in the K8S cluster. When the health status of an HPC task in the K8S cluster is abnormal, the deployment in the K8S cluster can automatically restart the failed task, enabling the task to return to normal to ensure the smooth progress of the HPC task, thus eliminating manual operation and maintenance. When a traditional HPC application runs and a certain computing node fails, it will directly cause the task to fail and requires manual intervention. The entire troubleshooting process is complex and cumbersome, greatly increasing the operation and maintenance cost. When the HPC task pod status in this application's cloud-native HPC application is abnormal, the Monitor will detect this situation and give an alarm.

[0100] In a possible embodiment, the method may further include: monitoring the current resource occupancy information of each K8S cluster host; if a target host whose current resource occupancy information exceeds the resource occupancy threshold is monitored, migrate the HPC task to be performed on the target host to a K8S cluster host whose current resource occupancy information does not exceed the resource occupancy threshold to complete the HPC task to be performed.

[0101] In this embodiment, the Monitor monitor in the cloud-native HPC application processing system can also monitor the resource situation of the K8S cluster hosts. When a certain host node has insufficient resources, the volcano scheduler will automatically migrate the tasks of this node to other idle nodes to further ensure the smooth progress of the HPC tasks.

[0102] In this embodiment, instead of deploying the HPC application to a physical host, it is deployed on a cloud platform. When a user wants to install or update a certain HPC application, the user directly selects or enters the corresponding application parameters in the user interface, and then the installation or update of the HPC application can be completed. This reduces the deployment and maintenance costs of the HPC application, and this process does not require manual intervention, greatly improving the efficiency and accuracy of HPC application configuration / updating. Further, during the HPC application processing, the HPC client will generate and start the corresponding HPC task according to the application parameter request, and the scheduler will perform task scheduling, send the HPC task to the appropriate K8S cluster host, and dynamically adjust the computing resources according to the actual requirements of the task, which can not only meet the resource requirements but also avoid wasting resources, further reducing the deployment and maintenance costs of the HPC application.

[0103] The following uses a specific embodiment to elaborate on the cloud-native HPC application processing method of the present application.

[0104] Embodiment 2

[0105] In a specific embodiment, an enterprise needs to deploy an HPC application. The specific cloud-native HPC application processing process is as follows:

[0106] First step, after the user logs in to the user interface of the cloud-native HPC application processing system, the user selects the HPC application image in the user interface according to the requirements and enters the custom parameter information. The user interface generates a task start rule file according to the custom parameter information entered by the user and the default template parameter information corresponding to the HPC application image, generates an application parameter request according to the HPC application information to be processed corresponding to the HPC application image and the task start rule file, and sends it to the HPC client.

[0107] Second step, the HPC client of the cloud-native HPC application processing system generates an HPC client process corresponding to the HPC application to be processed according to the application parameter request; generates and starts the corresponding HPC task according to each HPC client process.

[0108] Third step, the volcano scheduler of the cloud-native HPC application processing system determines the host resources required for each HPC task; determines the K8S cluster host corresponding to each HPC task according to the current resource occupancy information of each K8S cluster host and the host resources required for each HPC task; schedules the HPC tasks and sends each HPC task to its corresponding K8S cluster host.

[0109] In the fourth step, the Monitor monitor of the cloud-native HPC application processing system monitors the current resource occupancy information of each K8S cluster host and the health status of each HPC task. If a target HPC task with an abnormal health status is monitored, the deployment component in the K8S cluster is used to automatically restart the target HPC task and output task warning information; if a target host with current resource occupancy information exceeding the resource occupancy threshold is monitored, the HPC tasks to be processed on the target host are migrated to a K8S cluster host with current resource occupancy information not exceeding the resource occupancy threshold to complete the HPC tasks to be processed.

[0110] In the fifth step, the K8S cluster hosts of the cloud-native HPC application processing system complete the operation of the corresponding HPC tasks to complete the deployment of the HPC application.

[0111] Figure 2 It is a schematic structural diagram of the cloud-native HPC application processing system according to an embodiment of the present application. As Figure 2 shown, the cloud-native HPC application processing system includes: a user interface 21 for obtaining application parameters input by a user, generating an application parameter request according to the application parameters, and sending the parameter request to the HPC client; the application parameters include the HPC application information to be processed and the corresponding task start rule file; an HPC client 22 for generating an HPC client process corresponding to the HPC application to be processed according to the application parameter request; generating and starting a corresponding HPC task according to each HPC client process; a scheduler 23 for determining the K8S cluster host corresponding to each HPC task and issuing the HPC task; a K8S cluster host 24 for completing the operation of the corresponding HPC task; a monitor 25 for monitoring the health status of each HPC task and the current resource occupancy information of each K8S cluster host; a high-performance storage 26 for providing a data disk required for running each HPC task; a high-performance network 27 for providing a network environment required for running each HPC task. In one implementation manner, the description of the specific functions implemented by the cloud-native HPC application processing system can refer to steps S101-S104 in Embodiment 1, which will not be elaborated here.

[0112] In a possible implementation manner, the user interface 21 can also be used to:

[0113] Obtain the HPC application image selected by the user, and determine the corresponding HPC application information to be processed and the default template parameter information according to the HPC application image;

[0114] Generate a task start rule file according to the custom parameter information input by the user and the default template parameter information corresponding to the HPC application image. The task start rule file includes start parameters, the number of tasks, and the number of resources;

[0115] Generate an application parameter request according to the HPC application information to be processed and the task start rule file.

[0116] In this embodiment, for the description of the specific functions implemented by the user interface 21, reference can be made to step S101 in the first embodiment, which will not be elaborated here.

[0117] In a possible implementation manner, the scheduler 23 can also be used for:

[0118] Determine the host resources required for each HPC task;

[0119] According to the current resource occupancy information of each K8S cluster host and the host resources required for each HPC task, determine the K8S cluster host corresponding to each HPC task;

[0120] Schedule the HPC tasks and send each HPC task to its corresponding K8S cluster host. The HPC tasks include computing tasks and data processing tasks.

[0121] In this embodiment, for the description of the specific functions implemented by the scheduler 23, reference can be made to step S104 in the first embodiment, which will not be elaborated here.

[0122] In a possible implementation manner, the scheduler 23 can also be used for:

[0123] According to the task start rule file, determine the data disk information required for each HPC task. The data disk information includes the data disk capacity and the data disk type;

[0124] Correspondingly, sending each HPC task to its corresponding K8S cluster host includes:

[0125] Send each HPC task and the data disk information required for each HPC task to its corresponding K8S cluster host.

[0126] In this embodiment, for the description of the specific functions implemented by the scheduler 23, reference can be made to step S104 in the first embodiment, which will not be elaborated here.

[0127] In a possible implementation manner, the scheduler 23 can also be used for:

[0128] For each HPC task,

[0129] Determine the resource parameters corresponding to the HPC task. The resource parameters include resource sharing information, resource isolation information, task preemption and anti-preemption, the priority and quota of the task, and the associated processes at the task level;

[0130] According to the resource parameters corresponding to the HPC task, determine the host resources required for the HPC task.

[0131] In this embodiment, for the description of the specific implementation of the functions of the scheduler 23, reference can be made to step S104 in the first embodiment, which will not be elaborated here.

[0132] In a possible implementation, the monitor 25 can also be used for:

[0133] Monitoring the health status of each HPC task;

[0134] If a target HPC task with an abnormal health status is monitored, the deployment component in the K8S cluster is used to automatically restart the target HPC task and output a task warning message.

[0135] In this embodiment, for the description of the specific implementation of the functions of the monitor 25, reference can be made to step S104 in the first embodiment, which will not be elaborated here.

[0136] In a possible implementation, the monitor 25 can also be used for:

[0137] Monitoring the current resource occupancy information of each K8S cluster host;

[0138] If a target host whose current resource occupancy information exceeds the resource occupancy threshold is monitored, the to-be-executed HPC task on the target host is migrated to a K8S cluster host whose current resource occupancy information does not exceed the resource occupancy threshold to complete the to-be-executed HPC task.

[0139] In this embodiment, for the description of the specific implementation of the functions of the monitor 25, reference can be made to step S104 in the first embodiment, which will not be elaborated here.

[0140] Figure 3 It is a schematic structural diagram of a cloud-native HPC application processing system according to another embodiment of the present application. As Figure 3 shown, the cloud-native HPC application processing system includes: a processor 101 and a memory 102 communicatively connected to the processor 101; the memory 102 stores computer-executable instructions; the processor 101 executes the computer-executable instructions stored in the memory 102 to implement the steps of the cloud-native HPC application processing method in the above-mentioned method embodiments.

[0141] The cloud-native HPC application processing system can be independent or a part of a cloud platform, and the processor 101 and the memory 102 can adopt the existing hardware of the cloud platform.

[0142] In the above cloud-native HPC application processing system, the memory 102 and the processor 101 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines, such as through a bus connection. The memory 102 stores computer-executable instructions for implementing the data access control method, including at least one software function module that can be stored in the memory 102 in the form of software or firmware. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102.

[0143] The memory 102 can be, but is not limited to, a random access memory (Random Access Memory, abbreviated as RAM), a read-only memory (Read Only Memory, abbreviated as ROM), a programmable read-only memory (Programmable Read-Only Memory, abbreviated as PROM), an erasable programmable read-only memory (Erasable Programmable Read-Only Memory, abbreviated as EPROM), an electrically erasable programmable read-only memory (Electric Erasable Programmable Read-Only Memory, abbreviated as EEPROM), etc. Among them, the memory 102 is used to store programs, and the processor 101 executes the programs after receiving the execution instructions. Further, the software programs and modules in the memory 102 may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide a running environment for other software components.

[0144] The processor 101 can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 101 can be a general-purpose processor, including a central processing unit (Central Processing Unit, abbreviated as CPU), a network processor (Network Processor, abbreviated as NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0145] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the steps of the method embodiments of the present application.

[0146] An embodiment of the present application further provides a computer program product, including a computer program which, when executed by a processor, implements the steps of the method embodiments of the present application.

[0147] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be in other sequences or be performed simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0148] Furthermore, it should be noted that although the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0149] It should be understood that the above device embodiments are illustrative, and the devices of the present application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0150] In addition, without special explanation, in each embodiment of the present application, each functional unit / module can be integrated in one unit / module, or each unit / module can exist physically alone, or two or more units / modules can be integrated together. The above integrated unit / module can be implemented in the form of hardware or in the form of a software program module.

[0151] When the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0152] When the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned memory includes: USB flash drives, read-only memory (ROM), random access memory (RAM), mobile hard disks, magnetic disks, or optical discs, etc., which are various media that can store program codes.

[0153] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0154] Other embodiments of the present application will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include well-known knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only illustrative, and the true scope and spirit of the present application are pointed out by the appended claims.

[0155] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A cloud-native HPC application processing method, characterized in that, Including: Using a K8S cluster as the basic deployment environment for high-performance computing (HPC) applications, generating an application parameter request according to the application parameters input through the user interface, where the application parameters include the HPC application information to be processed and the corresponding task start rule file; Generating an HPC client process corresponding to the HPC application to be processed according to the application parameter request; Generating and starting a corresponding HPC task according to each HPC client process; Determining the K8S cluster host corresponding to each HPC task and issuing the HPC task, so that each K8S cluster host completes the operation of the corresponding HPC task.

2. The method according to claim 1, wherein The generating an application parameter request according to the application parameters input through the user interface specifically includes: Obtaining the HPC application image selected by the user, and determining the corresponding HPC application information to be processed and the default template parameter information according to the HPC application image; Generating a task start rule file according to the custom parameter information input by the user and the default template parameter information corresponding to the HPC application image, where the task start rule file includes start parameters, the number of tasks, and the number of resources; Generating an application parameter request according to the HPC application information to be processed and the task start rule file.

3. The method according to claim 2, wherein The determining the K8S cluster host corresponding to each HPC task and issuing the HPC task specifically includes: Determining the host resources required for each HPC task; Determining the K8S cluster host corresponding to each HPC task according to the current resource occupancy information of each K8S cluster host and the host resources required for each HPC task; Scheduling the HPC tasks and issuing each HPC task to its corresponding K8S cluster host, where the HPC tasks include computing tasks and data processing tasks.

4. The method according to claim 3, characterized in that The method further includes: Determining the data disk information required for each HPC task according to the task start rule file, where the data disk information includes the data disk capacity and the data disk type; Correspondingly, the issuing each HPC task to its corresponding K8S cluster host includes: Issuing each HPC task and the data disk information required for each HPC task to its corresponding K8S cluster host.

5. The method according to claim 4, characterized in that, The determining the host resources required for each HPC task specifically includes: For each HPC task, Determining the resource parameters corresponding to the HPC task, where the resource parameters include resource sharing information, resource isolation information, task preemption and anti-preemption, the priority and quota of the task, and the associated processes at the task level; Determining the host resources required for the HPC task according to the resource parameters corresponding to the HPC task.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Monitoring the health status of each HPC task; If a target HPC task with an abnormal health status is monitored, automatically restarting the target HPC task using the deployment components in the K8S cluster and outputting a task warning message.

7. The method according to claim 6, wherein The method further includes: Monitoring the current resource occupancy information of each K8S cluster host; If a target host whose current resource occupancy information exceeds the resource occupancy threshold is monitored, then the to-be-performed HPC tasks on the target host are migrated to a K8S cluster host whose current resource occupancy information does not exceed the resource occupancy threshold, so as to complete the to-be-performed HPC tasks.

8. A cloud-native HPC application processing system, characterized in that, It includes: A user interface, which is used to obtain application parameters input by a user, generate an application parameter request according to the application parameters, and send the parameter request to an HPC client; The application parameters include to-be-processed HPC application information and a corresponding task start rule file; An HPC client, which is used to generate an HPC client process corresponding to the to-be-processed HPC application according to the application parameter request; Generate and start corresponding HPC tasks according to each of the HPC client processes; A scheduler, which is used to determine a K8S cluster host corresponding to each HPC task and issue the HPC task; A K8S cluster host, which is used to complete the running of the corresponding HPC task; A monitor, which is used to monitor the health status of each HPC task and the current resource occupancy information of each K8S cluster host; A high-performance storage, which is used to provide a data disk required for running each HPC task; A high-performance network, which is used to provide a network environment required for running each HPC task.

9. A cloud-native HPC application processing system, characterized in that, It includes a processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

Images