Resource scheduling method, resource scheduling system and storage medium in HPC cluster

By designing a cross-platform resource scheduling method in HPC cluster, using the client and the scheduling center to uniformly schedule resources between multi-operating system clusters, the problem of users running Windows professional software on HPC cluster is solved, and resource utilization and users are improved Experience.

CN118550676BActive Publication Date: 2024-11-26TIANJIN TIANHE COMPUTER TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411031979.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-26
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

In HPC clusters, it is difficult for users to run professional Windows software, resulting in insufficient flexibility and compatibility of resource scheduling systems, complex user operation processes, and low resource utilization.

Method used

A resource scheduling method under HPC cluster is designed, and resources are uniformly scheduled between multiple operating system clusters through the client and the scheduling center, and target resources are scheduled using the resource access interface of the target operating system cluster to realize cross-platform resource scheduling.

Benefits of technology

It improves the flexibility and compatibility of the resource scheduling system, simplifies user operation processes, improves resource utilization, solves the difficulties of users in running Windows professional software on HPC clusters, and realizes unified scheduling of cross-platform resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118550676B_ABST
    Figure CN118550676B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of HPC clusters, and discloses a resource scheduling method, a resource scheduling system and a storage medium under an HPC cluster. The method comprises: a client in the resource scheduling system generates and sends an application running request to a scheduling center in the resource scheduling system in response to detecting an application startup command, and then the scheduling center determines a target operating system cluster in two clusters of the resource scheduling system according to a target operating system identifier, schedules target resources through a resource access interface provided by the scheduling center, controls the target resources to run the target application, realizes unified scheduling of resources on the two operating system clusters, achieves the purpose of cross-platform resource scheduling, improves the flexibility and compatibility of the resource scheduling system, and improves resource utilization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of HPC clusters, and in particular to a resource scheduling method, a resource scheduling system and a storage medium in an HPC cluster. Background Art

[0002] In the field of high-performance computing (HPC) clusters, scientific computing software usually runs on the Linux operating system. Linux has become the preferred operating system in the HPC environment due to its stability, efficiency and open source characteristics. Scientific computing software such as Gaussian, LAMMPS and GROMACS are all designed and optimized for the Linux platform.

[0003] However, despite Linux's dominance in the HPC cluster field, there are still some software that only provides Windows versions, such as animation rendering, financial modeling, data graphics analysis and other software. These software may contain specific functions or algorithms, and it is not easy to find alternatives in certain professional fields. This leads to challenges for users when running Windows professional software on HPC clusters, and they cannot use Windows professional software.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a resource scheduling method, a resource scheduling system and a storage medium under an HPC cluster, which solves the problem that users have difficulty running Windows professional software on an HPC cluster in the prior art, improves the flexibility and compatibility of the resource scheduling system, simplifies the user operation process, and improves resource utilization.

[0006] An embodiment of the present invention provides a resource scheduling method under an HPC cluster, which is applied to a resource scheduling system. The resource scheduling system includes a client, a first operating system cluster, a second operating system cluster, and a scheduling center. The method includes:

[0007] The client generates an application running request in response to detecting the application start command, and sends the application running request to the scheduling center, wherein the application running request includes a target operating system identifier and a target application identifier;

[0008] The scheduling center determines the target operating system cluster in the first operating system cluster and the second operating system cluster based on the target operating system identifier, schedules target resources in the target operating system cluster through a resource access interface provided by the target operating system cluster, and controls the target resources to run the target application corresponding to the target application identifier.

[0009] An embodiment of the present invention provides a resource scheduling system, the resource scheduling system comprising a client, a first operating system cluster, a second operating system cluster and a scheduling center, wherein:

[0010] The client is used to generate an application running request in response to detecting an application start command, and send the application running request to the scheduling center, wherein the application running request includes a target operating system identifier and a target application identifier;

[0011] The scheduling center is used to determine the target operating system cluster in the first operating system cluster and the second operating system cluster based on the target operating system identifier, schedule target resources in the target operating system cluster through a resource access interface provided by the target operating system cluster, and control the target resources to run the target application corresponding to the target application identifier.

[0012] An embodiment of the present invention provides an electronic device, the electronic device comprising:

[0013] Processor and memory;

[0014] The processor is used to execute the steps of the resource scheduling method under the HPC cluster described in any embodiment by calling the program or instruction stored in the memory.

[0015] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of the resource scheduling method under an HPC cluster described in any embodiment.

[0016] The embodiments of the present invention have the following technical effects:

[0017] In response to detecting an application startup command, the client in the resource scheduling system generates an application running request and sends it to the scheduling center in the resource scheduling system. Then, the scheduling center determines the target operating system cluster in the first operating system cluster and the second operating system cluster in the resource scheduling system according to the target operating system identifier, schedules the target resources in the target operating system cluster through the resource access interface provided by it, and controls the target resources to run the target application corresponding to the target application identifier, thereby realizing unified scheduling of resources on the two operating system clusters, achieving the purpose of cross-platform resource scheduling, improving the flexibility and compatibility of the resource scheduling system, improving resource utilization, and being able to quickly schedule the resources of the operating system corresponding to the user's required application, solving the problem of difficulty for users in running Windows professional software on the HPC cluster in the prior art. In addition, the user can select the corresponding operating system according to the required application, simplifying the user operation process and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of a resource scheduling method under an HPC cluster provided by an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of resource management of a resource scheduling system provided by an embodiment of the present invention;

[0021] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0023] The resource scheduling method under the HPC cluster provided by the embodiment of the present invention is applicable to the situation where resources are scheduled in the corresponding operating system cluster to run the required software of the user. Figure 1 The present invention provides a flowchart of a resource scheduling method under an HPC cluster provided by an embodiment of the present invention. The resource scheduling method under an HPC cluster is applied to a resource scheduling system, and the resource scheduling system includes a client, a first operating system cluster, a second operating system cluster and a scheduling center.

[0024] The client may be a program facing users and providing services to users; in the embodiment of the present invention, the client may specifically provide services to users in the form of a web page or in the form of an application.

[0025] The first operating system cluster may be an HPC cluster composed of nodes (which may be understood as hosts) installed with Windows operating systems; the second operating system cluster may be an HPC cluster composed of nodes (which may be understood as hosts) installed with Linux operating systems. In an embodiment of the present invention, the operating system of each node may be determined in advance, and then the first operating system cluster and the second operating system cluster may be constructed according to the operating systems of all nodes; or, the Windows operating system may be pre-installed for some nodes, so that some of the nodes constitute the first operating system cluster, and the Linux operating system may be pre-installed for another part of the nodes, so that some of the nodes constitute the second operating system cluster.

[0026] The scheduling center is connected to the client in communication. The scheduling center may be a component for scheduling corresponding resources in the two operating system clusters according to user demand applications, that is, responsible for unified scheduling of resources in the two operating system clusters.

[0027] See also Figure 1 , Figure 1 : is a flow chart of a resource scheduling method under an HPC cluster provided by an embodiment of the present invention. The resource scheduling method under the HPC cluster specifically includes:

[0028] S110 . In response to detecting the application start command, the client generates an application run request and sends the application run request to the scheduling center, wherein the application run request includes a target operating system identifier and a target application identifier.

[0029] The application startup command may be a command for starting a target application through resources in the HPC cluster. Specifically, the application startup command may be generated according to a user operation, for example, the user may directly input the application startup command through a command line.

[0030] In the embodiment of the present invention, considering that if the client runs a Linux operating system, the user usually performs operations such as application startup through a command line, in this case, the application startup command can be directly obtained according to the user's input, and if the client runs a Windows operating system, the user usually performs operations such as application startup by clicking controls, in this case, it is necessary to further generate an application startup command according to the user's control trigger operation.

[0031] In one example, the method provided by the embodiment of the present invention further includes:

[0032] The client detects the operating environment and determines the corresponding operating system; when the operating system is the first operating system, the client responds to detecting the application startup operation and converts the application startup operation into an application startup command through the node agent service.

[0033] Specifically, after being started, the client may detect the operating environment of the device used by the user to determine the operating system of the device used by the user, that is, the operating system corresponding to the client.

[0034] If the corresponding running operating system is the second operating system, ie, the Linux operating system, the client can directly respond to the command line input operation of the user to obtain the application startup command input in the command line.

[0035] If the corresponding operating system is the first operating system, i.e., the Windows operating system, the client can call the node proxy service in response to detecting an application startup operation, such as a user triggering operation on an icon control of a target application, so as to generate a corresponding application startup command through the node proxy service. The node proxy service can be a service pre-set in the client for converting a control trigger operation in the Windows operating system into a command.

[0036] Through the above example, the client can detect the running environment. When the detected running operating system is the Windows operating system, the application startup operation is converted into an application startup command through the node agent service, so that the application startup command can be generated, and then the request is sent to the scheduling center to achieve resource adaptation between the scheduling center and the client, thereby ensuring the compatibility of the client under different platform operating systems.

[0037] In an embodiment of the present invention, in response to detecting an application startup command, the client can generate an application running request according to the identifier of the target application declared in the application startup command, and send the application running request to the scheduling center. The application running request is used to request resources in the HPC cluster to run the target application.

[0038] Specifically, the application running request includes a target operating system identifier and a target application identifier, wherein the target operating system identifier may be the identifier of the target operating system declared in the application startup command, that is, the identifier entered by the user in the command line or selected by the user through the control displayed by the client.

[0039] In addition, the target operating system identifier can also be determined based on the target application identifier. For example, the target application's attribute information can be obtained based on the target application identifier, and the target operating system identifier can be determined based on the compatible system version number in the attribute information; or, the target operating system identifier can be obtained by querying the target operating system corresponding to the target application identifier based on the pre-established operating system correspondence relationship.

[0040] S120. The scheduling center determines the target operating system cluster in the first operating system cluster and the second operating system cluster based on the target operating system identifier, schedules the target resources in the target operating system cluster through the resource access interface provided by the target operating system cluster, and controls the target resources to run the target application corresponding to the target application identifier.

[0041] Specifically, after receiving the application running request, the scheduling center may determine, according to the target operating system identifier in the request, the cluster in which the target operating system is installed in the two operating system clusters as the target operating system cluster.

[0042] Furthermore, the scheduling center may call a resource access interface provided by the target operating system cluster to use a resource access service provided by the target operating system cluster, and then schedule target resources in the target operating system cluster through the resource access service.

[0043] Among them, the first operating system cluster and the second operating system cluster can integrate the resource scheduling function for each resource in the cluster into the resource access interface, and then provide the resource access interface, and the scheduling center can schedule the resources in the cluster by calling the resource access interface. In an embodiment of the present invention, the resource can specifically be a node in the cluster, such as a CPU (Central Processing Unit) host or a GPU (Graphics Processing Unit) host.

[0044] In a specific implementation, scheduling target resources in the target operating system cluster through a resource access interface provided by the target operating system cluster includes:

[0045] When the target operating system cluster is the first operating system cluster, resources are scheduled for the first operating system cluster through a resource access interface in cluster interface package software provided by the first operating system cluster to determine target resources;

[0046] Furthermore, when the target operating system cluster is the second operating system cluster, resource scheduling is performed on the second operating system cluster through a resource access interface in a resource management utility program of the second operating system cluster to determine target resources.

[0047] The resource access interface provided by the first operating system cluster may be an API in a cluster interface package software, such as a resource access API in Windows HPC Pack. The resource access interface provided by the second operating system cluster may be an API in a resource management utility, such as a resource access API in SLURM (Simple Linux Utility for Resource Management).

[0048] Specifically, if the target operating system cluster is the first operating system cluster, the scheduling center can call the resource access interface in the cluster interface package software provided by the first operating system cluster to use the resource access service provided by the first operating system cluster, thereby performing resource scheduling on the first operating system cluster through the resource access service, that is, determining the target resources in the first operating system cluster.

[0049] If the target operating system cluster is the second operating system cluster, the scheduling center can call the resource access interface in the resource management utility provided by the second operating system cluster to use the resource access service provided by the second operating system cluster, thereby scheduling resources for the second operating system cluster through the resource access service, that is, determining the target resource in the second operating system cluster. The target resource can be understood as a target node.

[0050] Through the above implementation, the resource access interface in the cluster interface package software (Windows HPC Pack) and the resource management utility (SLURM) can be used to implement resource scheduling for Windows operating system clusters and Linux operating system clusters, so that Windows resources and Linux resources can be used together by users, thereby improving resource utilization.

[0051] In the embodiment of the present invention, the scheduling center schedules the target resources in the target operating system cluster through the resource access interface provided by the target operating system cluster. Specifically, after the scheduling center calls the resource access interface, the resource access service schedules in the target operating system cluster through the built-in scheduling algorithm to determine the target resources for running the target application, such as selecting the target resources by the remaining occupancy rate of each resource. That is, the resource access interface can realize the scheduling of resources in the cluster.

[0052] Alternatively, the target resource can be scheduled in the target operating system cluster by the resource access service after the scheduling center calls the resource access interface, and the scheduling center then schedules each resource according to the relevant information of each resource and determines the target resource. That is, the resource access interface can realize the acquisition and feedback of the relevant information of the resources in the cluster.

[0053] For example, the dispatch center may select target resources based on the remaining occupancy rate of each resource, or may select target resources based on the statistical usage frequency of each resource, so as to give priority to resources with high usage frequency, that is, resources with high job processing efficiency.

[0054] In an embodiment of the present invention, considering that if target resources are selected based on statistical usage frequency, some resources may be used too little, resulting in uneven resource utilization. In order to avoid uneven resource utilization while ensuring job processing efficiency, it is also possible to sort the target resources by frequency when there are multiple resources, and select multiple target resources from them, so as to avoid the problem of uneven resource utilization caused by directly selecting resources with high usage frequency.

[0055] In one example, scheduling target resources in a target operating system cluster through a resource access interface provided by the target operating system cluster includes the following steps:

[0056] Step 11: determining resource demand information of the corresponding target application based on the target application identifier, wherein the resource demand information includes resource type, resource quantity, and resource occupancy rate;

[0057] Step 12: calling a resource access interface provided by the target operating system cluster, and determining the statistical usage frequency and remaining occupancy rate of each resource in the target operating system cluster through the resource access interface;

[0058] Step 13: From among the resources that meet the resource type in the target operating system cluster, select candidate resources whose remaining occupancy rate meets the resource occupancy rate;

[0059] Step 14: If there are multiple resources, sort the candidate resources in descending order of statistical usage frequency, take the first candidate resource in the sorting result as the resource to be judged, and determine the difference rate between the statistical usage frequency of the resource to be judged and other candidate resources in the sorting result;

[0060] Step 15: take the resource to be judged as the target resource, and remove the target resource and the candidate resources whose difference rate is less than the preset difference rate threshold from the sorting result, and return to the step of taking the first candidate resource in the sorting result as the resource to be judged, until the number of determined target resources meets the number of resources.

[0061] In step 11, the dispatch center can determine the resource demand information of the target application, such as the resource type, resource quantity, and resource occupancy rate, according to the target application identifier. The resource type can be the host type corresponding to the resource, such as CPU or GPU; the resource quantity can be the number of resources expected to run the target application; and the resource occupancy rate can be the occupancy rate of the resource caused by the operation of the target application.

[0062] Furthermore, in step 12, the scheduling center may call the resource access interface provided by the target operating system cluster to determine the statistical usage frequency and remaining occupancy of each resource in the target operating system cluster through the resource access service, wherein the statistical usage frequency may be the total number of historical usage of the resource, and the remaining occupancy may be the occupancy of the resource that is not currently in use.

[0063] Furthermore, in step 13, the scheduling center may first screen out resources that meet the resource type in the target operating system cluster, and then screen out resources whose remaining occupancy rate is not less than the resource occupancy rate as candidate resources.

[0064] Furthermore, in step 14, if the number of resources required by the target application is multiple, the scheduling center can sort the candidate resources in descending order of statistical usage frequency, and then use the first candidate resource as the resource to be judged, and calculate the difference ratio between the statistical usage frequency of the resource to be judged and other candidate resources. The difference ratio can be obtained by first calculating the difference between the statistical usage frequency of the resource to be judged and other candidate resources, and taking the ratio between the difference and the statistical usage frequency of the resource to be judged as the difference ratio.

[0065] Furthermore, in step 15, the dispatch center can use the resource to be judged as the target resource, and remove the target resource and the candidate resource whose phase difference rate is less than the preset phase difference rate threshold from the sorting result, and then return to execute step 14 to use the first candidate resource in the sorting result as the resource to be judged, so as to re-determine the phase difference rate between the statistical usage frequency of the resource to be judged and other candidate resources, until the number of determined target resources meets the number of resources. The preset phase difference rate threshold can be calibrated according to the number of resources in the cluster, or the dispatch center can continuously detect the performance of the cluster and optimize the preset phase difference rate threshold according to feedback.

[0066] Through the above steps 11 to 15, we can start from the candidate resources with high statistical usage frequency, first screen out the candidate resources with the highest statistical usage frequency as the target resources to ensure the operating efficiency of the target application, and then screen out the candidate resources with a large difference in statistical usage frequency as the target resources. This can ensure that the frequencies of the multiple target resources determined have a large difference, avoiding the problem of unbalanced resource utilization caused by only using resources with a higher frequency. At the same time, some resources with a higher frequency can be left for the cluster for other users to use, ensuring the operating efficiency of all users, thereby improving the user experience.

[0067] Of course, if there is only one resource, they can be directly sorted in descending order according to the statistical usage frequency, and the first candidate resource is used as the target resource to ensure the user's work efficiency.

[0068] In one example, after controlling the target resource to run the target application corresponding to the target application identifier, the method provided by the embodiment of the present invention further includes:

[0069] In response to detecting a job execution command, the client generates a job execution request and sends the job execution request to the scheduling center; based on the job execution request, the scheduling center controls the target application running on the target resource to execute the corresponding target job, and during the execution of the target job, the performance status of the target resource and the execution status of the target job are detected, and in response to detecting an abnormal performance status or an abnormal execution status, an alarm information is fed back to the client.

[0070] Specifically, after the target resource runs the target application, the user can perform related tasks through the running target application, such as generating a video file through the running animation processing application, or processing a data file through the running data analysis application.

[0071] The user may enter a job execution command in the command line, or trigger a job execution control in the client, so that the client generates a job execution command, and then the client generates a job execution request based on the job execution command and sends it to the scheduling center.

[0072] Furthermore, the scheduling center may control the target application running on the target resource to execute the corresponding target job according to the job execution request. For example, the scheduling center may forward the job execution request to the target application through the target resource, so that the target application executes the target job.

[0073] During the process of the target application executing the target job, the scheduling center can detect the performance status of the target resources and the execution status of the target job. The performance status can reflect whether the target resources are running normally and whether the performance is abnormal (such as decreased), and the execution status can reflect whether the target job is executed normally.

[0074] Specifically, if the scheduling center detects that the performance status of the target resource is abnormal or the execution status of the target job is abnormal, the scheduling center can generate an alarm message and send it to the client to implement a fault alarm.

[0075] Through the above examples, the dispatch center can detect jobs and resources and issue fault alarms, which can help users promptly understand the existence of abnormalities in jobs or resources and facilitate timely processing.

[0076] In an embodiment of the present invention, considering that resources in the first operating system cluster or the second operating system cluster may use data generated by resources in the other cluster, for example, a data drawing application running in a Windows operating system cluster needs to use data statistics files generated by a data analysis application in a Linux operating system cluster. In order to facilitate data acquisition by resources in the two clusters, an embodiment of the present invention can also set a storage pool shared by the two clusters in the resource scheduling system.

[0077] In some implementations, the resource scheduling system further includes a storage pool, which is used to store data written by resources in the first operating system cluster and the second operating system cluster. After controlling the target resource to run the target application corresponding to the target application identifier, the method provided by the embodiment of the present invention further includes:

[0078] The target resource responds to detecting a data transmission request of the target application, and when the corresponding operating system cluster is the second operating system cluster, writes the to-be-recorded data of the target application to the storage pool through the information block protocol service provided by the second operating system cluster, or reads the required data of the target application from the storage pool;

[0079] And, in the case that the corresponding operating system cluster is the first operating system cluster, the to-be-recorded data of the target application is written into the storage pool, or the required data of the target application is read from the storage pool.

[0080] The storage pool may be a Lustre high-performance file system. The second operating system cluster may provide a block protocol service, which may be a Samba service, which may be used to implement an SMB (Server Message Block) protocol on different operating systems.

[0081] Specifically, if the target resource detects a data transmission request of the target application, if it is a second operating system cluster, the data to be recorded can be written to the storage pool or the required data of the target application can be read from the storage pool through the information block protocol service provided by the cluster.

[0082] If the target resource detects a data transmission request of the target application, if it is the first operating system, the data to be recorded can be directly written into the storage pool or the required data of the target application can be read from the storage pool.

[0083] Through the above implementation, unified data storage and access across platforms can be achieved. Compared with the method of establishing file systems for two clusters respectively, and then obtaining data from the file systems and then converting them, this implementation can facilitate the resources in the two clusters to read and write data at any time, and then facilitate applications between different operating systems to use data with each other, simplifying user operations and improving user convenience.

[0084] In one example, the method provided by the embodiment of the present invention further includes:

[0085] In response to receiving the resource configuration file, the scheduling center determines a configuration cluster in the first operating system cluster and the second operating system cluster based on the resource configuration file, and updates configuration information of at least one resource in the configuration cluster according to the resource configuration file.

[0086] The resource configuration file may be a file for configuring node information corresponding to a resource. The management platform for administrators or the scheduling client corresponding to the scheduling center may provide a resource configuration interface for the administrator, and the administrator may issue the resource configuration file in the resource configuration interface.

[0087] Specifically, in response to receiving the resource configuration file, the scheduling center can run the batch resource configuration code, and determine the configuration cluster in the first operating system cluster and the second operating system cluster according to the cluster specified in the resource configuration file through the running batch resource configuration code, and further, according to the resource specified in the resource configuration file, update the configuration information of at least one resource in the configuration cluster. Among them, the batch resource configuration code can be a self-developed batch management tool developed using Python, which is used to achieve efficient batch operation and management of cluster resources.

[0088] Through the above example, batch update of cluster resource configuration can be realized, which is convenient for administrators to operate without sending the configuration information of each resource separately, thereby improving the administrator's operation convenience and the update efficiency of resource configuration.

[0089] In one example, the method provided by the embodiment of the present invention further includes:

[0090] In response to receiving the operating system change request, the scheduling center determines the resources to be changed and the corresponding expected operating system based on the operating system change request, installs the expected operating system in the resources to be changed, removes the resources to be changed from the corresponding operating system cluster, and adds the resources to be changed to the operating system cluster corresponding to the expected operating system.

[0091] Among them, the management platform for administrators or the scheduling client corresponding to the scheduling center can provide the administrator with an operating system change interface, and the administrator can change the operating system of one or more resources in the operating system change interface to generate an operating system change request.

[0092] Specifically, in response to receiving the operating system change request, the scheduling center may determine the resources to be changed according to the resources specified in the operating system change request, and determine the expected operating system corresponding to the resources to be changed according to the operating system specified in the operating system change request.

[0093] Furthermore, the dispatch center can install the desired operating system in the resource to be changed. After the installation of the resource to be changed is completed, the resource to be changed will be removed from the corresponding operating system cluster and added to the operating system cluster corresponding to the desired operating system. For example, after reinstalling the Linux operating system, the resource in the Windows operating system cluster needs to be deleted from the Windows operating system cluster and added to the Linux operating system cluster.

[0094] Through the above examples, the operating system of each resource can be updated. When the resource usage in a certain cluster is too high or the resources are insufficient, the resources in another cluster can be converted to this cluster to ensure that both clusters can provide resources for all users, reduce the burden on the cluster, and avoid excessive pressure on the cluster.

[0095] The present invention has the following technical effects: a client in a resource scheduling system generates an application running request in response to detecting an application start command and sends the request to a scheduling center in the resource scheduling system, and then the scheduling center determines a target operating system cluster in a first operating system cluster and a second operating system cluster in the resource scheduling system according to a target operating system identifier, schedules target resources in the target operating system cluster through a resource access interface provided by the scheduling center, and controls the target resources to run a target application corresponding to a target application identifier, thereby realizing unified scheduling of resources on the two operating system clusters, achieving the purpose of cross-platform resource scheduling, improving the flexibility and compatibility of the resource scheduling system, improving resource utilization, being able to quickly schedule the resources of the operating system corresponding to the user's required application, solving the problem of difficulty for users in running Windows professional software on an HPC cluster in the prior art, and allowing users to select a corresponding operating system according to the required application, thereby simplifying the user's operation process and improving the user experience.

[0096] The embodiment of the present invention further provides a resource scheduling system, the resource scheduling system comprising a client, a first operating system cluster, a second operating system cluster and a scheduling center, wherein:

[0097] The client is used to generate an application running request in response to detecting an application start command, and send the application running request to the scheduling center, wherein the application running request includes a target operating system identifier and a target application identifier;

[0098] The scheduling center is used to determine the target operating system cluster in the first operating system cluster and the second operating system cluster based on the target operating system identifier, schedule target resources in the target operating system cluster through a resource access interface provided by the target operating system cluster, and control the target resources to run the target application corresponding to the target application identifier.

[0099] Optionally, the client is also used to detect the operating environment and determine the corresponding operating operating system; when the operating operating system is the first operating system, in response to detecting an application startup operation, the application startup operation is converted into an application startup command through a node agent service.

[0100] Optionally, the scheduling center is also used to, when the target operating system cluster is a first operating system cluster, perform resource scheduling on the first operating system cluster and determine target resources through a resource access interface in a cluster interface package software provided by the first operating system cluster; and, when the target operating system cluster is a second operating system cluster, perform resource scheduling on the second operating system cluster and determine target resources through a resource access interface in a resource management utility of the second operating system cluster.

[0101] Optionally, the dispatch center is also used for:

[0102] Determine resource demand information of the corresponding target application based on the target application identifier, wherein the resource demand information includes resource type, resource quantity and resource occupancy rate; call the resource access interface provided by the target operating system cluster, and determine the statistical usage frequency and remaining occupancy rate of each resource in the target operating system cluster through the resource access interface;

[0103] Screening out candidate resources whose remaining occupancy rate satisfies the resource occupancy rate from among the resources that meet the resource type in the target operating system cluster;

[0104] If there are multiple resources, the candidate resources are sorted in descending order of statistical usage frequency, the first candidate resource in the sorting result is used as the resource to be judged, and the difference rate between the statistical usage frequency of the resource to be judged and other candidate resources in the sorting result is determined;

[0105] The resource to be judged is taken as the target resource, and the target resource and the candidate resources whose difference rate is less than the preset difference rate threshold are eliminated from the sorting result, and the step of taking the first candidate resource in the sorting result as the resource to be judged is returned until the number of determined target resources meets the number of resources.

[0106] Optionally, the client is further configured to generate a job execution request in response to detecting the job execution command, and send the job execution request to the scheduling center;

[0107] The scheduling center is also used to control the target application running on the target resource to execute the corresponding target job based on the job execution request, and during the execution of the target job, detect the performance status of the target resource and the execution status of the target job, and in response to detecting the performance status abnormality or the execution status abnormality, feedback alarm information to the client.

[0108] Optionally, the resource scheduling system also includes a storage pool, which is used to store data written by resources in the first operating system cluster and the second operating system cluster; the target resource is used to respond to the detection of a data transmission request of the target application, and when the corresponding operating system cluster is the second operating system cluster, write the target application's to-be-recorded data to the storage pool through the information block protocol service provided by the second operating system cluster, or read the target application's required data from the storage pool; and, when the corresponding operating system cluster is the first operating system cluster, write the target application's to-be-recorded data to the storage pool, or read the target application's required data from the storage pool.

[0109] Optionally, the scheduling center is further used to determine a configuration cluster in the first operating system cluster and the second operating system cluster based on the resource configuration file in response to receiving the resource configuration file, and update configuration information of at least one resource in the configuration cluster according to the resource configuration file.

[0110] Optionally, the scheduling center is also used to respond to receiving an operating system change request, determine the resources to be changed and the corresponding expected operating system based on the operating system change request, install the expected operating system in the resources to be changed, remove the resources to be changed from the corresponding operating system cluster, and add the resources to be changed to the operating system cluster corresponding to the expected operating system.

[0111] Figure 2 FIG. 1 is a schematic diagram of resource management of a resource scheduling system provided by an embodiment of the present invention. Figure 2As shown, the scheduling center in the resource scheduling system includes a job manager and a resource manager.

[0112] Among them, users can submit jobs or view jobs through the client, providing users with a friendly job submission and monitoring interface to simplify user operations.

[0113] The job manager can perform cross-platform scheduling based on submitted jobs (which can be understood as application running requests), that is, schedule Windows or Linux resources through the adapter interface provided by the cluster, where Windows resources and Linux resources can be connected to the storage pool.

[0114] Moreover, the resource manager can also manage the resources in the two clusters, perform system configuration on the resources through batch management tools, or install software or operating system on the resources. In addition, the resource manager can also obtain the job status and monitor the performance of the resources, as well as monitor the execution status of the job through log statistics, and can issue fault alarms in the event of anomalies, realizing the full process management from job submission to completion, including job scheduling, execution, monitoring and logging, as well as continuously monitoring the cluster performance and optimizing the scheduling strategy based on feedback.

[0115] In addition, the resource manager can also manage user resources and implement unified user authentication through AD (Active Directory) domain or LDAP (Lightweight Directory Access Protocol).

[0116] In addition, the resource manager can monitor cluster performance, optimize scheduling strategies based on monitoring results, and improve resource utilization. Security enhancements can also be designed in the scheduling center to ensure the security of cluster operations, prevent data leakage and unauthorized access, and design fault-tolerant mechanisms to ensure that the cluster can still operate normally when some nodes fail.

[0117] The resource scheduling system provided by the embodiment of the present invention does not rely on a specific operating system and can run seamlessly on Windows and Linux platforms, thereby improving the flexibility and compatibility of the system; through a unified scheduling center, cross-platform resource management and scheduling are realized, the resource allocation process is simplified, and resource utilization is improved. In addition, users can select the corresponding platform according to application requirements, simplifying the operation process and improving user experience.

[0118] In addition, the scheduling center uses the existing HPC Pack and SLURM APIs to reduce development workload while ensuring compatibility with existing HPC clusters. Unified user authentication through AD or LDAP simplifies user management and improves system security. In addition, combined with Lustre and Samba, it provides high-performance and cross-platform storage access, solving the problem of data sharing between different platforms.

[0119] In addition, the scheduling center improves the efficiency of cluster management and automated operations through batch management tools, provides full-process management from job submission to completion, and makes job monitoring and scheduling more efficient and transparent.

[0120] Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, the electronic device 400 includes one or more processors 401 and a memory 402 .

[0121] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0122] The memory 402 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may run the program instructions to implement the resource scheduling method under the HPC cluster of any embodiment of the present invention described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0123] In one example, the electronic device 400 may further include: an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 403 may include, for example, a keyboard, a mouse, etc. The output device 404 may output various information to the outside, including early warning prompt information, braking force, etc. The output device 404 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0124] Of course, to simplify, Figure 3Only some of the components related to the present invention in the electronic device 400 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 400 may also include any other appropriate components.

[0125] In addition to the above methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the resource scheduling method under the HPC cluster provided by any embodiment of the present invention.

[0126] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0127] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the resource scheduling method under an HPC cluster provided by any embodiment of the present invention.

[0128] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0129] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0130] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A resource scheduling method in an HPC cluster, characterized in that: Applied to a resource scheduling system, the resource scheduling system includes a client, a first operating system cluster, a second operating system cluster and a scheduling center, the method includes: The client generates an application running request in response to detecting the application start command, and sends the application running request to the scheduling center, wherein the application running request includes a target operating system identifier and a target application identifier; The scheduling center determines a target operating system cluster in the first operating system cluster and the second operating system cluster based on the target operating system identifier, schedules target resources in the target operating system cluster through a resource access interface provided by the target operating system cluster, and controls the target resources to run a target application corresponding to the target application identifier; Among them, when there are multiple target resources, they are sorted according to the statistical usage frequency, and multiple target resources are selected from them, starting with the candidate resources with high statistical usage frequency. The candidate resource with the highest statistical usage frequency is first screened out as one of the target resources, and then the candidate resources whose difference rate is not less than the preset difference rate threshold are screened out as the remaining target resources.

2. The method according to claim 1, characterized in that The method further comprises: The client detects the operating environment and determines the corresponding operating system; In the case that the running operating system is the first operating system, in response to detecting the application startup operation, the client converts the application startup operation into an application startup command through a node agent service.

3. The method according to claim 1, characterized in that Scheduling target resources in the target operating system cluster through a resource access interface provided by the target operating system cluster includes: In the case where the target operating system cluster is a first operating system cluster, performing resource scheduling on the first operating system cluster through a resource access interface in a cluster interface package software provided by the first operating system cluster to determine target resources; and In the case that the target operating system cluster is a second operating system cluster, resource scheduling is performed on the second operating system cluster through a resource access interface in a resource management utility program of the second operating system cluster to determine target resources.

4. The method according to claim 1, characterized in that: Scheduling target resources in the target operating system cluster through a resource access interface provided by the target operating system cluster includes: Determining resource requirement information of the corresponding target application based on the target application identifier, wherein the resource requirement information includes resource type, resource quantity, and resource occupancy rate; Calling a resource access interface provided by the target operating system cluster, and determining the statistical usage frequency and remaining occupancy rate of each resource in the target operating system cluster through the resource access interface; Screening out candidate resources whose remaining occupancy rate satisfies the resource occupancy rate from among the resources that meet the resource type in the target operating system cluster; If there are multiple resources, the candidate resources are sorted in descending order of statistical usage frequency, the first candidate resource in the sorting result is used as the resource to be judged, and the difference rate between the statistical usage frequency of the resource to be judged and other candidate resources in the sorting result is determined; The resource to be judged is taken as the target resource, and the target resource and the candidate resources whose difference rate is less than the preset difference rate threshold are eliminated from the sorting result, and the step of taking the first candidate resource in the sorting result as the resource to be judged is returned until the number of determined target resources meets the number of resources.

5. The method according to claim 1, characterized in that After controlling the target resource to run the target application corresponding to the target application identifier, the method further includes: The client generates a job execution request in response to detecting the job execution command, and sends the job execution request to the scheduling center; Based on the job execution request, the scheduling center controls the target application running on the target resource to execute the corresponding target job, and during the execution of the target job, detects the performance status of the target resource and the execution status of the target job, and in response to detecting the performance status abnormality or the execution status abnormality, feeds back alarm information to the client.

6. The method according to claim 1, characterized in that The resource scheduling system further includes a storage pool, the storage pool being used to store data written by resources in the first operating system cluster and the second operating system cluster. After controlling the target resource to run the target application corresponding to the target application identifier, the method further includes: In response to detecting a data transmission request of the target application, the target resource writes the to-be-recorded data of the target application to the storage pool through an information block protocol service provided by the second operating system cluster, or reads the required data of the target application from the storage pool, in a case where the corresponding operating system cluster is the second operating system cluster; and In a case where the corresponding operating system cluster is the first operating system cluster, the to-be-recorded data of the target application is written to the storage pool, or the required data of the target application is read from the storage pool.

7. The method according to claim 1, characterized in that The method further comprises: In response to receiving the resource configuration file, the scheduling center determines a configuration cluster in the first operating system cluster and the second operating system cluster based on the resource configuration file, and updates configuration information of at least one resource in the configuration cluster according to the resource configuration file.

8. The method according to claim 1, characterized in that The method further comprises: In response to receiving an operating system change request, the scheduling center determines the resources to be changed and the corresponding expected operating system based on the operating system change request, installs the expected operating system in the resources to be changed, removes the resources to be changed from the corresponding operating system cluster, and adds the resources to be changed to the operating system cluster corresponding to the expected operating system.

9. A resource scheduling system, characterized in that: The resource scheduling system includes a client, a first operating system cluster, a second operating system cluster and a scheduling center, wherein: The client is used to generate an application running request in response to detecting an application start command, and send the application running request to the scheduling center, wherein the application running request includes a target operating system identifier and a target application identifier; The scheduling center is used to determine a target operating system cluster in the first operating system cluster and the second operating system cluster based on the target operating system identifier, schedule target resources in the target operating system cluster through a resource access interface provided by the target operating system cluster, and control the target resources to run the target application corresponding to the target application identifier; Among them, when there are multiple target resources, they are sorted according to the statistical usage frequency, and multiple target resources are selected from them, starting with the candidate resources with high statistical usage frequency. The candidate resource with the highest statistical usage frequency is first screened out as one of the target resources, and then the candidate resources whose difference rate is not less than the preset difference rate threshold are screened out as the remaining target resources.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the resource scheduling method under the HPC cluster according to any one of claims 1 to 8.