Cluster computing power resource scheduling method and device

Abstract

The invention relates to a cluster computing power resource scheduling method and device, and belongs to the technical field of cluster computing power resource scheduling, the method comprises the following steps: a unified computing power scheduler receives a computing task submitted by a user, and caches the computing task in a computing task scheduling queue; respectively issuing the calculation tasks to a Kubernetes scheduler and an HPC scheduler according to the types of the calculation tasks; the Kubernetes scheduler and the HPC scheduler issue the calculation task to a proper calculation node according to the resource use condition and the scheduling strategy of the cluster calculation node; sending a scheduling result of the scheduler to the user; wherein the unified computing power scheduler is deployed on the Kubernetes scheduler and the HPC scheduler. According to the method, computing clusters do not need to be built for the two types of applications respectively, the computing power resource utilization rate is increased, no intrusive modification is conducted on the dispatchers deployed on the clusters at the same time, and the dispatchers can be independently upgraded and evolved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cluster management, and in particular, relates to a cluster computing resource scheduling method and device. Background Art

[0002] Computing resources are the key force driving scientific and technological innovation. Like the "digital engine" of modern society, they play a vital role in promoting the development of all aspects of society and economy. In many scientific research fields, such as astronomy, meteorology, biology, etc., a large number of complex calculations and simulations require strong computing support. Without sufficient computing power, many cutting-edge research will be difficult to carry out, and new discoveries and breakthroughs will also be limited. At the same time, with the rapid application of artificial intelligence technology in traditional industries and the rapid upgrading of industrial intelligence, the training and reasoning of artificial intelligence models are also highly dependent on the support of massive computing resources.

[0003] However, in the current construction of computing clusters, computing clusters for high-performance computing usually use HPC schedulers such as Slurm and PBS to manage clusters and schedule computing resources, while computing clusters for artificial intelligence computing usually use Kubernetes to manage clusters and schedule computing resources. Therefore, it is necessary to build computing clusters for different types of applications, resulting in the two types of clusters usually being deployed and run independently.

[0004] When the computing power resources of the high-performance computing cluster are insufficient, even if the artificial intelligence computing cluster has sufficient computing power resources, it cannot provide computing power services for high-performance computing applications. Vice versa, when the computing power resources of the artificial intelligence computing cluster are insufficient, even if the high-star computing cluster has sufficient computing power resources, it cannot provide computing power services for artificial intelligence computing applications. Therefore, the computing power resources of the two types of computing power clusters cannot be shared, resulting in low computing power resource utilization. In particular, expensive technical resources such as GPUs cannot be shared across clusters, which greatly increases the cost of computing power service providers. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the invention is to provide a cluster computing resource scheduling method and device, which does not require the construction of separate computing clusters for two types of applications, thereby improving the utilization of computing resources and making no intrusive modifications to the schedulers deployed simultaneously on the cluster. The schedulers can be upgraded and evolved independently.

[0006] In a first aspect of the present invention, a cluster computing resource scheduling method is proposed, which is applied to a system including: a Kubernetes scheduler, an HPC scheduler, and a unified computing scheduler, including:

[0007] The unified computing power scheduler receives computing tasks submitted by users and caches the computing tasks in the computing task scheduling queue;

[0008] Dispatches computing tasks to the Kubernetes scheduler and HPC scheduler respectively according to their types and the order of computing tasks in the computing task scheduling queue;

[0009] The Kubernetes scheduler and HPC scheduler dispatch computing tasks to appropriate computing nodes based on the resource usage and scheduling policies of the cluster computing nodes;

[0010] Send the scheduling results of the Kubernetes scheduler and HPC scheduler to the user;

[0011] Among them, the unified computing power scheduler is deployed on the Kubernetes scheduler and HPC scheduler.

[0012] Furthermore, the above cluster computing resource scheduling method further includes:

[0013] Deploy resource monitoring agents on all computing nodes in the cluster;

[0014] The resource monitoring agent receives commands from the unified computing scheduler and sends heartbeats to the Kubernetes scheduler and HPC scheduler.

[0015] The Kubernetes scheduler and HPC scheduler update the resource usage of cluster computing nodes based on heartbeats.

[0016] Furthermore, the above cluster computing resource scheduling method further includes:

[0017] Send the first, continuous, and same-type computing tasks in the computing task scheduling queue to the corresponding Kubernetes scheduler;

[0018] After each computing task is scheduled, the unified computing scheduler sends the scheduling result of the Kubernetes scheduler to the user;

[0019] After the scheduling of the first, continuous, and same type of computing tasks is completed, the unified computing power scheduler counts the computing node list where the first, continuous, and same type of computing tasks are finally deployed;

[0020] The resource monitoring agent deployed on the computing nodes in the node list receives the command from the unified computing scheduler to collect the resource usage of the computing nodes and send a heartbeat to the HPC scheduler;

[0021] The HPC scheduler updates the resource usage statistics of the cluster computing nodes based on the heartbeat;

[0022] The HPC scheduler schedules computing tasks based on the updated resource usage statistics of the cluster computing nodes.

[0023] Furthermore, the above cluster computing resource scheduling method further includes:

[0024] Send the first, continuous, and same-type computing tasks in the computing task scheduling queue to the corresponding HPC scheduler;

[0025] After each computing task is scheduled, the unified computing power scheduler sends the scheduling result of the HPC scheduler to the user;

[0026] After the scheduling of the first, continuous, and same type of computing tasks is completed, the unified computing power scheduler counts the computing node list where the first, continuous, and same type of computing tasks are finally deployed;

[0027] The resource monitoring agent deployed on the computing node in the node list receives the command from the unified computing scheduler to collect the resource usage of the computing node and send a heartbeat to the Kubernetes scheduler;

[0028] The Kubernetes scheduler updates the resource usage statistics of the cluster computing nodes based on the heartbeat;

[0029] The Kubernetes scheduler schedules computing tasks based on the updated resource usage statistics of the cluster computing nodes.

[0030] Furthermore, in the above-mentioned cluster computing power resource scheduling method, the unified computing power scheduler provides a command line tool, and the program name of the command line tool is the same as the client tool of Kubernetes and HPC scheduler.

[0031] Furthermore, in the above cluster computing resource scheduling method, the computing tasks are sent to the Kubernetes scheduler and the HPC scheduler respectively according to the types of computing tasks, and the unified computing scheduler sends the computing tasks respectively through the standard interfaces of the Kubernetes scheduler and the HPC scheduler according to the types of computing tasks;

[0032] The scheduling results of the Kubernetes scheduler and the HPC scheduler are sent to the user by the unified computing power scheduler through the standard interfaces of the Kubernetes scheduler and the HPC scheduler.

[0033] Furthermore, in the above cluster computing resource scheduling method, the list of computing nodes to which the computing task is finally deployed includes:

[0034] Statistics on the scheduling results of each computing task;

[0035] According to the type of computing tasks and the scheduling results of each scheduling task, the relevant Pods are counted and a Pod list is obtained;

[0036] Count the compute nodes where each Pod in the Pod list is located to form a compute node set;

[0037] A list of computing nodes is determined based on the computing node set.

[0038] In a second aspect of the present invention, a cluster computing resource scheduling device is further proposed, which is applied to a system including: a Kubernetes scheduler, an HPC scheduler, and a unified computing scheduler, including:

[0039] Receiving module and storage module: used for the unified computing power scheduler to receive computing tasks submitted by users and cache the computing tasks in the computing task scheduling queue;

[0040] The first delivery module is used to deliver computing tasks to the Kubernetes scheduler and the HPC scheduler respectively according to the type of computing tasks and the order of computing tasks in the computing task scheduling queue;

[0041] The second delivery module: used by the Kubernetes scheduler and HPC scheduler to deliver computing tasks to appropriate computing nodes based on the resource usage and scheduling policies of the cluster computing nodes;

[0042] Sending module: used to send the scheduling results of Kubernetes scheduler and HPC scheduler to users;

[0043] Among them, the unified computing power scheduler is deployed on the Kubernetes scheduler and HPC scheduler.

[0044] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0045] The processor is used to execute any one of the above cluster computing power resource scheduling methods by calling the program or instruction stored in the memory.

[0046] In a fourth aspect of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute any one of the above cluster computing resource scheduling methods.

[0047] The beneficial effects of the present invention are as follows: by simultaneously deploying schedulers for high-performance computing scenarios such as Kubernetes scheduler, HPC scheduler and Slurm scheduler in the same computing cluster, the present invention enables applications for artificial intelligence and high-performance computing to be deployed and run in the same computing cluster, thereby eliminating the need to build computing clusters separately for the two types of applications, thereby improving the utilization of computing resources; at the same time, the new system does not require any modification to the Kubernetes and HPC schedulers, and is a non-invasive modification. Subsequent schedulers such as Kubernetes and HPC can be independently upgraded as their respective research and development progresses; in addition, the present invention does not change the user's usage method, so the new scheduling system will not increase the user's learning cost and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0049] Figure 1 A cluster computing resource scheduling method provided by an embodiment of the present invention Figure 1 ;

[0050] Figure 2 A cluster computing resource scheduling method provided by an embodiment of the present invention Figure 2 ;

[0051] Figure 3 A cluster computing resource scheduling method provided by an embodiment of the present invention Figure 3 ;

[0052] Figure 4 A cluster computing resource scheduling method provided by an embodiment of the present invention Figure 4 ;

[0053] Figure 5 A diagram of a computing node list method for final deployment of a computing task provided by an embodiment of the present invention;

[0054] Figure 6 A diagram of a cluster computing resource scheduling device provided by an embodiment of the present invention;

[0055] Figure 7 A schematic block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0057] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0058] In the description of the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0060] The present invention proposes a cluster computing resource scheduling method, device, electronic device and storage medium, which eliminates the need to build computing clusters separately for two types of applications, improves the computing resource utilization rate, and does not make any invasive modifications to the schedulers deployed simultaneously on the cluster, so the schedulers can be upgraded and evolved independently.

[0061] Method Embodiment

[0062] Figure 1 A cluster computing resource scheduling method provided by an embodiment of the present invention Figure 1 .

[0063] In a first aspect of the present invention, a cluster computing resource scheduling method is proposed, which is applied to a system including: a Kubernetes scheduler, an HPC scheduler, and a unified computing scheduler, and combined with Figure 1 , including four steps from S1 to S4:

[0064] S1: The unified computing power scheduler receives computing tasks submitted by users and caches the computing tasks in the computing task scheduling queue.

[0065] Specifically, in an embodiment of the present invention, a unified computing power scheduler is deployed on top of a Kubernetes scheduler and an HPC scheduler, and uniformly receives computing tasks submitted by various types of users. The unified computing power scheduler caches the users' computing tasks in an internal computing task scheduling queue.

[0066] S2: Send computing tasks to the Kubernetes scheduler and HPC scheduler respectively according to the types of computing tasks and the order of computing tasks in the computing task scheduling queue.

[0067] Specifically, in an embodiment of the present invention, if the first computing task in the computing task scheduling queue is a Kubernetes task, it is sent to the Kubernetes scheduler through the Kubernetes scheduler standard interface and scheduled by the Kubernetes scheduler. If the second computing task in the computing task scheduling queue is an HPC task, it is sent to the HPC scheduler through the HPC scheduler standard interface and scheduled by the HPC scheduler.

[0068] S3: The Kubernetes scheduler and HPC scheduler send computing tasks to appropriate computing nodes based on the resource usage and scheduling policies of the cluster computing nodes.

[0069] Specifically, in an embodiment of the present invention, Kubernetes and the HPC scheduler independently manage all computing nodes in the cluster, update the resource usage of the cluster computing nodes according to the computing node heartbeat messages received by each, receive and schedule computing task requests from applications, and send computing tasks to appropriate computing nodes according to their respective scheduling strategies.

[0070] S4: Send the scheduling results of the Kubernetes scheduler and HPC scheduler to the user.

[0071] Specifically, in the embodiment of the present invention, the scheduling results of the Kubernetes scheduler and the HPC scheduler are sent to the user, so that the user can grasp the scheduling process and results in a timely manner, thereby improving the user experience.

[0072] In some embodiments, the computing task may also be Slurm, etc., and the scheduler may also be a Slurm scheduler.

[0073] Figure 2 A cluster computing resource scheduling method provided by an embodiment of the present invention Figure 2 .

[0074] Furthermore, the above cluster computing resource scheduling method is combined with Figure 2 , and also includes three steps S21 to S23:

[0075] S21: Deploy resource monitoring agents on all computing nodes in the cluster;

[0076] S22: The resource monitoring agent receives the command from the unified computing scheduler and sends a heartbeat to the Kubernetes scheduler and the HPC scheduler.

[0077] S23: The Kubernetes scheduler and HPC scheduler update the resource usage of the cluster computing nodes based on the heartbeat.

[0078] Specifically, in an embodiment of the present invention, a resource monitoring agent is deployed on all computing nodes of the cluster. Under the command of the unified computing power scheduler, the agent of Kubernetes and HPC scheduler on the computing node is simulated to send heartbeats to the Kubernetes scheduler and HPC scheduler, and the resource usage statistics of the scheduler are updated in time.

[0079] Figure 3 A cluster computing resource scheduling method provided by an embodiment of the present invention Figure 3 .

[0080] Furthermore, the above cluster computing resource scheduling method is combined with Figure 3 , also includes:

[0081] S31: Send the first, continuous, and same-type computing tasks in the computing task scheduling queue to the corresponding Kubernetes scheduler.

[0082] Specifically, in an embodiment of the present invention, if the continuous computing tasks at the head of the computing task scheduling queue in the unified computing power scheduler are Kubernetes, the unified computing power scheduler sends the continuous computing tasks Kubernetes at the head of the computing task scheduling queue to the Kubernetes scheduler for scheduling by the Kubernetes scheduler. For example, if the first three computing tasks in the queue are all Kubernetes tasks, and the fourth computing task is an HPC task, the unified computing power scheduler takes out the first three scheduling tasks from the queue at one time and sends them to the corresponding Kubernetes scheduling.

[0083] S32: After each computing task is scheduled, the unified computing scheduler sends the scheduling result of the Kubernetes scheduler to the user.

[0084] Specifically, exemplarily, after each Kubernetes scheduling of the first three computing tasks is completed, the unified computing power scheduler forwards the scheduling results of each Kubernetes computing task to the user.

[0085] S33: After the scheduling of the first, continuous, and same type of computing tasks is completed, the unified computing power scheduler counts the list of computing nodes where the first, continuous, and same type of computing tasks are finally deployed;

[0086] Specifically and exemplarily, after the first three computing tasks are scheduled by Kubernetes, the unified computing power scheduler will count the list of computing nodes deployed by the first three computing tasks Kubernetes scheduling tasks. The specific method is described in detail below.

[0087] S34: The resource monitoring agent deployed on the computing node in the node list receives the command of the unified computing power scheduler to collect the resource usage of the computing node and sends a heartbeat to the HPC scheduler.

[0088] Specifically, in an embodiment of the present invention, the unified computing power scheduler commands the unified resource monitoring agent deployed on all computing nodes in the computing node list to collect resource occupancy information of each computing node, simulate the heartbeat message format of HPC, and send a heartbeat to the HPC scheduler.

[0089] S35: The HPC scheduler updates the resource usage statistics of the cluster computing nodes according to the heartbeat.

[0090] Specifically, in the embodiment of the present invention, the HPC scheduler updates the resource usage statistics of the cluster computing nodes according to the heartbeat, so that the subsequent scheduling of the HPC scheduler is more accurate.

[0091] S36: The HPC scheduler schedules computing tasks according to the updated resource usage statistics of the cluster computing nodes.

[0092] Specifically, in the embodiment of the present invention, after updating the resource usage statistics of the cluster computing nodes, the unified computing power scheduler starts computing power scheduling of the HPC scheduler.

[0093] Figure 4 A cluster computing resource scheduling method provided by an embodiment of the present invention Figure 4 .

[0094] Furthermore, the above cluster computing resource scheduling method is combined with Figure 4 , and also includes six steps from S41 to S46:

[0095] S41: Send the first, continuous, and same-type computing tasks in the computing task scheduling queue to the corresponding HPC scheduler.

[0096] Specifically, in an embodiment of the present invention, if the continuous computing tasks at the head of the computing task scheduling queue in the unified computing power scheduler are HPC, the unified computing power scheduler sends the continuous computing tasks HPC at the head of the computing task scheduling queue to the HPC scheduler for scheduling by the HPC scheduler. For example, if the first three computing tasks in the queue are all HPC tasks, and the fourth computing task is a Kubernetes task, the unified computing power scheduler takes out the first three scheduling tasks from the queue at one time and sends them to the corresponding HPC scheduling.

[0097] S42: After each computing task is scheduled, the unified computing power scheduler sends the scheduling result of the HPC scheduler to the user.

[0098] Specifically, exemplarily, after each HPC scheduling in the first three computing tasks is completed, the unified computing power scheduler forwards the scheduling results of each HPC computing task to the user.

[0099] S43: After the scheduling of the first, continuous and same type of computing tasks is completed, the unified computing power scheduler counts the list of computing nodes where the first, continuous and same type of computing tasks are finally deployed.

[0100] Specifically and exemplarily, after the HPC scheduling of the first three computing tasks is completed, the unified computing power scheduler will count the list of computing nodes deployed by the HPC scheduling tasks of the first three computing tasks. The specific method is described in detail below.

[0101] S44: The resource monitoring agent deployed on the computing node in the node list receives a command from the unified computing power scheduler to collect the resource usage of the computing node and sends a heartbeat to the Kubernetes scheduler.

[0102] Specifically, in an embodiment of the present invention, the unified computing power scheduler commands the unified resource monitoring agent deployed on all computing nodes in the computing node list to collect resource occupancy information of each computing node, simulate the heartbeat message format of Kubernetes, and send a heartbeat to the Kubernetes scheduler.

[0103] S45: The Kubernetes scheduler updates the resource usage statistics of the cluster computing nodes based on the heartbeat.

[0104] Specifically, in an embodiment of the present invention, the Kubernetes scheduler updates the resource usage statistics of the cluster computing nodes according to the heartbeat, so that the subsequent scheduling of the Kubernetes scheduler is more accurate.

[0105] S46: The Kubernetes scheduler schedules computing tasks according to the updated resource usage statistics of the cluster computing nodes.

[0106] Specifically, in the embodiment of the present invention, after updating the resource usage statistics of the cluster computing nodes, the unified computing power scheduler starts the computing power scheduling of the Kubernetes scheduler.

[0107] Furthermore, in the above-mentioned cluster computing power resource scheduling method, the unified computing power scheduler provides a command line tool, and the program name of the command line tool is the same as the client tool of Kubernetes and HPC scheduler.

[0108] Specifically, in an embodiment of the present invention, if the unified computing power scheduler provides command line tools such as Kubectl, srun, and sbatch, the user can use these tools with the same name according to the usage methods of Kubernetes and Slurm client tools to submit computing tasks; if the user uses the Kubectl command line tool provided by the unified computing power scheduler to submit yaml, command parameters, etc. to the unified computing power scheduler, the unified computing power scheduler caches the computing tasks to the computing task scheduling queue; if the user uses the srun command line tool provided by the unified computing power scheduler to submit command parameters, source code or executable programs, etc. to the unified computing power scheduler, the unified computing power scheduler caches the computing tasks to the computing task scheduling queue.

[0109] Furthermore, in the above cluster computing resource scheduling method, the computing tasks are sent to the Kubernetes scheduler and the HPC scheduler respectively according to the types of computing tasks, and the unified computing scheduler sends the computing tasks respectively through the standard interfaces of the Kubernetes scheduler and the HPC scheduler according to the types of computing tasks;

[0110] The scheduling results of the Kubernetes scheduler and the HPC scheduler are sent to the user by the unified computing power scheduler through the standard interfaces of the Kubernetes scheduler and the HPC scheduler.

[0111] Specifically, in an embodiment of the present invention, the unified computing power scheduler sends tasks to Kubernetes and HPC schedulers according to task types and through the standard interfaces of each scheduler, and then returns the task scheduling results of Kubernetes and HPC schedulers to the user.

[0112] Figure 5 A diagram of a computing node list method for final deployment of a computing task provided in an embodiment of the present invention.

[0113] Furthermore, in the above cluster computing resource scheduling method, Figure 5 , the computing task finally deploys a list of computing nodes, including four steps from S51 to S54:

[0114] S51: Count the scheduling results of each computing task;

[0115] S52: Count the relevant Pods according to the resource type of the scheduling result and obtain a Pod list;

[0116] S53: Count the computing nodes where each Pod in the Pod list is located to form a computing node set;

[0117] S54: Determine a computing node list according to the computing node set.

[0118] Specifically, in the embodiment of the present invention, the resource type of the scheduling result of each computing task is first counted (for example, resource types such as ReplicationController or StatefulSet, etc., and then the relevant Pods are counted according to the resource type to obtain a Pod list, and then the computing nodes where each Pod in the Pod list is located are counted to form a computing node set, and the computing node set constitutes the computing node list.

[0119] Device Embodiment

[0120] Figure 6 A diagram of a cluster computing resource scheduling device provided in an embodiment of the present invention.

[0121] In a second aspect of the present invention, a cluster computing resource scheduling device is further proposed, which is applied to a system including: a Kubernetes scheduler, an HPC scheduler, and a unified computing scheduler, and is combined with Figure 6 ,include:

[0122] Receiving module 61 and storage module 62: used for the unified computing power scheduler to receive computing tasks submitted by users and cache the computing tasks in the computing task scheduling queue.

[0123] Specifically, in an embodiment of the present invention, the unified computing power scheduler is deployed on the Kubernetes scheduler and the HPC scheduler, the receiving module 61 uniformly receives computing tasks submitted by various types of users, and the storage module 62 of the unified computing power scheduler caches the user's computing tasks in the internal computing task scheduling queue.

[0124] The first sending module 63 is used to send the computing tasks to the Kubernetes scheduler and the HPC scheduler respectively according to the types of computing tasks and the order of computing tasks in the computing task scheduling queue.

[0125] Specifically, in an embodiment of the present invention, if the first computing task in the computing task scheduling queue is a Kubernetes task, the first sending module 63 is sent to the Kubernetes scheduler through the Kubernetes scheduler standard interface and scheduled by the Kubernetes scheduler. If the second computing task in the computing task scheduling queue is an HPC task, the first sending module 63 is sent to the HPC scheduler through the HPC scheduler standard interface and scheduled by the HPC scheduler.

[0126] The second sending module 64 is used for the Kubernetes scheduler and the HPC scheduler to send the computing tasks to the appropriate computing nodes according to the resource usage and scheduling strategy of the cluster computing nodes.

[0127] Specifically, in an embodiment of the present invention, Kubernetes and the HPC scheduler independently manage all computing nodes in the cluster. The second sending module 64 updates the resource usage of the cluster computing nodes according to the heartbeat messages of the computing nodes received respectively, receives and schedules the computing task requests of the applications, and sends the computing tasks to the appropriate computing nodes according to their respective scheduling strategies.

[0128] Sending module 65: used to send the scheduling results of the Kubernetes scheduler and the HPC scheduler to the user.

[0129] Specifically, in the embodiment of the present invention, the sending module 65 sends the scheduling results of the Kubernetes scheduler and the HPC scheduler to the user, so that the user can grasp the scheduling process and results in a timely manner, thereby improving the user experience.

[0130] In some embodiments, the computing task may also be Slurm, etc., and the scheduler may also be a Slurm scheduler.

[0131] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor and a memory;

[0132] The processor is used to execute any one of the above cluster computing power resource scheduling methods by calling the program or instruction stored in the memory.

[0133] In a fourth aspect of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute any one of the above cluster computing resource scheduling methods.

[0134] Figure 7 It is a schematic block diagram of an electronic device provided by an embodiment of the present invention.

[0135] like Figure 7As shown, the electronic device includes: at least one processor 701, at least one memory 702 and at least one communication interface 703. The various components in the electronic device are coupled together through a bus system 704. The communication interface 703 is used to transmit information between external devices. It can be understood that the bus system 704 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 704 .

[0136] It can be understood that the memory 702 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0137] In some implementations, the memory 702 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0138] Among them, the operating system includes various system programs, such as the framework layer, the core library layer, the driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. A program that implements any method in a cluster computing resource scheduling method provided in an embodiment of the present invention can be included in the application program.

[0139] In an embodiment of the present invention, the processor 701 calls a program or instruction stored in the memory 702, specifically, a program or instruction stored in an application, and the processor 701 is used to execute the steps of each embodiment of a cluster computing resource scheduling method provided in an embodiment of the present invention.

[0140] The unified computing power scheduler receives computing tasks submitted by users and caches the computing tasks in the computing task scheduling queue;

[0141] Dispatches computing tasks to the Kubernetes scheduler and HPC scheduler respectively according to their types and the order of computing tasks in the computing task scheduling queue;

[0142] The Kubernetes scheduler and HPC scheduler dispatch computing tasks to appropriate computing nodes based on the resource usage and scheduling policies of the cluster computing nodes;

[0143] Send the scheduling results of the Kubernetes scheduler and HPC scheduler to the user;

[0144] Among them, the unified computing power scheduler is deployed on the Kubernetes scheduler and HPC scheduler.

[0145] Any method in a cluster computing resource scheduling method provided in an embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 701 or an instruction in the form of software. The above-mentioned processor 701 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0146] The steps of any method in a cluster computing resource scheduling method provided by an embodiment of the present invention can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in the decoding processor. The software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the method in combination with its hardware.

[0147] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present invention and to form different embodiments.

[0148] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

[0150] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A cluster computing resource scheduling method, characterized in that: Applied to systems including Kubernetes scheduler, HPC scheduler, and unified computing scheduler, including: The unified computing power scheduler receives computing tasks submitted by users and caches the computing tasks in a computing task scheduling queue; Dispatches computing tasks to the Kubernetes scheduler and HPC scheduler respectively according to their types and the order of computing tasks in the computing task scheduling queue; The Kubernetes scheduler and HPC scheduler dispatch computing tasks to appropriate computing nodes based on the resource usage and scheduling policies of the cluster computing nodes; Send the scheduling results of the Kubernetes scheduler and HPC scheduler to the user; Among them, the unified computing power scheduler is deployed on the Kubernetes scheduler and the HPC scheduler.

2. A cluster computing resource scheduling method according to claim 1, characterized in that: The method further comprises: Deploy resource monitoring agents on all computing nodes in the cluster; The resource monitoring agent receives the command from the unified computing scheduler and sends heartbeats to the Kubernetes scheduler and the HPC scheduler; The Kubernetes scheduler and HPC scheduler update the resource usage of cluster computing nodes based on heartbeats.

3. A cluster computing resource scheduling method according to claim 1, characterized in that: The method further comprises: Send the first, continuous, and same-type computing tasks in the computing task scheduling queue to the corresponding Kubernetes scheduler; After each computing task is scheduled, the unified computing power scheduler sends the scheduling result of the Kubernetes scheduler to the user; After the scheduling of the first, continuous, and same type of computing tasks is completed, the unified computing power scheduler counts the computing node list where the first, continuous, and same type of computing tasks are finally deployed; The resource monitoring agent deployed on the computing nodes in the node list receives the command from the unified computing scheduler to collect the resource usage of the computing nodes and send a heartbeat to the HPC scheduler; The HPC scheduler updates the resource usage statistics of the cluster computing nodes based on the heartbeat; The HPC scheduler schedules computing tasks based on the updated resource usage statistics of the cluster computing nodes.

4. A cluster computing resource scheduling method according to claim 1, characterized in that: The method further comprises: Send the first, continuous, and same-type computing tasks in the computing task scheduling queue to the corresponding HPC scheduler; After each computing task is scheduled, the unified computing power scheduler sends the scheduling result of the HPC scheduler to the user; After the scheduling of the first, continuous, and same type of computing tasks is completed, the unified computing power scheduler counts the computing node list where the first, continuous, and same type of computing tasks are finally deployed; The resource monitoring agent deployed on the computing node in the node list receives the command from the unified computing scheduler to collect the resource usage of the computing node and send a heartbeat to the Kubernetes scheduler; The Kubernetes scheduler updates the resource usage statistics of the compute nodes based on the heartbeat; The Kubernetes scheduler schedules computing tasks based on the updated resource usage statistics of the cluster computing nodes.

5. A cluster computing resource scheduling method according to claim 1, characterized in that: The unified computing power scheduler provides a command line tool, and the program name of the command line tool is the same as the client tool of Kubernetes and HPC scheduler.

6. A cluster computing resource scheduling method according to claim 1, characterized in that: The unified computing scheduler sends computing tasks to the Kubernetes scheduler and the HPC scheduler respectively according to the type of computing tasks. The unified computing scheduler sends computing tasks to the Kubernetes scheduler and the HPC scheduler respectively through the standard interfaces of the Kubernetes scheduler and the HPC scheduler according to the type of computing tasks. The scheduling results of the Kubernetes scheduler and the HPC scheduler are sent to the user by the unified computing power scheduler through the standard interfaces of the Kubernetes scheduler and the HPC scheduler.

7. A cluster computing resource scheduling method according to claim 1, characterized in that: The list of computing nodes where the computing task is finally deployed includes: Statistics on the scheduling results of each computing task; According to the type of computing tasks and the scheduling results of each scheduling task, the relevant Pods are counted and a Pod list is obtained; Count the compute nodes where each Pod in the Pod list is located to form a compute node set; A list of computing nodes is determined based on the computing node set.

8. A cluster computing resource scheduling device, characterized in that: Applied to systems including Kubernetes scheduler, HPC scheduler, and unified computing scheduler, including: Receiving module and storage module: used for the unified computing power scheduler to receive computing tasks submitted by users and cache the computing tasks in the computing task scheduling queue; The first delivery module is used to deliver computing tasks to the Kubernetes scheduler and the HPC scheduler respectively according to the type of computing tasks and the order of computing tasks in the computing task scheduling queue; The second delivery module: used by the Kubernetes scheduler and HPC scheduler to deliver computing tasks to appropriate computing nodes based on the resource usage and scheduling policies of the cluster computing nodes; Sending module: used to send the scheduling results of Kubernetes scheduler and HPC scheduler to users; Among them, the unified computing power scheduler is deployed on the Kubernetes scheduler and the HPC scheduler.

9. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute a cluster computing resource scheduling method as described in any one of claims 1 to 7 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute a cluster computing resource scheduling method as described in any one of claims 1 to 7.

Images