A job scheduling method and device, electronic equipment and storage medium

Abstract

The application discloses a job scheduling method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining a to-be-processed computing job from a first scheduling queue in a plurality of scheduling queues; determining a target computing cluster from a plurality of computing clusters based on the metadata of the to-be-processed computing job, wherein the plurality of computing clusters comprise heterogeneous computing power clusters of different cluster types; converting the to-be-processed computing job into a target computing job consistent with the job submission mode of the target computing cluster and distributing the target computing job to the target computing cluster; in response to a resource state update triggered based on the release of the heterogeneous computing power resource, performing scheduling state evaluation on a computing job in a second scheduling queue, and when the result of the scheduling state evaluation indicates that the computing job can be scheduled, transferring the computing job to the first scheduling queue. The application improves the job processing efficiency while realizing the joint scheduling of computing clusters of different cluster types.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a job scheduling method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, computing power has become a core resource supporting complex computing tasks and intelligent applications. Computing centers, as critical computing hubs, integrate heterogeneous computing clusters such as large-scale servers, GPU clusters, and high-speed interconnect components, providing a unified platform for massive data processing and large model training, thus becoming the core infrastructure of the intelligent era. Currently, the heterogeneous computing clusters in computing centers mainly include two types: high-performance computing clusters (such as Slurm computing clusters) and containerized clusters (such as Kubernetes computing clusters).

[0003] However, in related technologies, job scheduling for computing clusters of different cluster types is independent of each other, making global scheduling management impossible and resulting in low job processing efficiency. Summary of the Invention

[0004] To address the problems of the prior art, embodiments of this application provide a job scheduling method, apparatus, electronic device, and storage medium. The technical solution is as follows:

[0005] On the one hand, a job scheduling method is provided, the method comprising:

[0006] Obtain computation jobs to be processed from a first scheduling queue among multiple scheduling queues; the multiple scheduling queues include the first scheduling queue and the second scheduling queue, the first scheduling queue is used to store computation jobs that can be scheduled, and the second scheduling queue is used to store computation jobs that cannot be scheduled temporarily;

[0007] Based on the metadata of the computing job to be processed, a target computing cluster for processing the computing job is determined from multiple computing clusters; the multiple computing clusters include heterogeneous computing power clusters of different cluster types.

[0008] The pending computing job is converted into a target computing job that conforms to the job submission method of the target computing cluster, and the target computing job is distributed to the target computing cluster so that the target computing cluster executes the target computing job. When the execution of the target computing job ends, the heterogeneous computing resources occupied by the execution of the target computing job are released.

[0009] In response to a resource state update triggered based on releasing the heterogeneous computing resource, a scheduling state evaluation is performed on the computing job in the second scheduling queue, and when a result of the scheduling state evaluation indicates that the computing job can be scheduled, the computing job is transferred to the first scheduling queue.

[0010] In some example embodiments, the metadata of the to-be-processed computing job includes a cluster identifier; and determining, based on the metadata of the to-be-processed computing job, a target computing cluster for processing the to-be-processed computing job from a plurality of computing clusters includes:

[0011] obtaining resource pool configuration information, the resource pool configuration information including configuration files of the computing clusters in the plurality of computing clusters;

[0012] determining, based on the resource pool configuration information, a target configuration file matching the cluster identifier;

[0013] determining, based on the target configuration file, a target computing cluster for processing the to-be-processed computing job from the plurality of computing clusters.

[0014] In some example embodiments, the scheduling state evaluation on the computing job in the second scheduling queue includes:

[0015] obtaining updated current resource state information, the current resource state information indicating current resource states of the computing clusters in the plurality of computing clusters;

[0016] taking a computing job at a head of the second scheduling queue as a to-be-evaluated computing job, determining, based on metadata of the to-be-evaluated computing job, a first computing cluster for processing the to-be-evaluated computing job, and an expected resource state required for executing the to-be-evaluated computing job;

[0017] when the current resource state of the first computing cluster meets the expected resource state required for executing the to-be-evaluated computing job, determining that a result of the scheduling state evaluation is that the computing job can be scheduled;

[0018] when the current resource state of the first computing cluster does not meet the expected resource state required for executing the to-be-evaluated computing job, determining that the result of the scheduling state evaluation is that the computing job cannot be scheduled temporarily.

[0019] In some example embodiments, the method further includes:

[0020] obtaining a first computing job submitted by a target account based on a job configuration interface;

[0021] Based on the metadata of the first computing job, a second computing cluster for processing the first computing job is determined from the plurality of computing clusters, as well as the expected resource status required to execute the first computing job;

[0022] Based on the matching between the current resource status of the second computing cluster and the expected resource status required to execute the first computing job, it is determined whether the first computing job can be scheduled at present;

[0023] If the first computing job is currently schedulable, the first computing job is stored in the first scheduling queue; if the first computing job is currently not schedulable, the first computing job is stored in the second scheduling queue.

[0024] In some exemplary embodiments, the method further includes:

[0025] Based on the correspondence between accounts and computing clusters, the current resource status of the computing cluster corresponding to the target account is obtained from the current resource status information;

[0026] Display the current resource status of the computing cluster corresponding to the target account to the target account.

[0027] In some exemplary embodiments, converting the pending computing job into a target computing job that conforms to the job submission method of the target computing cluster, and distributing the target computing job to the target computing cluster includes:

[0028] If the target computing cluster is a containerized cluster, the computing job to be processed is encapsulated into a workflow of the containerized cluster, and the workflow is written into the local queue of the target computing cluster.

[0029] If the target computing cluster is a high-performance computing cluster, the computing job to be processed is converted into a target command script for submitting a batch processing job, and the target command script is sent to the target computing cluster through the application programming interface of the high-performance computing cluster.

[0030] On the other hand, a job scheduling device is provided, the device comprising:

[0031] The pending job acquisition module is used to acquire pending computing jobs from a first scheduling queue among multiple scheduling queues; the multiple scheduling queues include the first scheduling queue and the second scheduling queue, the first scheduling queue is used to store computing jobs that can be scheduled, and the second scheduling queue is used to store computing jobs that cannot be scheduled temporarily;

[0032] a target cluster determination module configured to determine, based on metadata of the to-be-processed computing job, a target computing cluster for processing the to-be-processed computing job from a plurality of computing clusters; the plurality of computing clusters include heterogeneous computing clusters of different cluster types;

[0033] a job distribution module configured to convert the to-be-processed computing job into a target computing job that is consistent with a job submission manner of the target computing cluster, and distribute the target computing job to the target computing cluster, so that the target computing cluster executes the target computing job, and releases heterogeneous computing resources occupied by the target computing job when execution of the target computing job ends;

[0034] an activation module configured to, in response to a resource state update triggered based on the release of the heterogeneous computing resources, perform a scheduling state evaluation on a computing job in the second scheduling queue, and when a result of the scheduling state evaluation indicates that the computing job is schedulable, transfer the computing job to the first scheduling queue.

[0035] In some example embodiments, the metadata of the to-be-processed computing job includes a cluster identifier; and the target cluster determination module includes:

[0036] a resource pool configuration acquisition module configured to acquire resource pool configuration information, the resource pool configuration information including configuration files of each computing cluster in the plurality of computing clusters;

[0037] a target configuration file determination module configured to determine, based on the resource pool configuration information, a target configuration file that matches the cluster identifier;

[0038] a target cluster determination submodule configured to determine, based on the target configuration file, a target computing cluster for processing the to-be-processed computing job from the plurality of computing clusters.

[0039] In some example embodiments, the activation module includes:

[0040] a current resource state acquisition module configured to acquire updated current resource state information, the current resource state information indicating current resource states of each computing cluster in the plurality of computing clusters;

[0041] a first determination module configured to determine, based on metadata of a to-be-evaluated computing job that is a computing job at a head of the second scheduling queue, a first computing cluster for processing the to-be-evaluated computing job, and an expected resource state required for execution of the to-be-evaluated computing job;

[0042] The first evaluation module is configured to determine that the result of the scheduling state evaluation is schedulable if the current resource state of the first computing cluster meets the expected resource state required for executing the to-be-evaluated computing job.

[0043] The second evaluation module is configured to determine that the result of the scheduling state evaluation is temporarily unschedulable if the current resource state of the first computing cluster does not meet the expected resource state required for executing the to-be-evaluated computing job.

[0044] In some example embodiments, the apparatus further includes:

[0045] The submitted job obtaining module is configured to obtain a first computing job submitted by a target account based on a job configuration interface;

[0046] The second determining module is configured to determine, based on the metadata of the first computing job, a second computing cluster from the plurality of computing clusters for processing the first computing job, and an expected resource state required for executing the first computing job;

[0047] The third determining module is configured to determine, according to a matching condition between a current resource state of the second computing cluster and the expected resource state required for executing the first computing job, whether the first computing job is currently schedulable;

[0048] The job caching module is configured to store the first computing job to the first scheduling queue if the first computing job is currently schedulable, and store the first computing job to the second scheduling queue if the first computing job is currently unschedulable.

[0049] In some example embodiments, the apparatus further includes:

[0050] The account resource state obtaining module is configured to obtain, based on a correspondence between an account and a computing cluster, a current resource state of a computing cluster corresponding to the target account from the current resource state information;

[0051] The resource state presenting module is configured to present the current resource state of the computing cluster corresponding to the target account to the target account.

[0052] In some example embodiments, the job distribution module is specifically configured to, if the cluster type of the target computing cluster is a containerized cluster, encapsulate the to-be-processed computing job as a workflow of the containerized cluster, and write the workflow into a local queue of the target computing cluster;

[0053] In a case where the cluster type of the target computing cluster is a high-performance computing cluster, the to-be-processed computing job is converted into a target command script for submitting a batch processing job, and the target command script is sent to the target computing cluster through an application programming interface of the high-performance computing cluster.

[0054] In another aspect, an electronic device is provided, including a processor and a memory having stored therein at least one instruction or at least one program, which is loaded and executed by the processor to implement the job scheduling method of any of the above aspects.

[0055] In another aspect, a computer-readable storage medium is provided, having stored therein at least one instruction or at least one program, which is loaded and executed by a processor to implement the job scheduling method of any of the above aspects.

[0056] In another aspect, a computer program product or computer program is provided, including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the electronic device to perform the job scheduling method of any of the above aspects.

[0057] The embodiments of the present application obtain a to-be-processed computing job from a first scheduling queue in a plurality of scheduling queues, the plurality of scheduling queues including the first scheduling queue and a second scheduling queue, the first scheduling queue being used to store computing jobs that can be scheduled, and the second scheduling queue being used to store computing jobs that are temporarily not schedulable, determine a target computing cluster for processing the to-be-processed computing job from a plurality of computing clusters based on metadata of the to-be-processed computing job, the plurality of computing clusters including heterogeneous computing clusters of different cluster types, then convert the to-be-processed job into a target computing job that is consistent with a job submission manner of the target computing cluster, and distribute the target computing job to the target computing cluster, so that the target computing cluster executes the target computing job, and releases heterogeneous computing resources occupied by executing the target computing job when the execution of the target computing job ends. At the same time, by responding to a resource state update triggered by releasing the heterogeneous computing resources, a scheduling state evaluation is performed on a computing job in the second scheduling queue, and when the result of the scheduling state evaluation indicates that the computing job is schedulable, the computing job is transferred to the first scheduling queue, thereby scheduling computing jobs to computing clusters of different cluster types based on unified management of different cluster types of computing clusters, combining two-level scheduling buffering and dynamic scheduling strategies, and scheduling computing jobs to computing clusters of different cluster types, thereby realizing joint scheduling of computing clusters of different cluster types, and greatly improving job processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0059] Figure 1 is a system architecture diagram of a joint scheduling provided by an embodiment of the present application;

[0060] Figure 2 is a flow diagram of a job scheduling method provided by an embodiment of the present application;

[0061] Figure 3 is a flow diagram of another job scheduling method provided by an embodiment of the present application;

[0062] Figure 4 is a structural block diagram of a job scheduling device provided by an embodiment of the present application;

[0063] Figure 5 is a hardware structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the scope of protection of the present application.

[0065] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0066] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuitry or memory) or a combination thereof. Similarly, one processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0067] It can be understood that in the specific embodiments of the present application, data related to user information and the like are involved, and when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions.

[0068] Please refer to Figure 1 , which is a system architecture diagram of joint scheduling provided by an embodiment of the present application, including a joint scheduling node 110 and at least one computing power center 120. The joint scheduling node 110 and each computing power center 120 can communicate through network connection.

[0069] The computing power center 120 can include computing clusters of different cluster types, where the computing clusters can be heterogeneous computing clusters, and the heterogeneous computing power in the heterogeneous computing clusters can include but not limited to graphics processing units (GPUs), neural network processing units (NPUs), tensor processing units (TPUs), FPGA (Field Programmable Gate Array) hardware, etc.

[0070] Different cluster types can include containerized clusters, such as Kubernetes clusters as shown in Figure 1 , and high-performance computing clusters for high-performance computing tasks, such as Slurm clusters as shown in Figure 1 . Kubernetes clusters are widely used in cloud-native scenarios, are good at automatic deployment, elastic scaling and service orchestration of containerized applications, and have good scalability and ecological compatibility. Slurm clusters support large-scale batch job scheduling, resource reservation and queue management, and are suitable for scenarios with high demand for computing resources such as AI training and scientific computing.

[0071] The joint scheduling node 110 uniformly manages each computing cluster of the at least one computing power center 120 and can realize joint scheduling of computing clusters of different cluster types, so that each computing cluster of the at least one computing power center 120 constitutes a resource pool of the joint scheduling node 110. Specifically, the joint scheduling node 110 can synchronize configuration files of each computing cluster through a backend API (application programming interface) to generate resource pool configuration information. For example, for a Kubernetes cluster, the cluster is managed through a KubeConfig configuration file, and fields in the KubeConfig configuration file include: cluster name, cluster availability zone, platform type of the cluster, state of the cluster, server entry of the cluster, and user verification Token; for a Slurm cluster, the cluster is managed through a set SlurmConfig configuration file, and fields in the SlurmConfig configuration file include: cluster name, platform type of the cluster, state of the cluster, server entry of the cluster, username, and user verification Token.

[0072] The joint scheduling node 110 can synchronize resource states of each computing cluster in real time to obtain current resource state information. For example, the resource states of each computing cluster can be monitored based on Prometheus to collect resource state information of each computing cluster in real time, including CPU utilization, memory occupancy, disk I / O occupancy, network traffic information, and GPU utilization.

[0073] In the embodiments of the present application, the joint scheduling node 110 is provided with a first scheduling queue and a second scheduling queue, wherein the first scheduling queue and the second scheduling queue are respectively used to store computing jobs in different scheduling states, the first scheduling queue is used to store schedulable computing jobs, and the second scheduling queue is used to store temporarily unschedulable computing jobs. The joint scheduling node 110 obtains a to-be-processed computing job from the first scheduling queue, determines a target computing cluster for processing the to-be-processed computing job from a plurality of computing clusters based on metadata of the to-be-processed computing job, converts the to-be-processed computing job into a target computing job consistent with a job submission mode of the target computing cluster, and distributes the target computing job to the target computing cluster, so that the target computing cluster executes the target computing job and releases heterogeneous computing power resources occupied by executing the target computing job when the execution of the target computing job ends. Thus, the joint scheduling node 110 responds to resource state updates triggered based on the release of the above-mentioned heterogeneous computing power resources, performs scheduling state evaluation on a computing job in the second scheduling queue, and when the result of the scheduling state evaluation indicates that the computing job is schedulable, the computing job is transferred to the first scheduling queue. In this way, two-level scheduling buffering and dynamic scheduling strategies are realized through the first scheduling queue and the second scheduling queue, the joint scheduling of computing clusters of different cluster types is realized, and the job processing efficiency is greatly improved.

[0074] In one specific application scenario, the joint scheduling node 110 shows a job configuration interface to a user in the form of a web interface, the user can configure various job parameters of a required computing job through the job configuration interface, and submit a computing job generated based on the configured various job parameters through the job configuration interface, so that the joint scheduling node 110 obtains the submitted computing job, and combines the first scheduling queue and the second scheduling queue to perform job scheduling by using two-level scheduling buffering and a dynamic scheduling strategy. Details will be described later in the subsequent content of the embodiments of the present application.

[0075] It should be noted that the nodes / servers involved in the embodiments of the present application can be independent physical servers, or server clusters or distributed systems formed by multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs (Content Delivery Networks), and big data and artificial intelligence platforms. The terminal devices involved in the embodiments of the present application include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle-mounted terminals, and the like.

[0076] In one exemplary embodiment, each computing cluster in the joint scheduling node 110 and the computing power center 120 can be a node device in a blockchain system, and can share the obtained and generated information to other node devices in the blockchain system, to realize information sharing between multiple node devices. The multiple node devices in the blockchain system can be configured with the same blockchain, the blockchain is composed of multiple blocks, and the adjacent blocks have an association relationship, so that when the data in any block is tampered with, it can be detected through the next block, thereby avoiding the data in the blockchain being tampered with, and ensuring the security and reliability of the data in the blockchain.

[0077] Please refer to Figure 2 , which shows a flowchart of a job scheduling method provided by the embodiments of the present application. The method can be applied to the joint scheduling node in Figure 1 . It should be noted that the present specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of the many execution orders, and does not represent the only execution order. In actual system or product execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, as shown in Figure 2 , the job scheduling method of the embodiments of the present application can include:

[0078] S201, obtaining a to-be-processed computing job from a first scheduling queue in a plurality of scheduling queues.

[0079] The plurality of scheduling queues include the first scheduling queue and a second scheduling queue, the first scheduling queue is used to store a computing job that can be scheduled, and the second scheduling queue is used to store a computing job that is temporarily not schedulable.

[0080] Specifically, for the obtained computing job, the joint scheduling node first determines whether the current resource state of the plurality of computing clusters managed by the joint scheduling node meets the requirements of the computing job, and if so, determines that the scheduling state of the computing job is currently schedulable, and stores the computing job in the first scheduling queue, otherwise, if not, determines that the scheduling state of the computing job is currently not schedulable, and stores the computing job in the second scheduling queue.

[0081] The to-be-processed computing job can be a computing job at the head of the first scheduling queue.

[0082] Based on this, in some exemplary embodiments, as shown in Figure 3 Another flowchart of a job scheduling method is shown, and the method can further include:

[0083] S301, obtaining a first computing job submitted by a target account based on a job configuration interface.

[0084] The target account can be any computing resource tenant. The job configuration interface is used to configure job parameters of a required computing job, and then the required computing job can be generated based on the configured job parameters. Specifically, the job parameters can include but are not limited to cluster identifier, expected resource state, job type, etc. The job type can include container instance, network mapping, and distributed job, etc.

[0085] In a specific implementation, the terminal device can display the job configuration interface to the target account. The terminal device can generate a corresponding computing job based on the job parameters configured in the job configuration interface in response to a job submission instruction triggered based on the job configuration interface, which is referred to as a first computing job below, and send the first computing job to the joint scheduling node, so that the joint scheduling node obtains the first computing job submitted by the target account based on the job configuration interface.

[0086] S303, determining a second computing cluster for processing the first computing job from the plurality of computing clusters based on the metadata of the first computing job, and determining an expected resource state required for executing the first computing job.

[0087] Specifically, the metadata of the first computing job is used to describe the first computing job, and the metadata can include job basic information, resource requirement information, etc., which can be configured by the target account based on a job configuration interface. The job basic information can include a job identifier, a specified cluster identifier, a job type, etc., and the resource requirement information can include an expected resource state, which can represent the computing resource required to execute the job. The joint scheduling node can obtain the specified cluster identifier and the expected resource state in the metadata of the first computing job by parsing the metadata.

[0088] In the embodiments of the present application, the joint scheduling node can maintain resource pool configuration information, which includes configuration files of each computing cluster in the plurality of computing clusters of the computing center, such as KubeConfig configuration files and SlurmConfig configuration files. Each configuration file includes a cluster identifier of the corresponding cluster, wherein the cluster identifier is used to uniquely identify a computing cluster. It can be understood that each configuration file can also include other necessary information, such as the cluster availability zone, the platform type of the cluster, the state of the cluster, the server entry of the cluster, and the user authentication Token in the KubeConfig configuration file, and the platform type of the cluster, the state of the cluster, the server entry of the cluster, the username, and the user authentication Token in the SlurmConfig configuration file.

[0089] In a specific implementation, the joint scheduling node obtains the specified cluster identifier in the metadata of the first computing job based on the metadata, finds a target configuration file in the resource pool configuration information that matches the cluster identifier, and then determines a second computing cluster for processing the first computing job based on the target configuration file.

[0090] S305, according to the matching between the current resource state of the second computing cluster and the expected resource state required to execute the first computing job, determining whether the first computing job is currently schedulable.

[0091] In the embodiments of the present application, the joint scheduling node can collect the current resource state of each computing cluster in the plurality of computing clusters of the computing center in real time, and then obtain the current resource state information of the resource pool. It can be understood that the current resource state information of the resource pool can be updated with real-time collected data, wherein the current resource state of each computing cluster can represent the current usage of computing resources in the corresponding computing cluster, such as CPU utilization, memory occupancy, disk I / O occupancy, network traffic information, and GPU utilization, etc.

[0092] In a specific implementation, the joint scheduling node can obtain the current resource state of the second computing cluster from the current resource state information, and compare the current resource state of the second computing cluster with the expected resource state obtained by analyzing the metadata of the first computing job. If the current resource state of the second computing cluster meets the expected resource state required by the first computing job, it is determined that the scheduling state of the first computing job is currently schedulable. Otherwise, if the current resource state of the second computing cluster does not meet the expected resource state required by the first computing job, it is determined that the scheduling state of the first computing job is currently not schedulable.

[0093] S307, in the case that the first computing job is currently schedulable, storing the first computing job to the first scheduling queue; in the case that the first computing job is currently not schedulable, storing the first computing job to the second scheduling queue.

[0094] The above implementation determines the scheduling state of the first computing job submitted by the target account based on the job configuration interface according to the demand of the first computing job and the resource state of the computing cluster, and then stores it in the scheduling queue matching the scheduling state, so as to perform dynamic scheduling based on the scheduling state, which is beneficial to improve the job processing efficiency of the computing job.

[0095] S203, determining a target computing cluster for processing the to-be-processed computing job from a plurality of computing clusters based on the metadata of the to-be-processed computing job, wherein the plurality of computing clusters include heterogeneous computing clusters of different cluster types.

[0096] Specifically, the metadata of the to-be-processed computing job is used to describe the to-be-processed computing job, and can include a specified cluster identifier. Therefore, the above step S203 can include: obtaining resource pool configuration information, wherein the resource pool configuration information includes configuration files of each computing cluster in the plurality of computing clusters; determining a target configuration file matching the cluster identifier based on the resource pool configuration information; and determining a target computing cluster for processing the to-be-processed computing job from the plurality of computing clusters based on the target configuration file, so as to uniformly manage the plurality of computing clusters based on the resource pool configuration information, and accurately determine the target computing cluster for processing the to-be-processed computing job in combination with the resource pool configuration information.

[0097] S205, converting the to-be-processed computing job into a target computing job consistent with the job submission mode of the target computing cluster, and distributing the target computing job to the target computing cluster, so that the target computing cluster executes the target computing job, and releases the heterogeneous computing resources occupied by executing the target computing job when the execution of the target computing job ends.

[0098] Specifically, since the job submission modes of computing clusters of different cluster types are different, after determining the target computing cluster for processing the to-be-processed computing job, the to-be-processed computing job needs to be converted into a target computing job consistent with the job submission mode of the target computing cluster, so as to realize joint scheduling of computing clusters of different cluster types.

[0099] In some exemplary embodiments, the cluster types can include containerized clusters (such as Kubernetes clusters) and high-performance computing clusters (such as Slurm clusters), and thus the above step S205 can include, when implemented:

[0100] In the case where the cluster type of the target computing cluster is a containerized cluster, the to-be-processed computing job is encapsulated as a workflow of the containerized cluster, and the workflow is written into a local queue of the target computing cluster;

[0101] In the case where the cluster type of the target computing cluster is a high-performance computing cluster, the to-be-processed computing job is converted into a target command script for submitting a batch job, and the target command script is sent to the target computing cluster through an application programming interface of the high-performance computing cluster.

[0102] In a specific implementation, if the cluster type of the target computing cluster is a Kubernetes cluster, the joint scheduling node can encapsulate the target computing job as a workflow (WorkLoad) of the Kubernetes cluster through a Kueue submodule, and write the workflow (WorkLoad) into a local queue (LocalQueue) of the target computing cluster, so that the workflow (WorkLoad) can perform WorkLoad admission, resource allocation, POD creation, and resource backfilling operations under the unified quota constraints of the ClusterQueue of the target computing cluster, to execute the target computing job, and release the heterogeneous computing power resources occupied during execution when the target computing job is executed.

[0103] If the cluster type of the target computing cluster is a Slurm cluster, the federated scheduling node can convert the target computing job into a Sbatch script and distribute the Sbatch script to the target computing cluster through the REST backend API interface of the Slurm cluster, and the target computing cluster executes the target computing job. When the Slurm cluster executes the computing job, the computing job usually enters the PENDING queue in the Slurm cluster and is scheduled by the Slurm Scheduler based on the MultiFactor priority. When the system resources of the Slurm cluster meet the requirements, the state of the computing job changes to RUNNING. When the system resources of the Slurm cluster cannot meet the requirements of the computing job, the computing job enters the BackFill window and waits.

[0104] S207, in response to the resource state update triggered based on the release of the heterogeneous computing resource, performing scheduling state evaluation on the computing job in the second scheduling queue, and when the result of the scheduling state evaluation indicates that the computing job can be scheduled, transferring the computing job to the first scheduling queue.

[0105] Specifically, when the target computing cluster finishes executing the target computing job and releases the heterogeneous computing resource occupied by the execution of the target computing job, the federated scheduling node can monitor the change of the resource state and update the current resource state information of the resource pool, and then, in response to the resource state update triggered based on the release of the heterogeneous computing resource, perform scheduling state evaluation on the computing job in the second scheduling queue. The result of the scheduling state evaluation indicates whether the computing job can be scheduled, and when the result of the scheduling state evaluation indicates that the computing job can be scheduled, the computing job in the second scheduling queue is transferred to the first scheduling queue, and specifically, the computing job can be transferred to the tail of the first scheduling queue.

[0106] In a specific embodiment, the step S207 can include the following when performing the scheduling state evaluation on the computing jobs in the second scheduling queue: obtaining the updated current resource state information, the current resource state information indicating the current resource state of each computing cluster in the plurality of computing clusters; taking the computing job at the head of the second scheduling queue as a to-be-evaluated computing job, determining a first computing cluster for processing the to-be-evaluated computing job based on the metadata of the to-be-evaluated computing job, and determining the expected resource state required for executing the to-be-evaluated computing job; in a case where the current resource state of the first computing cluster meets the expected resource state required for executing the to-be-evaluated computing job, determining that the result of the scheduling state evaluation is schedulable; in a case where the current resource state of the first computing cluster does not meet the expected resource state required for executing the to-be-evaluated computing job, determining that the result of the scheduling state evaluation is temporarily unschedulable. Thus, when the heterogeneous computing power resource is released after the target computing job is executed, the scheduling state update of the second scheduling queue can be activated in time, and the admission condition of the computing job at the head of the second scheduling queue is re-evaluated, thereby realizing the dynamic heterogeneous computing power scheduling based on the two-level scheduling buffer, and improving the job processing efficiency.

[0107] In actual application, after executing the target computing job, the target computing cluster can return the corresponding computing job execution result to the joint scheduling node, and can display the computing job execution result to the account corresponding to the target computing job through the front-end web page.

[0108] In some exemplary embodiments, the joint scheduling node can store the correspondence between the accounts and the computing clusters in the resource pool, which can represent the resource group owned by the tenant of the corresponding account, and the resource group can include one or more computing clusters in the resource pool. Further, the job scheduling method of the embodiments of the present application can further include: obtaining the current resource state of the computing cluster corresponding to the target account from the current resource state information of the resource pool; and displaying the current resource state of the computing cluster corresponding to the target account to the target account. Specifically, the target account is any tenant, and the joint scheduling node can visually display the current resource state of the resource group owned by each tenant through the web interface of the front-end based on the current resource state information of the resource pool, thereby realizing the joint management of computing clusters of different cluster types.

[0109] From the above technical solutions of the embodiments of the present application, it can be seen that the embodiments of the present application can cooperatively manage the computing clusters of different cluster types (such as Kubernetes clusters and Slurm clusters) between the computing power centers, realize unified management and joint scheduling of heterogeneous computing power resources, and based on the first scheduling queue and the second scheduling queue corresponding to different scheduling states, adopt a two-level scheduling buffer and a dynamic scheduling strategy to schedule the computing jobs of users to the computing clusters of different cluster types, thereby greatly improving the job processing efficiency while realizing joint scheduling of the computing clusters of different cluster types.

[0110] Corresponding to the job scheduling method provided by the above several embodiments, the embodiments of the present application also provide a job scheduling device. Since the job scheduling device provided by the embodiments of the present application corresponds to the job scheduling method provided by the above several embodiments, the implementation modes of the foregoing job scheduling method are also applicable to the job scheduling device provided by the embodiments of the present application, which will not be described in detail in the embodiments.

[0111] Please refer to Figure 4 which shows a structure schematic diagram of a job scheduling device provided by the embodiments of the present application. The device has the function of realizing the job scheduling method in the above method embodiments. The function can be realized by hardware, or the corresponding software can be executed by hardware. As Figure 4 shown, the job scheduling device 400 can include:

[0112] A to-be-processed job obtaining module 410 is configured to obtain a to-be-processed computing job from a first scheduling queue in a plurality of scheduling queues. The plurality of scheduling queues include the first scheduling queue and a second scheduling queue. The first scheduling queue is used to store computing jobs that can be scheduled, and the second scheduling queue is used to store computing jobs that are temporarily unschedulable.

[0113] A target cluster determining module 420 is configured to determine a target computing cluster for processing the to-be-processed computing job from a plurality of computing clusters based on the metadata of the to-be-processed computing job. The plurality of computing clusters include heterogeneous computing power clusters of different cluster types.

[0114] A job distribution module 430 is configured to convert the to-be-processed computing job into a target computing job consistent with the job submission mode of the target computing cluster, and distribute the target computing job to the target computing cluster, so that the target computing cluster executes the target computing job, and releases the heterogeneous computing power resources occupied by executing the target computing job when the execution of the target computing job ends.

[0115] The activation module 440 is configured to perform a scheduling state evaluation on a computing job in the second scheduling queue in response to a resource state update triggered based on releasing the heterogeneous computing resource, and transfer the computing job to the first scheduling queue when a result of the scheduling state evaluation indicates that the computing job is schedulable.

[0116] In some example embodiments, the metadata of the to-be-processed computing job includes a cluster identifier; and the target cluster determination module 420 includes:

[0117] A resource pool configuration acquisition module is configured to acquire resource pool configuration information, the resource pool configuration information including configuration files of each computing cluster in the plurality of computing clusters;

[0118] A target configuration file determination module is configured to determine a target configuration file matched with the cluster identifier based on the resource pool configuration information;

[0119] A target cluster determination sub-module is configured to determine a target computing cluster for processing the to-be-processed computing job from the plurality of computing clusters based on the target configuration file.

[0120] In some example embodiments, the activation module 440 includes:

[0121] A current resource state acquisition module is configured to acquire updated current resource state information, the current resource state information indicating current resource states of each computing cluster in the plurality of computing clusters;

[0122] A first determination module is configured to determine, as a to-be-evaluated computing job, a computing job at a head of the second scheduling queue, determine, based on metadata of the to-be-evaluated computing job, a first computing cluster for processing the to-be-evaluated computing job, and a desired resource state required for executing the to-be-evaluated computing job;

[0123] A first evaluation module is configured to determine, in a case where a current resource state of the first computing cluster meets the desired resource state required for executing the to-be-evaluated computing job, that a result of the scheduling state evaluation is schedulable;

[0124] A second evaluation module is configured to determine, in a case where the current resource state of the first computing cluster does not meet the desired resource state required for executing the to-be-evaluated computing job, that the result of the scheduling state evaluation is temporarily not schedulable.

[0125] In some example embodiments, the apparatus 400 further includes:

[0126] A submitted job acquisition module is configured to acquire a first computing job submitted by a target account based on a job configuration interface;

[0127] a second determining module, configured to determine, based on the metadata of the first computing job, a second computing cluster for processing the first computing job from the plurality of computing clusters, and a desired resource state required by the first computing job;

[0128] a third determining module, configured to determine, according to matching between a current resource state of the second computing cluster and the desired resource state required by the first computing job, whether the first computing job is currently schedulable;

[0129] a job caching module, configured to cache the first computing job to the first scheduling queue in a case that the first computing job is currently schedulable, and cache the first computing job to the second scheduling queue in a case that the first computing job is currently not schedulable.

[0130] In some exemplary embodiments, the apparatus 400 further includes:

[0131] an account resource state obtaining module, configured to obtain, based on a correspondence between an account and a computing cluster, a current resource state of a computing cluster corresponding to the target account from the current resource state information;

[0132] a resource state presenting module, configured to present the target account with the current resource state of the computing cluster corresponding to the target account.

[0133] In some exemplary embodiments, the job distribution module 430 is specifically configured to: in a case that the cluster type of the target computing cluster is a containerized cluster, encapsulate the to-be-processed computing job as a workflow of the containerized cluster, and write the workflow into a local queue of the target computing cluster;

[0134] in a case that the cluster type of the target computing cluster is a high-performance computing cluster, convert the to-be-processed computing job into a target command script for submitting a batch processing job, and send the target command script to the target computing cluster through an application programming interface of the high-performance computing cluster.

[0135] It should be noted that the apparatus provided in the above embodiments, in realizing its functions, only takes the above-mentioned division of each functional module as an example for illustration, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0136] The electronic device provided in the embodiments of the present application comprises a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to implement any one of the job scheduling methods provided in the above method embodiments.

[0137] The memory can be used to store software programs and modules, and the processor executes various function applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory can also include a memory controller to provide access of the processor to the memory.

[0138] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal, a server or a similar computing device, that is, the above electronic device can include a computer terminal, a server or a similar computing device. Taking the case of running on a server as an example, Figure 5 is a hardware structure block diagram of a server running a job scheduling method provided by the embodiments of the present application, as Figure 5 shown, the server 500 can have a large difference due to different configurations or performances, and can include one or more central processing units (CPU) 510 (the central processing unit 510 can include but is not limited to a microprocessor MCU or a programmable logic device FPGA processing device), a memory 530 for storing data, one or more storage media 520 (such as one or more mass storage devices) for storing application programs 523 or data 522. Among them, the memory 530 and the storage medium 520 can be temporary storage or persistent storage. The program stored in the storage medium 520 can include one or more modules, each module can include a series of instruction operations in the server. Further, the central processing unit 510 can be configured to communicate with the storage medium 520 to execute a series of instruction operations in the storage medium 520 on the server 500. The server 500 can also include one or more power supplies 560, one or more wired or wireless network interfaces 550, one or more input and output interfaces 540, and / or one or more operating systems 521, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0139] The input / output interface 540 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 500. In one example, the input / output interface 540 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 540 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0140] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 500 may also include... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown.

[0141] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a job scheduling method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the job scheduling methods provided in the above-described method embodiments.

[0142] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the job scheduling methods provided in the above-described method embodiments.

[0143] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0144] It should be noted that the above-mentioned order of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. And the above describes the specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

[0145] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0146] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0147] The above only describes the preferred embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A job scheduling method characterized by comprising: The method comprises: obtaining a to-be-processed computing job from a first scheduling queue in a plurality of scheduling queues; the plurality of scheduling queues comprise the first scheduling queue and a second scheduling queue, the first scheduling queue is used to store a computing job that can be scheduled, and the second scheduling queue is used to store a computing job that is temporarily unschedulable; determining a target computing cluster for processing the to-be-processed computing job from a plurality of computing clusters based on metadata of the to-be-processed computing job; the plurality of computing clusters comprise heterogeneous computing clusters of different cluster types; converting the to-be-processed computing job into a target computing job consistent with a job submission mode of the target computing cluster, and distributing the target computing job to the target computing cluster, so that the target computing cluster executes the target computing job, and releases heterogeneous computing resources occupied by executing the target computing job when the execution of the target computing job ends; in response to a resource state update triggered based on the release of the heterogeneous computing resources, obtaining updated current resource state information indicating current resource states of each computing cluster in the plurality of computing clusters; taking a computing job at the head of the second scheduling queue as a to-be-evaluated computing job, determining a first computing cluster for processing the to-be-evaluated computing job based on metadata of the to-be-evaluated computing job, and determining an expected resource state required for executing the to-be-evaluated computing job; in a case where the current resource state of the first computing cluster meets the expected resource state required for executing the to-be-evaluated computing job, determining that a result of a scheduling state evaluation is schedulable; in a case where the current resource state of the first computing cluster does not meet the expected resource state required for executing the to-be-evaluated computing job, determining that the result of the scheduling state evaluation is temporarily unschedulable; and in a case where the result of the scheduling state evaluation indicates that the computing job is schedulable, transferring the computing job to the first scheduling queue.

2. The method of claim 1, wherein, The metadata of the to-be-processed computing job comprises a cluster identifier; and the determining of the target computing cluster for processing the to-be-processed computing job from the plurality of computing clusters based on the metadata of the to-be-processed computing job comprises: obtaining resource pool configuration information, the resource pool configuration information comprising configuration files of each computing cluster in the plurality of computing clusters; determining a target configuration file matched with the cluster identifier based on the resource pool configuration information; determining the target computing cluster for processing the to-be-processed computing job from the plurality of computing clusters based on the target configuration file.

3. The method of claim 1, wherein, The method further comprises: obtaining a first computing job submitted by a target account based on a job configuration interface; determining a second computing cluster for processing the first computing job from the plurality of computing clusters based on metadata of the first computing job, and an expected resource state required for executing the first computing job; determining whether the first computing job is currently schedulable according to a matching condition between a current resource state of the second computing cluster and the expected resource state required for executing the first computing job. In a case that the first computing job can be currently scheduled, the first computing job is stored to the first scheduling queue; in a case that the first computing job cannot be currently scheduled, the first computing job is stored to the second scheduling queue.

4. The method of claim 3, wherein, The method further includes: obtaining, from the current resource state information, a current resource state of a computing cluster corresponding to the target account based on a correspondence between accounts and computing clusters; displaying, to the target account, the current resource state of the computing cluster corresponding to the target account.

5. The method of claim 1, wherein, The converting the to-be-processed computing job into a target computing job consistent with a job submission mode of the target computing cluster and distributing the target computing job to the target computing cluster includes: in a case that a cluster type of the target computing cluster is a containerized cluster, encapsulating the to-be-processed computing job as a workflow of the containerized cluster, and writing the workflow into a local queue of the target computing cluster; in a case that the cluster type of the target computing cluster is a high-performance computing cluster, converting the to-be-processed computing job into a target command script for submitting a batch processing job, and sending the target command script to the target computing cluster through an application programming interface of the high-performance computing cluster.

6. A job scheduling apparatus characterized by comprising: The apparatus includes: a to-be-processed job obtaining module configured to obtain a to-be-processed computing job from a first scheduling queue in a plurality of scheduling queues; the plurality of scheduling queues include the first scheduling queue and a second scheduling queue, the first scheduling queue is configured to store schedulable computing jobs, and the second scheduling queue is configured to store temporarily unschedulable computing jobs; a target cluster determining module configured to determine, based on metadata of the to-be-processed computing job, a target computing cluster for processing the to-be-processed computing job from a plurality of computing clusters; the plurality of computing clusters include heterogeneous computing clusters of different cluster types; a job distributing module configured to convert the to-be-processed computing job into a target computing job consistent with a job submission mode of the target computing cluster, and distribute the target computing job to the target computing cluster, so that the target computing cluster executes the target computing job, and releases heterogeneous computing resources occupied by the execution of the target computing job when the execution of the target computing job ends. The activation module is configured to, in response to a resource state update triggered based on the release of the heterogeneous computing power resource, acquire updated current resource state information, the current resource state information indicating a current resource state of each computing cluster in the plurality of computing clusters; take a computing job at a head of the second scheduling queue as an evaluated computing job, determine, based on metadata of the evaluated computing job, a first computing cluster for processing the evaluated computing job and an expected resource state required for execution of the evaluated computing job; in a case where the current resource state of the first computing cluster satisfies the expected resource state required for execution of the evaluated computing job, determine that a result of the scheduling state evaluation is schedulable; in a case where the current resource state of the first computing cluster does not satisfy the expected resource state required for execution of the evaluated computing job, determine that the result of the scheduling state evaluation is temporarily non-schedulable; and in a case where the result of the scheduling state evaluation indicates schedulable, transfer the computing job to the first scheduling queue.

7. An electronic device, comprising: The computer readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the job scheduling method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the job scheduling method according to any one of claims 1-5.

9. A computer program, characterized in that, The computer program is executed by the processor to implement the job scheduling method according to any one of claims 1-5.

Images