Cluster scheduling method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a cluster scheduling method and device, computer equipment, a computer readable storage medium and a computer program product, and relates to the technical field of cloud native. According to the invention, under the condition of not changing the structure of the cluster and the scheduling system thereof, the influence of the scale expansion of the cluster on the cluster scheduling performance can be effectively dealt with. The method comprises the following steps: acquiring a target job, finding out a target node meeting a running condition of the target job in a plurality of partitions according to a partition priority sequence and node states and node snapshot data of nodes, and scheduling the target job to the target node, and updating node snapshot information of the target node in the node snapshot data and node real-time data of the target node in the node real-time data according to a scheduling result of the target node.

Description

Technical Field

[0001] The present application relates to the field of cloud native technology, and in particular to a cluster scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Kubernetes (K8s for short, an abbreviation that uses 8 to replace the eight characters in the middle of the name "ubernete" and is an open source containerized application for managing multiple hosts in a cloud platform) has become the de facto standard for resource management and container orchestration, and can easily manage massive container and node resources. More and more companies use it as a key technology for building cloud-native infrastructure, deploying and running big data and artificial intelligence operations, etc.

[0003] As the business scale grows, the number of nodes and jobs in the cluster also increases. The number of nodes can reach tens of thousands or even hundreds of thousands, and the number of jobs can even reach millions. In this case, in addition to the stability of the entire system, the job scheduling system also faces great pressure.

[0004] In this regard, in some related technologies, multiple clusters are combined into a large logical cluster, and a unified distribution system distributes jobs to each cluster. Then the scheduler in the cluster is responsible for scheduling the jobs, which requires two schedulings before they can run. In other related technologies, multiple schedulers in the cluster are scheduled in parallel, and a special component load needs to be added to coordinate the scheduling results of multiple schedulers. This technology requires each scheduler to save snapshot information of the entire cluster, and does not effectively reduce the pressure on the scheduler's storage. Therefore, there is a technical problem in the current related technologies that it is difficult to effectively deal with the impact of cluster scheduling performance caused by the expansion of cluster scale. Summary of the invention

[0005] Based on this, it is necessary to provide a cluster scheduling method, apparatus, computer equipment, computer-readable storage medium and computer program product to address the above technical issues.

[0006] In a first aspect, the present application provides a cluster scheduling method, comprising:

[0007] Get the target job;

[0008] In the order of partition priorities, the target nodes that meet the running conditions of the target job are found in multiple partitions according to the node status and node snapshot data of the nodes; wherein each of the partitions includes multiple nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each of the nodes; the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each of the nodes; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions;

[0009] Scheduling the target job to the target node;

[0010] According to the scheduling result of the target node, the node snapshot information of the target node in the node snapshot data is updated, and the node real-time information of the target node in the node real-time data is updated.

[0011] In one of the embodiments, the method also includes: obtaining node information to be updated; if the node real-time information of the first node to which the node information to be updated belongs exists in the node real-time data, then updating the node real-time information to the node information to be updated when the node information to be updated is different from the node real-time information; and, when the partition where the first node is located is in an unoccupied state, updating the node snapshot information of the first node in the node snapshot data according to the updated node real-time information; when the partition where the first node is located is in an occupied state, setting the node state of the first node to the state to be updated.

[0012] In one of the embodiments, after obtaining the node information to be updated, it also includes: if the node real-time information of the first node to which the node information to be updated belongs does not exist in the node real-time data, then adding a second node in the partition that meets the node quantity condition among the multiple partitions; and, based on the node information to be updated, adding the node real-time information of the second node in the node real-time data, and adding the node snapshot information of the second node in the node snapshot data.

[0013] In one of the embodiments, the method of searching for a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of partition priorities includes: polling the multiple partitions in the order of the partition priorities; in the polling of the multiple partitions, setting the current partition to an occupied state and setting the non-current partition to an unoccupied state; and, traversing the nodes included in the current partition according to the node status and node snapshot data in the current partition; in the node traversal of the current partition, if the node status of the current node is the to-be-updated state, updating the node snapshot information according to the node real-time information of the current node, and judging whether the running conditions are met according to the updated node snapshot information; if met, determining the current node as the target node; if not met, entering the target node judgment of the next node of the current partition; if the target node is not found in the node traversal of the current partition, entering the node traversal of the next partition among the multiple partitions.

[0014] In one of the embodiments, after finding the target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of partition priorities, the method further includes: for the first partition where the target node is found, increasing the partition priority of the first partition; for the second partition that has been polled and the target node has not been found, lowering the partition priority of the second partition.

[0015] In one of the embodiments, the method further includes: obtaining the total number of nodes in the cluster, and obtaining the number of nodes contained in each partition in the configuration file; creating the multiple partitions based on the total number of nodes and the number of nodes, and dividing the nodes in the cluster into the multiple partitions in turn.

[0016] In a second aspect, the present application further provides a cluster scheduling device, including:

[0017] A job acquisition unit, used for acquiring a target job;

[0018] A node search unit, used to search for a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of partition priority; wherein each of the partitions includes multiple nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each of the nodes; the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each of the nodes; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions;

[0019] A job scheduling unit, configured to schedule the target job to the target node;

[0020] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0021] Obtain a target job; in the order of partition priorities, find a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes; wherein each of the partitions includes multiple nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each of the nodes; the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each of the nodes; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions; schedule the target job to the target node; according to the scheduling result of the target node, update the node snapshot information of the target node in the node snapshot data and update the node real-time information of the target node in the node real-time data.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0023] Obtain a target job; in the order of partition priorities, find a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes; wherein each of the partitions includes multiple nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each of the nodes; the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each of the nodes; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions; schedule the target job to the target node; according to the scheduling result of the target node, update the node snapshot information of the target node in the node snapshot data and update the node real-time information of the target node in the node real-time data.

[0024] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0025] Obtain a target job; in the order of partition priorities, find a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes; wherein each of the partitions includes multiple nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each of the nodes; the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each of the nodes; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions; schedule the target job to the target node; according to the scheduling result of the target node, update the node snapshot information of the target node in the node snapshot data and update the node real-time information of the target node in the node real-time data.

[0026] The above-mentioned cluster scheduling method, device, computer equipment, computer-readable storage medium and computer program product obtain a target job, find out the target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of partition priority, schedule the target job to the target node, update the node snapshot information of the target node in the node snapshot data according to the scheduling result of the target node, and update the node real-time data of the target node in the node real-time data, wherein each partition includes multiple nodes in the cluster, the partition priority is related to the scheduling success rate of the partition, the node snapshot data stores the node snapshot information of each node, the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state, the node status indicates whether the node snapshot information of the node is in a pending update state, and the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions.

[0027] Therefore, this solution based on independent node snapshot data can avoid copying all node information during each scheduling, reducing time overhead. Partition-based node management can avoid locking the entire node snapshot data during the scheduling process. The partition-based snapshot update method can update the node information of nodes in different partitions in parallel to ensure the efficiency and timeliness of node information update. The partition priority related to the scheduling success rate can ensure that each scheduling selects the partition with a higher scheduling success rate as much as possible. This can effectively respond to the impact of cluster scale expansion on cluster scheduling performance without changing the cluster and its scheduling system structure, reduce the time consumption of copying node snapshot information, and ensure that the scheduling result meets the global optimality in terms of scheduling success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0029] FIG1 is an application environment diagram of a cluster scheduling method in one embodiment;

[0030] FIG2 is a schematic diagram of a flow chart of a cluster scheduling method in one embodiment;

[0031] FIG3 is a schematic diagram of updating node information in one embodiment;

[0032] FIG4 is a schematic diagram of updating node information in another embodiment;

[0033] FIG5 is a structural block diagram of a cluster scheduling device in one embodiment;

[0034] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] Explanation of some terms involved in this application:

[0037] Partition: A logical collection of nodes. The nodes in the cluster are divided into several partitions.

[0038] Node real-time data: The data structure that stores node information inside the scheduler can always be consistent with the node information in the interface service module. Node real-time data is organized by partition.

[0039] Node snapshot data: A data structure that stores node information inside the scheduler. It can be used as a copy of the node's real-time data and may differ from the node's real-time data. Node snapshot data is organized by partition.

[0040] As shown in FIG. 1 , the application environment of the cluster scheduling method of the present application is shown. The application environment may be called a scheduling system. The scheduling system may include an interface service module and a scheduler, wherein the scheduler may include a scheduling module and a synchronization module.

[0041] Among them, the interface service module can be used as the system access entrance of the scheduling system, and can provide interface operations such as job submission / update / search / deletion and node creation / update / search / deletion. The interface service module can also store data based on the ETCD (a distributed, reliable, and fast key-value storage system that can be used for shared configuration, service discovery, and consistency assurance) cluster, and can also be responsible for the authentication / authorization / verification of various resources, etc., and can also provide external RESTful APIs (Representational State Transfer application program interface API), and can use the native kubernetes api-server (one of the core components of Kubernetes, responsible for exposing the Kubernetes API to users and clients, receiving and processing requests from clients, and storing them in ETCD).

[0042] The scheduler may store node real-time data and node snapshot data.

[0043] Among them, the scheduling module in the scheduler can be responsible for job scheduling. In this application, the scheduling module can perform job scheduling based on node snapshot data, and can also be responsible for the synchronization of some node information between node real-time data and node snapshot data. The scheduling module can also dynamically maintain partitions and their partition priorities, and perform job scheduling according to the partition priorities.

[0044] Among them, the synchronization module in the scheduler can be responsible for the synchronization of node information, which can include the synchronization between the interface service module and the node real-time data, and can also include the synchronization of partial node information between the node real-time data and the node snapshot data.

[0045] Based on the scheduling system shown in FIG1 , the cluster scheduling method of the present application is described in combination with various embodiments and corresponding drawings.

[0046] In an exemplary embodiment, as shown in FIG2 , a cluster scheduling method is provided. The method can be applied to the scheduler in FIG1 , and can be executed by a scheduling module and a synchronization module in the scheduler. The cluster scheduling method can include the following steps:

[0047] Step S201, obtaining a target job.

[0048] The target job may be a job to be scheduled. In this step, the scheduling module in the scheduler may obtain a job from the multiple jobs to be scheduled as the target job.

[0049] Step S202 , searching for a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of the partition priorities.

[0050] The order of the partition priorities may be from higher to lower partition priorities. In this step, one partition may be taken out at a time to search for the target node in the order of higher to lower partition priorities.

[0051] Each partition contains multiple nodes in the cluster, and the nodes can be used to process jobs.

[0052] Each partition has its own partition priority, and the partition priority is related to the scheduling success rate of the partition. A partition with a higher scheduling success rate may have a higher partition priority.

[0053] In this step, the scheduling module can take out one partition at a time in order from higher to lower partition priorities. For this partition, the scheduling module can search for a target node in the partition that meets the running conditions of the target job based on the node status and node snapshot data of the node.

[0054] Wherein, the node snapshot data stores the node snapshot information of each node. As shown in FIG1 , the node snapshot data can manage the node snapshot information of each node in partitions. For example, for partitions 0 to k, each partition can contain, for example, N nodes. Then, the node snapshot data can manage the node snapshot information of different nodes according to partitions 0 to k. Wherein, when the node snapshot information of a node is in an unoccupied state in the partition where the node is located, the synchronization module can update the node real-time information of the node in the node real-time data, thereby ensuring the timeliness of the node snapshot information update. Wherein, the node real-time data stores the node real-time information of each node. Accordingly, as shown in FIG1 , the node real-time data also manages the node real-time information of each node in partitions. For example, for partitions 0 to k, each partition can contain, for example, N nodes. Then, the node real-time data can manage the node real-time information of different nodes according to partitions 0 to k.

[0055] Among them, the node status of the node indicates whether the node snapshot information of the node is in a pending update state. The pending update state can be used to instruct the scheduling module to update the node snapshot information of the node according to the node real-time information of the node before judging whether the node meets the operating conditions. This can ensure the timeliness of the node snapshot information update and the accuracy of judging whether the node is the target node.

[0056] Among them, the synchronization module and the scheduling module can update the node snapshot information in different partitions in parallel, which can improve the efficiency of updating the node snapshot information.

[0057] In this step, the scheduling module can determine whether the node needs to update the node snapshot information according to the node status of the node, and can determine whether the node is a target node that meets the running conditions of the target job according to the updated node snapshot information. As an example, the node snapshot information can be used to represent information such as the current service performance of the node, and the scheduling module can use this to determine whether the node can meet the running conditions of the target job, such as what indicators the service performance needs to reach under the running conditions. Therefore, the scheduling module can find the target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the node in the order of partition priority.

[0058] Step S203: Schedule the target job to the target node.

[0059] Step S204: updating the node snapshot information of the target node in the node snapshot data and updating the node real-time information of the target node in the node real-time data according to the scheduling result of the target node.

[0060] For steps S203 and S204, scheduling the target job to the target node can indicate that the target job has been successfully scheduled to the target node for processing, and the scheduling result of the target node can indicate that the job currently to be processed includes the target job. Therefore, according to the scheduling result, the node snapshot information and the node real-time information of the target node need to be updated. In conjunction with Figure 3, the target node may be node 1-3 in partition 1. According to the scheduling result, the scheduling module may update the node snapshot information of node 1-3 in the node snapshot data and update the node real-time information of node 1-3 in the node real-time data. The node snapshot information of node 1-3 in the node snapshot data may be updated first, and then the node real-time information of node 1-3 in the node real-time data may be updated based on the updated node snapshot information of node 1-3. This process may be referred to as post-scheduling update. Different from the pre-scheduling update, the pre-scheduling update may correspond to the scheduling module in step S202 determining whether the node needs to update the node snapshot information based on the node status of the node. Taking node 1-2 in partition 1 as an example, the scheduling module may determine whether the node needs to update the node snapshot information based on the node status of node 1-2. At this time, the scheduling module may update the node snapshot information of node 1-2 in the node snapshot data based on the node real-time information of node 1-2 in the node real-time data, and then determine whether node 1-2 belongs to the target node based on the updated node snapshot information of node 1-2.

[0061] After the scheduling of the target job is completed, the scheduling module can continue to schedule the next job.

[0062] The method of the above embodiment obtains the target job, searches for the target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of partition priority, schedules the target job to the target node, updates the node snapshot information of the target node in the node snapshot data according to the scheduling result of the target node, and updates the node real-time data of the target node in the node real-time data, wherein each partition includes multiple nodes in the cluster, the partition priority is related to the scheduling success rate of the partition, the node snapshot data stores the node snapshot information of each node, the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state, the node status indicates whether the node snapshot information of the node is in a pending update state, and the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions.

[0063] Therefore, the method of the above embodiment can avoid copying all node information during each scheduling based on independent node snapshot data, thereby reducing time overhead. Partition-based node management can avoid locking the entire node snapshot data during the scheduling process. The partition-based snapshot update method can update the node information of nodes in different partitions in parallel, thereby ensuring the efficiency and timeliness of node information updates. The partition priority related to the scheduling success rate can ensure that each scheduling selects the partition with a higher scheduling success rate as much as possible for scheduling. This can effectively address the impact of cluster scale expansion on cluster scheduling performance without changing the cluster and its scheduling system structure, reduce the time consumption of copying node snapshot information, and ensure that the scheduling results meet the global optimum in terms of scheduling success rate.

[0064] In an exemplary embodiment, the cluster scheduling method of the present application can also create partitions and divide nodes through the following steps before obtaining the target job:

[0065] Get the total number of nodes in the cluster and the number of nodes in each partition in the configuration file; create multiple partitions based on the total number of nodes and the number of nodes, and divide the nodes in the cluster into multiple partitions in turn.

[0066] In this embodiment, the scheduling module can obtain the total number of nodes M in the cluster, and the scheduling module can obtain the number of nodes N contained in each partition in the configuration file. The configuration file may be a related file for partitioning, and the configuration file may contain the number of nodes N contained in each partition. Thus, the scheduling module can determine the number of partitions K= M / N based on the total number of nodes M and the number of nodes N, and create partitions such as P1, P2, ..., Pk (as a representation of a partition identifier), etc. The scheduling module can traverse all nodes in the cluster and classify the nodes into partitions P1, P2, ..., Pk in turn until all nodes in the cluster are processed.

[0067] In an exemplary embodiment, the cluster scheduling method of the present application can also update the relevant node information through the following steps:

[0068] Obtain node information to be updated; if the node real-time information of the first node to which the node information to be updated belongs exists in the node real-time data, then update the node real-time information to the node information to be updated when the node information to be updated is different from the node real-time information; and, when the partition where the first node is located is in an unoccupied state, update the node snapshot information of the first node in the node snapshot data according to the updated node real-time information; when the partition where the first node is located is in an occupied state, set the node state of the first node to a state to be updated.

[0069] The solution of this embodiment can be implemented by the synchronization module in the scheduler to update the relevant node information of the node real-time data and the node snapshot data according to the node information to be updated.

[0070] 1 and 4, the synchronization module in the scheduler can obtain the node information to be updated through the interface service module. After obtaining the node information to be updated, the synchronization module can first search in the node real-time data whether there is the node real-time information of the first node to which the node information to be updated belongs.

[0071] If the node real-time information of the first node to which the node information to be updated belongs exists in the node real-time data, the synchronization module may further determine whether the node information to be updated is the same as (whether there is any change) the node real-time information.

[0072] In the case that the node information to be updated is the same as the node real-time information (no change), the synchronization module can process the next node information to be updated.

[0073] When the node information to be updated is different from the node real-time information (has changed), the synchronization module can update the node real-time information to the node information to be updated. At this time, the synchronization module also needs to determine whether the partition where the first node is located is occupied (or called used) by the scheduling module in the scheduler (in an unoccupied state or in an occupied state).

[0074] When the partition where the first node is located is in an unoccupied state (partition 0, partition k, etc. as shown in Figure 4), the synchronization module can update the node snapshot information of the first node in the node snapshot data according to the updated node real-time information, and then the synchronization module can process the next node information to be updated.

[0075] When the partition where the first node is located is in an occupied state (partition 1 as shown in FIG4 ), the synchronization module can set the node state of the first node to a pending update state, thereby indicating that the scheduling module needs to update the node snapshot information of the first node before determining whether the first node belongs to the target node. Then the synchronization module can process the next node information to be updated.

[0076] In an exemplary embodiment, further, after obtaining the node information to be updated in the above embodiment, the following may be included:

[0077] If the node real-time information of the first node to which the node information to be updated belongs does not exist in the node real-time data, then a second node is added to the partition that meets the node quantity condition among the multiple partitions; and, based on the node information to be updated, the node real-time information of the second node is added to the node real-time data, and the node snapshot information of the second node is added to the node snapshot data.

[0078] In this embodiment, for the node real-time information of the first node to which the node information to be updated belongs that does not exist in the node real-time data, the synchronization module can request the scheduling module to add a second node in a partition that meets the node quantity condition among multiple partitions (that is, create a new node in the partition), and the synchronization module can also add the node real-time information of the second node in the node real-time data, and add the node snapshot information of the second node in the node snapshot data. Then the synchronization module can process the next node information to be updated. Among them, the node quantity condition can be the partition with the least number of nodes, that is, the synchronization module can request the scheduling module to add a second node in the partition with the least number of nodes among multiple partitions, so that the scheduling performance of each partition can be balanced. In some other embodiments, if one or more nodes need to be deleted, the scheduling module can delete the nodes to be deleted in the partition to which they belong, and delete the node real-time information and node snapshot information of the node in the node real-time data and node snapshot data.

[0079] In an exemplary embodiment, step S202 of searching for a target node that meets the running condition of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of the partition priorities may include:

[0080] Poll multiple partitions in order of partition priority; in the polling of multiple partitions, set the current partition to an occupied state and set the non-current partition to an unoccupied state; and, in the current partition, traverse the nodes included in the current partition according to the node status and node snapshot data; in the node traversal of the current partition, if the node status of the current node is a pending update state, update the node snapshot information according to the node real-time information of the current node, and judge whether the running conditions are met according to the updated node snapshot information; if met, determine the current node as the target node; if not met, enter the target node judgment of the next node of the current partition; if the target node is not found in the node traversal of the current partition, enter the node traversal of the next partition in the multiple partitions.

[0081] In this embodiment, the scheduling module may poll multiple partitions in order from higher to lower partition priorities. The scheduling module may sort the multiple partitions in order from higher to lower partition priorities, with the partition with the highest partition priority being at the front and the partition with the lowest partition priority being at the back. In the polling of these partitions, the scheduling module may set the current partition (being polled) to an occupied state (in use), and other partitions, i.e., non-current partitions, to an unoccupied state (which may be indicated by not marking in use).

[0082] Then, the scheduling module traverses the nodes contained in the current partition according to the node status and node snapshot data in the current partition. In the node traversal of the current partition, the scheduling module can determine whether the node status of the current node (being traversed) is a state to be updated. If the node status of the current node is a state to be updated, the scheduling module updates the node snapshot information of the current node in the node snapshot data according to the node real-time information of the current node in the node real-time data. After the update, the scheduling module can delete the state to be updated. The scheduling module can determine whether the current node meets the running conditions according to the updated node snapshot information, that is, whether the target job can be run on the current node. If it is satisfied, the scheduling module can determine that the current node is the target node. If not satisfied, the scheduling module can enter the target node judgment of the next node of the current partition, that is, the next node of the current partition is used as the new current node, and continue to judge whether its node status is a state to be updated, whether it meets the running conditions, etc., until the target node is found or the current partition is traversed. If the scheduling module finds the target node in the node traversal of the current partition, the scheduling module enters the node traversal of the next partition among the multiple partitions, and takes the next partition among the multiple partitions as the new current partition, and continues to traverse the nodes contained in the new current partition according to the node status and node snapshot data until the target node is found or the entire partition is traversed. In some embodiments, if the target node is not found after traversing the entire partition, the scheduling module can determine that the scheduling of the target job has failed.

[0083] In an exemplary embodiment, further, after searching for a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of the partition priorities in step S202, the method of the present application may further include the following steps:

[0084] For the first partition that has found the target node, the partition priority of the first partition is increased; for the second partition that has been polled but has not found the target node, the partition priority of the second partition is reduced.

[0085] The solution of this embodiment can adjust the partition priority according to the scheduling success rate of the partition, ensuring that the partition with a higher scheduling success rate can be preferentially selected for scheduling each time, thereby improving the efficiency and effectiveness of cluster scheduling.

[0086] In this embodiment, for the first partition that finds the target node, the scheduling module can increase its partition priority, and for the second partition that has been polled but has not found the target node, the scheduling module can reduce its partition priority. As an implementation method, the increase and decrease of the partition priority can be carried out in the following ways:

[0087] You can set the upper limit H and lower limit L of the partition priority.

[0088] A first threshold T1 and a second threshold T2 for adjusting the partition priority may be set, and may be used to control the increase and decrease of the partition priority respectively. For the values ​​of the first threshold T1 and the second threshold T2, as an example: T1=T2=(H+L) / 2.

[0089] For a partition with a partition priority of P, when its partition priority is increased:

[0090] If the partition priority P is less than the first threshold value T1, the increased partition priority P is P*2; if the increased partition priority P is greater than the upper limit H, the increased partition priority P is set to the upper limit H;

[0091] If the partition priority P is greater than or equal to the first threshold T1, the increased partition priority P is P+1; if the increased partition priority P is greater than the upper limit H, the increased partition priority P is set to the upper limit H.

[0092] For a partition with a partition priority of P, when lowering its partition priority:

[0093] If the partition priority P is less than the second threshold value T2, the partition priority P after being reduced is P-2; if the partition priority P after being reduced is less than the lower limit L, the partition priority P after being reduced is set to the lower limit L;

[0094] If the partition priority P is greater than or equal to the second threshold T2, the reduced partition priority P is P / 2; if the reduced partition priority P is less than the lower limit L, the reduced partition priority P is set to the lower limit L.

[0095] In this way, the partition priority can be adjusted hierarchically according to the relationship between the partition priority and the corresponding threshold.

[0096] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0097] Based on the same inventive concept, the embodiment of the present application also provides a cluster scheduling device for implementing the cluster scheduling method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more cluster scheduling device embodiments provided below can refer to the above limitations on the cluster scheduling method, and will not be repeated here.

[0098] In an exemplary embodiment, as shown in FIG5 , a cluster scheduling device is provided. The device 500 may include:

[0099] A job acquisition unit 501 is used to acquire a target job;

[0100] A node search unit 502 is used to search for a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of partition priority; wherein each of the partitions includes multiple nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each of the nodes; the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each of the nodes; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions;

[0101] A job scheduling unit 503, configured to schedule the target job to the target node;

[0102] The first updating unit 504 is used to update the node snapshot information of the target node in the node snapshot data and update the node real-time information of the target node in the node real-time data according to the scheduling result of the target node.

[0103] In an exemplary embodiment, the device 500 may also include: a second update unit, used to obtain node information to be updated; if the node real-time information of the first node to which the node information to be updated belongs exists in the node real-time data, then when the node information to be updated is different from the node real-time information, the node real-time information is updated to the node information to be updated; and, when the partition where the first node is located is in an unoccupied state, the node snapshot information of the first node in the node snapshot data is updated according to the updated node real-time information; when the partition where the first node is located is in an occupied state, the node state of the first node is set to the state to be updated.

[0104] In an exemplary embodiment, the second update unit is also used to add a second node in a partition that meets the node quantity condition among the multiple partitions if the node real-time information of the first node to which the node information to be updated belongs does not exist in the node real-time data; and, based on the node information to be updated, add the node real-time information of the second node to the node real-time data, and add the node snapshot information of the second node to the node snapshot data.

[0105] In an exemplary embodiment, the node search unit 502 is used to poll the multiple partitions in the order of the partition priorities; in the polling of the multiple partitions, the current partition is set to an occupied state and the non-current partition is set to an unoccupied state; and, in the current partition, the nodes included in the current partition are traversed according to the node status and node snapshot data; in the node traversal of the current partition, if the node status of the current node is the to-be-updated state, the node snapshot information is updated according to the node real-time information of the current node, and whether the operating conditions are met is determined according to the updated node snapshot information; if met, the current node is determined to be the target node; if not met, the target node judgment of the next node of the current partition is entered; if the target node is not found in the node traversal of the current partition, the node traversal of the next partition in the multiple partitions is entered.

[0106] In an exemplary embodiment, the node search unit 502 is also used to increase the partition priority of the first partition for which the target node is found; and to lower the partition priority of the second partition for which the target node has been polled but not found.

[0107] In an exemplary embodiment, the device 500 may also include: a partitioning unit, used to obtain the total number of nodes in the cluster and the number of nodes contained in each partition in the configuration file; create the multiple partitions based on the total number of nodes and the number of nodes, and divide the nodes in the cluster into the multiple partitions in turn.

[0108] Each module in the above cluster scheduling device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0109] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store node snapshot data, node real-time data, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a cluster scheduling method is implemented.

[0110] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0112] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0113] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0115] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0116] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0117] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A cluster scheduling method, characterized in that: The method comprises: Get the target job; In the order of partition priorities, the target nodes that meet the running conditions of the target job are found in multiple partitions according to the node status and node snapshot data of the nodes; wherein each of the partitions includes multiple nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each of the nodes; the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each of the nodes; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions; Scheduling the target job to the target node; According to the scheduling result of the target node, the node snapshot information of the target node in the node snapshot data is updated, and the node real-time information of the target node in the node real-time data is updated.

2. The method according to claim 1, characterized in that: The method further comprises: Get the node information to be updated; If the node real-time data contains node real-time information of the first node to which the node information to be updated belongs, then if the node information to be updated is different from the node real-time information, updating the node real-time information to the node information to be updated; and, When the partition where the first node is located is in an unoccupied state, updating the node snapshot information of the first node in the node snapshot data according to the updated node real-time information; When the partition where the first node is located is in an occupied state, the node state of the first node is set to the to-be-updated state.

3. The method according to claim 2, characterized in that After obtaining the node information to be updated, the method further includes: If the node real-time information of the first node to which the node information to be updated belongs does not exist in the node real-time data, then a second node is added to the partition that meets the node quantity condition among the multiple partitions; and, based on the node information to be updated, the node real-time information of the second node is added to the node real-time data, and the node snapshot information of the second node is added to the node snapshot data.

4. The method according to claim 1, characterized in that: The step of searching for a target node that meets the running condition of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of the partition priorities includes: polling the plurality of partitions in order of the partition priorities; In the polling of the plurality of partitions, setting the current partition to an occupied state and setting non-current partitions to an unoccupied state; and, In the current partition, traverse the nodes included in the current partition according to the node status and the node snapshot data; In the node traversal of the current partition, if the node state of the current node is the to-be-updated state, the node snapshot information is updated according to the node real-time information of the current node, and whether the running condition is met is judged according to the updated node snapshot information; if so, the current node is determined to be the target node; if not, the target node judgment of the next node of the current partition is entered; If the target node is not found in the node traversal of the current partition, the node traversal of the next partition among the multiple partitions is started.

5. The method according to claim 4, characterized in that After searching for a target node that meets the running condition of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of the partition priorities, the method further includes: For the first partition that has found the target node, the partition priority of the first partition is increased; for the second partition that has been polled but has not found the target node, the partition priority of the second partition is lowered.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Get the total number of nodes in the cluster and the number of nodes contained in each partition in the configuration file; The multiple partitions are created according to the total number of nodes and the number of nodes, and the nodes in the cluster are divided into the multiple partitions in sequence.

7. A cluster scheduling device, characterized in that: The device comprises: A job acquisition unit, used for acquiring a target job; A node search unit, used to search for a target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data of the nodes in the order of partition priority; wherein each of the partitions includes multiple nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each of the nodes; the node snapshot information is updated according to the node real-time information of the node in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each of the nodes; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before judging whether the node meets the running conditions; A job scheduling unit, configured to schedule the target job to the target node; The first updating unit is used to update the node snapshot information of the target node in the node snapshot data and update the node real-time information of the target node in the node real-time data according to the scheduling result of the target node.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images