A distributed scheduling system based on dynamic partitioning

Abstract

The application discloses a kind of distributed scheduling systems based on dynamic partition mode, comprising: cluster interface service, scheduling module, resource coordination module, scheduling feedback module;The cluster interface service provides the access interface of cluster resource;The scheduling module is responsible for selecting the appropriate node for each task to bind and run;The resource coordination module is responsible for each scheduling module to allocate schedulable node;The scheduling feedback module is responsible for the registration request of scheduling module and scheduling feedback result, and resource coordination module dynamically carries out node partition adjustment according to feedback result.The application is fixed under the condition that each scheduler handles partition data unchanged, when job quantity and cluster size increase, resource fragmentation is likely to occur, resulting in reduced resource utilization.The system adjusts the node distribution in each partition on demand through dynamic partitioning, reduces resource fragmentation, and improves cluster resource utilization.

Description

Technical Field

[0001] This invention relates to the technical field of data scheduling, and in particular to a distributed scheduling system based on dynamic partitioning. Background Technology

[0002] Currently, Kubernetes has become the de facto standard for cloud-native container orchestration. Numerous enterprises are using Kubernetes to build their information platforms, deploy containerized applications, and run various big data and AI jobs. The scheduling system, as the core component, plays a crucial role in improving cluster resource utilization and ensuring the stable operation of services within the cluster.

[0003] As business scales up, the number of nodes and different types of jobs in the cluster also increases, leading to performance issues with a single scheduling system. Traditional multi-cluster scheduling methods cannot share resources between clusters; traditional distributed scheduling methods, as the cluster size grows, increase the likelihood of conflicts between schedulers, which in turn reduces the performance and throughput of the scheduling system. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A distributed scheduling system based on dynamic partitioning includes:

[0007] Cluster interface service, scheduling module, resource coordination module, and scheduling feedback module;

[0008] The cluster interface service provides an access interface for cluster resources;

[0009] The scheduling module is responsible for selecting a suitable node for each task to bind and run, and at the same time, it reports the node usage to the scheduling feedback module to provide the resource coordination module with the basis for node allocation.

[0010] The resource coordination module is responsible for allocating schedulable nodes to each scheduling module and dynamically adjusting the schedulable nodes of the scheduling modules based on the feedback results from the scheduling modules.

[0011] The scheduling feedback module is responsible for receiving registration requests and scheduling feedback results from the scheduling module, and the resource coordination module dynamically adjusts node partitions based on the feedback results.

[0012] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the cluster interface service, scheduling module, resource coordination module, and scheduling feedback module all need to be deployed and implemented on a Kubernetes cluster. The cluster interface service is deployed as a Kubernetes API server, the scheduling module is deployed as a Kubernetes StatefulSet, the resource coordination module is deployed as a Kubernetes Deployment, and the scheduling feedback module uses Kafka to implement a distributed message middleware.

[0013] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the scheduling module and the resource feedback module are deployed to the cluster through configuration files, and the scheduling module registers its own information with the scheduling feedback module.

[0014] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the cluster interface service is provided with a list / watch interface, which works in conjunction with the scheduling module.

[0015] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the scheduling module obtains node information through the list / watch interface of the cluster interface service and puts it into the visible node cache.

[0016] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the scheduling module caches the jobs to be scheduled and the allocated nodes, and selects suitable nodes for each job to be executed through node pre-selection and node optimization according to different scheduling strategies, and binds the scheduling results to the cluster interface service.

[0017] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the scheduling module maintains an unavailable node cache, reports the unavailable node information to the scheduling feedback module, and waits for the feedback result. If the result is that the node is reclaimable, the scheduling module removes it from the allocated node cache and the unavailable node cache, and the resource coordination module performs secondary allocation.

[0018] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the resource coordination module dynamically adjusts the number of nodes in each partition according to the feedback result of the scheduling feedback module.

[0019] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the scheduling module will send a resource shortage feedback to the scheduling feedback module when the following conditions are met: the queue to be scheduled is not empty and the last submitted job has not undergone a binding operation within the configurable time last_job_waiting_time.

[0020] As a preferred embodiment of the distributed scheduling system based on dynamic partitioning described in this invention, the resource coordination module obtains full information through the cluster interface service, and randomly allocates [M / N] nodes to each scheduling module according to the total number of nodes M in the cluster.

[0021] The beneficial effects of this invention are:

[0022] 1. System Architecture: Compared to other distributed scheduling systems, when multiple schedulers perform scheduling simultaneously, scheduling conflicts are highly likely to occur. Therefore, a central arbitration system is needed to arbitrate the scheduling results, thereby reducing system performance and throughput. This system abandons the central arbitration system and avoids throughput reduction caused by conflicts through partitioning.

[0023] 2. Node Partitioning: Compared to traditional partitioning methods, where each scheduler processes fixed partition data, resource fragmentation can easily occur and reduce resource utilization as workload and cluster size increase. This system uses dynamic partitioning to adjust the node distribution in each partition as needed, reducing resource fragmentation and improving cluster resource utilization.

[0024] 3. Partitioning Strategy: Traditional partitioning strategies typically use a hash-based approach, which is relatively simplistic. This system employs multiple partitioning strategies, taking into account heterogeneous resources, node labels, node resource quantity, and partition balancing, to improve the parallel scheduling efficiency of the system after partitioning. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0026] Figure 1 This is a schematic diagram of the core module architecture of a distributed scheduling system based on dynamic partitioning proposed in this invention;

[0027] Figure 2 This is a schematic diagram of the module interaction architecture of a distributed scheduling system based on dynamic partitioning. Detailed Implementation

[0028] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0030] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0031] Secondly, the present invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of the present invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not according to the usual scale. Furthermore, the schematic diagrams are merely examples and should not limit the scope of protection of the present invention. In addition, actual fabrication should include three-dimensional spatial dimensions of length, width, and depth.

[0032] Reference Figure 1-2 This invention provides a distributed scheduling system based on dynamic partitioning, comprising:

[0033] Cluster interface service, scheduling module, resource coordination module, and scheduling feedback module;

[0034] The cluster interface service provides access interfaces for cluster resources;

[0035] The scheduling module is responsible for selecting a suitable node for each task to bind and run, and at the same time, it reports the node usage to the scheduling feedback module to provide a basis for node allocation to the resource coordination module.

[0036] The resource coordination module is responsible for allocating schedulable nodes to each scheduling module and dynamically adjusting the schedulable nodes of the scheduling modules based on the feedback results from the scheduling modules.

[0037] The scheduling feedback module is responsible for receiving registration requests and scheduling feedback results from the scheduling module, and the resource coordination module dynamically adjusts node partitions based on the feedback results.

[0038] The cluster interface service, scheduling module, resource coordination module, and scheduling feedback module all need to be deployed on the Kubernetes cluster. The cluster interface service is deployed as a Kubernetes API server, the scheduling module is deployed as a Kubernetes StatefulSet, the resource coordination module is deployed as a Kubernetes deployment, and the scheduling feedback module uses Kafka to implement a distributed message middleware. The cluster interface service achieves high availability (HA) through the Kubernetes API server by deploying multiple masters. The scheduling module obtains a unique ID information for each scheduling module through a stateful application via Kubernetes StatefulSet. The resource coordination module achieves HA through Kubernetes deployment by using multiple replicas.

[0039] Furthermore, the scheduling module and resource feedback module are deployed to the cluster through configuration files. The scheduling module registers its own information with the scheduling feedback module, waits for the resource coordination module to start up, and obtains the total number of scheduling modules N and its own index value k (0<=k<=N-1), indicating that the scheduling module belongs to partition k. The resource coordination module waits for all scheduling modules to complete registration and notifies each scheduling module of its sequence number k and total number N.

[0040] Furthermore, the cluster interface service has a list / watch interface, which works in conjunction with the scheduling module. The scheduling module obtains the job information submitted by the user through the list / watch interface of the cluster interface service, generates an integer hash value between 0 and N-1 based on the job name and submission time, and when the hash value is consistent with the index value of the current scheduling module, the job is scheduled by the current scheduling module and placed into the job cache to be scheduled.

[0041] Furthermore, the scheduling module obtains node information through the list / watch interface of the cluster interface service and places it in the visible node cache. If the node is marked as belonging to the partition where the current scheduling module is located, the node is placed in the allocated node cache. If the node is marked as not belonging to the partition where the current scheduling module is located but exists in the currently allocated node cache, the node is removed from the cache.

[0042] Furthermore, the scheduling module caches the jobs to be scheduled and the allocated nodes, and selects suitable nodes for each job to be executed according to different scheduling strategies through node pre-selection and node optimization, and binds the scheduling results to the cluster interface service.

[0043] Furthermore, the scheduling module maintains an unavailable node cache. The scheduling module reports unavailable node information to the scheduling feedback module and waits for the feedback result. If the result is that the node is reclaimable, the scheduling module removes it from both the allocated node cache and the unavailable node cache, and the resource coordination module performs secondary allocation. After each periodic scheduling process, a node will be placed in this cache if the following conditions are met: within the configurable time no_binding_action_timeout, the node has not undergone any job binding operation. The scheduling module reports unavailable node information to the scheduling feedback module and waits for the feedback result. If the result is that the node is reclaimable, the scheduling module removes it from both the allocated node cache and the unavailable node cache, and the resource coordination module performs secondary allocation.

[0044] Furthermore, the resource coordination module dynamically adjusts the number of nodes in each partition based on the feedback results from the scheduling feedback module. If a partition reports an unavailable node, and at least one partition reports insufficient resources, the unavailable node reported by that partition is reclaimed and added to the node pool to be allocated. If a partition reports an unavailable node, and no other partition reports insufficient resources, the unavailable node reported by that partition is not reclaimed. If the node pool to be allocated is not empty, and at least one partition reports insufficient resources, the resource coordination module allocates the nodes in the node pool to each partition according to the strategy.

[0045] Furthermore, the scheduling module will send a resource shortage feedback to the scheduling feedback module when the following conditions are met: the queue to be scheduled is not empty and the last submitted job has not undergone a binding operation within the configurable time last_job_waiting_time. The resource coordination module will try to add nodes to the partition based on the feedback result.

[0046] Furthermore, the resource coordination module obtains all information through the cluster interface service. Based on the total number of nodes M in the cluster, the resource coordination module randomly allocates [M / N] nodes to each scheduling module. The resource coordination module obtains all node information, all job information, all configuration information, etc. through the cluster interface service.

[0047] During use, users deploy the scheduling module and resource coordination module to the cluster via configuration files, specifying the total number of scheduling modules N. Each scheduling module registers its own information with the scheduling feedback module, starts the resource coordination module, and obtains the total number of scheduling modules N and its own index value k (0 <= k <= N-1), indicating that the scheduling module belongs to partition k. The resource coordination module waits for all scheduling modules to complete registration and notifies each scheduling module of its sequence number k and total number N. The resource coordination module obtains all information through the cluster interface service. Based on the total number of nodes M in the cluster, the resource coordination module randomly allocates [M / N] nodes to each scheduling module. Based on the user-submitted job volume, the number of nodes in the cluster, and the number of scheduling modules, the resource coordination module divides the nodes into several roughly equal partitions and assigns them to different scheduling modules for processing. Each scheduling module is only responsible for the scheduling requirements of the jobs and nodes assigned to its current partition and sends the scheduling results to the scheduling feedback module. Based on the feedback results, the resource coordination module dynamically adjusts the node distribution between partitions according to different strategies.

[0048] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A distributed dispatch system based on dynamic partitioning, characterized in that: The application relates to a cluster scheduling method and system. The cluster interface service provides an access interface of cluster resources. The scheduling module is responsible for selecting a node for each task to be bound and run, and feeds back node usage to the scheduling feedback module to provide a basis for node allocation for the resource coordination module. The resource coordination module is responsible for allocating schedulable nodes for each scheduling module, dynamically adjusts the schedulable nodes of the scheduling module according to the feedback result of the scheduling module, and dynamically adjusts the node partition of the scheduling module according to the feedback result of the scheduling feedback module. The scheduling feedback module is responsible for accepting the registration request and feedback result of the scheduling module, and the resource coordination module dynamically adjusts the node partition according to the feedback result. The cluster interface service, the scheduling module, the resource coordination module and the scheduling feedback module all need to be deployed on a kubernetes cluster, the cluster interface service is deployed as a kubernetes api-server, the scheduling module is deployed as a kubernetes statefulset, the resource coordination module is deployed as a kubernetes deployment, and the scheduling feedback module uses a kafka to realize a distributed message middleware. The scheduling module and the resource feedback module are deployed into the cluster through a configuration file, and the scheduling module registers the information of the scheduling module into the scheduling feedback module.

2. The distributed dispatch system based on dynamic partitioning according to claim 1, wherein: The cluster interface service is provided with a list / watch interface, and the list / watch interface cooperates with the scheduling module.

3. The distributed dispatch system based on dynamic partitioning as claimed in claim 2, wherein: The scheduling module obtains node information through the list / watch interface of the cluster interface service and puts the node information into a node cache.

4. The distributed dispatch system based on dynamic partitioning according to claim 3, wherein: The scheduling module maintains an unusable node cache, the scheduling module reports unusable node information to the scheduling feedback module, and waits for a feedback result, if the result is recyclable, the scheduling module removes the recyclable node from the allocated node cache and the unusable node cache, and the resource coordination module performs secondary allocation.

5. The distributed dispatch system based on dynamic partitioning as claimed in claim 4, wherein: The resource coordination module dynamically adjusts the number of nodes in each partition according to the feedback result of the scheduling feedback module.

6. The distributed dispatch system based on dynamic partitioning as claimed in claim 5, wherein: The scheduling module initiates a resource shortage feedback to the scheduling feedback module when the following conditions are met: the waiting queue is not empty and the last submitted job has not been bound within a configurable time last_job_waiting_time.

7. The distributed dispatch system based on dynamic partitioning as claimed in claim 6, wherein: The resource coordination module obtains total information through the cluster interface service, and the resource coordination module randomly allocates [M / N] nodes for each scheduling module according to the total number M of nodes in the cluster.

8. The distributed dispatch system based on dynamic partitioning as claimed in claim 7, wherein: ​

Images