Multi-cluster job scheduling method, device and system

Abstract

The invention relates to the technical field of big data processing, and discloses a multi-cluster job scheduling method, device and system.The job to be scheduled is scheduled to the corresponding target sub-cluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target sub-cluster, full use and fair use of resources are guaranteed, and the scheduling efficiency is improved. And moreover, the quota allocation condition can be known in time in actual use, and the use resources of other sets and subgroups can be shared while the own service is smoothly processed, so that the overall resource utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a multi-cluster job scheduling method, device and system. Background Art

[0002] With the popularization and development of cloud native technology, more and more enterprises choose to transform their business applications into containers and use kubernetes clusters to manage business applications. Due to different business needs and scenarios, enterprises often need to run multiple independent kubernetes clusters to manage different businesses, and different clusters often belong to different business departments. Therefore, when the number of clusters increases, the complexity of management and the difficulty of operation and maintenance will also increase. At the same time, due to the different nature of the business, the resource usage in each cluster is also different. Some cluster resources are idle, while other cluster resources are insufficient, resulting in inefficient resource utilization, which in turn affects the business. In response to this situation, the traditional method of manually migrating resources to shave peaks and fill valleys cannot meet the requirements of flexible resource utilization, so multi-cluster management came into being.

[0003] Related technologies: The first method focuses on improving the high availability of services in multiple clusters, splitting services and distributing them to different clusters to ensure the high availability of services in multiple clusters, that is, when one or several clusters have problems, users using the service are not affected. This method does not take into account the resource situation of the cluster. Some of the split clusters have sufficient resources, while others do not, resulting in the inability to fully utilize cluster resources. Therefore, the second method appears, which focuses on improving the resource utilization rate of multiple clusters. When deploying and distributing services or jobs, appropriate services or jobs are selected for distribution according to the resource usage of each sub-cluster, thereby improving the overall resource utilization rate and reducing production costs. In the first method, due to the lack of correlation and interoperability between different clusters, not only can resources not be shared, but also resources cannot be fully utilized. In the second method, since the quota ownership of each sub-cluster is not distinguished, in actual use, users of a sub-cluster cannot obtain the quota situation of the sub-cluster in a timely manner, thereby affecting the business of the sub-cluster itself. Summary of the invention

[0004] In view of this, the present invention provides a multi-cluster job scheduling method, device and system to solve the problem that resources cannot be shared and fully utilized, and sub-clusters fail to distinguish quota ownership, which affects the sub-cluster's own business.

[0005] In a first aspect, the present invention provides a multi-cluster job scheduling method comprising:

[0006] Obtain the resource information of the job to be scheduled and the target subcluster sent by the user; the resource information includes: the total resources of the target subcluster, the resources occupied by the currently running job, the resources occupied by the job to be run, and the occupied resources. The target subcluster is each subcluster in multiple subclusters;

[0007] Based on at least one of the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources, configure the exclusive quota and shared quota of the target subcluster; the exclusive quota and shared quota are divided based on the current quota of the target subcluster;

[0008] Allocate the jobs to be scheduled according to the first job queue and the second job queue;

[0009] Based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster, schedule the job to be scheduled to the corresponding target subcluster.

[0010] The disclosed embodiment schedules the job to be scheduled to the corresponding target subcluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster, which is conducive to ensuring full and fair use of resources, and is conducive to timely understanding of quota allocation in actual use. While smoothly processing its own business, it can also share the use resources of other subclusters, thereby improving the overall resource utilization rate.

[0011] In some optional implementations, based on at least one of the total resources of the target subcluster, the resources occupied by currently running jobs, the resources occupied by jobs to be run, and the occupied resources, the exclusive quota and the shared quota of the target subcluster are configured, including:

[0012] When a new job is added to the target subcluster, at least one job occupying the shared quota is evicted so that the exclusive quota is less than the occupied resources.

[0013] When the target subcluster's own jobs are executed, the resource usage of the target subcluster's own jobs is released, so that the exclusive quota is reduced and the shared quota is increased.

[0014] The disclosed embodiment combines the resource information of the target subcluster mentioned above, analyzes the usage of the exclusive quota and shared quota of the target subcluster, gives priority to the amount of resources occupied by the target subcluster's own jobs, avoids affecting the operation of its own jobs, and at the same time, partially reserves the exclusive quota to cope with the situation where users suddenly submit jobs, avoids affecting the business, and lends resources through the shared quota to improve the overall resource utilization rate.

[0015] In some optional implementations, allocating the to-be-scheduled jobs according to the first job queue and the second job queue includes:

[0016] Get the job type corresponding to the job to be scheduled;

[0017] If the job type of the job to be scheduled is a preemptible job, the job to be scheduled is assigned to the first job queue;

[0018] If the job type of the job to be scheduled is a non-preemptible job, the job to be scheduled is allocated to the second job queue.

[0019] The embodiment of the present disclosure places corresponding types of to-be-scheduled jobs in the first job queue and the second job queue respectively, which is conducive to ensuring the normal and orderly operation of the business of the target sub-cluster itself, and is also conducive to sharing the jobs of the own and other sub-clusters.

[0020] In some optional implementations, scheduling the job to be scheduled to the corresponding target subcluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster includes:

[0021] Based on the quota configuration result of the target subcluster, obtain the current remaining amount of the target subcluster's shared quota;

[0022] Select the target subcluster corresponding to the job to be scheduled based on the order of the current remaining amount of the shared quota from largest to smallest;

[0023] Get the free resource amount of the shared quota of the target subcluster corresponding to the job to be scheduled;

[0024] Determine whether the idle resources of the shared quota are sufficient to share the current resources of the job to be scheduled;

[0025] If yes, based on the job allocation result of the job to be scheduled, schedule the job to be scheduled to the shared quota of the target subcluster;

[0026] If not, continue to determine whether the next target sub-cluster among the multiple sub-clusters is sufficient to share the current resource amount of the job to be scheduled.

[0027] By performing the above actions, the disclosed embodiment is conducive to ensuring full and fair use of resources, and is conducive to timely understanding of quota allocation in actual use. While smoothly handling its own business, it can also share the use resources of other sub-groups, thereby improving the overall resource utilization rate.

[0028] According to a second aspect, an embodiment of the present disclosure provides a multi-cluster job scheduling device, the device comprising:

[0029] An acquisition unit is used to acquire resource information of the job to be scheduled and the target subcluster sent by the user; the resource information includes: the total amount of resources of the target subcluster, the amount of resources occupied by the currently running job, the amount of resources occupied by the job to be run, and the amount of occupied resources. The target subcluster is each subcluster in the multiple subclusters;

[0030] A configuration unit, configured to configure an exclusive quota and a shared quota of the target subcluster based on at least one of the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources; wherein the exclusive quota and the shared quota are divided based on the current quota of the target subcluster;

[0031] An allocating unit, configured to allocate the jobs to be scheduled according to the first job queue and the second job queue;

[0032] The scheduling unit is used to schedule the job to be scheduled to the corresponding target subcluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster.

[0033] According to a third aspect, an embodiment of the present disclosure provides a multi-cluster job scheduling system, including: a main cluster and multiple sub-clusters, wherein the main cluster includes: an interface module, a scheduling module, a quota management module and a data routing module,

[0034] Obtain the jobs to be scheduled initiated by the user through the interface module;

[0035] The resource information of the target subcluster is obtained through the data routing module; the resource information includes: the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources. The target subcluster is each subcluster in the multiple subclusters;

[0036] The exclusive quota and shared quota of the target subcluster are configured through the quota management module based on at least one of the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources; wherein the exclusive quota and shared quota are based on the current quota division of the target subcluster;

[0037] The scheduling module allocates the jobs to be scheduled according to the first job queue and the second job queue; based on the job allocation result of the jobs to be scheduled and the quota configuration result of the target subcluster, the jobs to be scheduled are scheduled to the corresponding target subcluster.

[0038] In a fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-cluster job scheduling method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0039] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multi-cluster job scheduling method of the first aspect or any corresponding embodiment thereof.

[0040] In a sixth aspect, a computer program product includes computer instructions, wherein the computer instructions are used to enable a computer to execute the multi-cluster job scheduling method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of a multi-cluster job scheduling method according to an embodiment of the present invention;

[0043] Figure 2 is a flow chart of another multi-cluster job scheduling method according to an embodiment of the present invention;

[0044] Figure 3 is a flowchart of another multi-cluster job scheduling method according to an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of expelling a job occupying a shared quota according to an embodiment of the present invention;

[0046] Figure 5 is a schematic diagram of releasing the resource usage of the target sub-cluster's own jobs according to an embodiment of the present invention;

[0047] Figure 6 is a flowchart of another multi-cluster job scheduling method according to an embodiment of the present invention;

[0048] Figure 7 is a structural block diagram of a multi-cluster job scheduling system according to an embodiment of the present invention;

[0049] Figure 8 is a structural block diagram of a multi-cluster job scheduling device according to an embodiment of the present invention;

[0050] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] According to an embodiment of the present invention, an embodiment of a multi-cluster job scheduling method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] In this embodiment, a multi-cluster job scheduling method is provided, which can be used for servers and mobile terminals, such as mobile phones and tablet computers. Figure 1 is a flowchart of a multi-cluster job scheduling method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0054] Step S101, obtaining the resource information of the job to be scheduled and the target subcluster sent by the user; the resource information includes: the total resources of the target subcluster, the resources occupied by the currently running job, the resources occupied by the job to be run, and the occupied resources. The target subcluster is each subcluster in the multiple subclusters.

[0055] Specifically, there can be one or more jobs to be scheduled, and the target sub-cluster is each sub-cluster in the multiple sub-clusters. Each sub-cluster corresponds to a user, and the user job can be run through each sub-cluster. The sub-cluster (SC) is managed by the master cluster. The master cluster (SC) is used to accept or respond to user request instructions and is responsible for managing the sub-cluster. The master cluster itself does not run user jobs. After receiving the request instruction of the job to be scheduled initiated by the user, the master cluster executes the multi-cluster job scheduling method in the embodiment of the present disclosure.

[0056] Furthermore, the total amount of resources of the target subcluster is represented by N, which represents the total amount of available resources of each subcluster; the amount of resources occupied by currently running jobs is represented by R, which represents the amount of resources occupied by jobs run by users of each subcluster; the amount of resources occupied by jobs to be run is represented by P; and the amount of occupied resources is represented by M, which represents the amount of resources occupied by jobs run by each subcluster.

[0057] Step S102, based on at least one of the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources, configure the exclusive quota and the shared quota of the target subcluster; wherein the exclusive quota and the shared quota are divided based on the current quota of the target subcluster.

[0058] Specifically, for each subcluster (target subcluster), the current quota of each subcluster is divided into an exclusive quota and a shared quota. The exclusive quota supports the use of users belonging to the target subcluster, and the shared quota supports users corresponding to other subclusters to use resources together.

[0059] In the disclosed embodiment, in each scheduling cycle, dynamic resource configuration is performed according to at least one of N, R, P and M mentioned above, which is conducive to ensuring full and fair use of resources, and distinguishing between exclusive quotas and shared quotas for each subcluster (target subcluster), which is conducive to timely understanding of quota allocation in actual use, and can share the use resources of other subcluster while smoothly handling its own business, thereby improving the overall resource utilization rate.

[0060] Step S103: Allocate the jobs to be scheduled according to the first job queue and the second job queue.

[0061] Specifically, the first job queue is also called the evictable job queue, PreemptableQueue in English, and the second job queue is also called the non-evictable job queue, NonPreemptableQueue in English. The multiple jobs to be scheduled placed in the first job queue are preemptible jobs, which can be run in the subcluster where the user is located or in other subclusters, and the job can tolerate the eviction operation caused by insufficient resources in other subclusters; the multiple jobs to be scheduled placed in the second job queue are non-preemptible jobs, which run in the subcluster where the user to be scheduled is located.

[0062] Step S104 : scheduling the job to be scheduled to the corresponding target subcluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster.

[0063] Specifically, for the job allocation results in the first job queue and the second job queue, and the quota results of the exclusive quota and the shared quota of each subcluster (target subcluster), the to-be-scheduled job is further scheduled to the corresponding target subcluster.

[0064] Based on the job assignment result of the job to be scheduled and the quota configuration result of the target sub-cluster, the embodiments of the present disclosure schedule the job to be scheduled to the corresponding target sub-cluster, which is beneficial to ensuring the full and fair use of resources. Moreover, in actual use, it is beneficial to timely understand the quota allocation situation, and while smoothly processing its own business, it can also share the usage resources of other sub-clusters, thereby improving the overall resource utilization rate.

[0065] In this embodiment, a multi-cluster job scheduling method is provided, which can be used in servers, mobile terminals, such as mobile phones, tablets, etc. Figure 2 It is a flowchart of the multi-cluster job scheduling method according to the embodiments of the present invention, as Figure 2 shown. The above steps of the process include the following steps:

[0066] Step S201, when the sum of the resource amount occupied by the running job and the resource amount occupied by the job to be run itself is greater than or equal to the total resource amount, determine that the exclusive quota is the total resource amount.

[0067] Specifically, for example, when R + P >= N, the exclusive quota is N and the shared quota is 0. At this time, it means that the job of the target sub-cluster itself needs to occupy a large amount of resources. To ensure the normal use of its own business first, the exclusive quota is N, and no shared resources can be provided to users of other sub-clusters, so the shared resource is 0.

[0068] Step S202, when the sum of the resource amount occupied by the running job, the resource amount occupied by the job to be run itself and the reserved resource amount is less than the total resource amount, determine that the exclusive quota is the sum of the resource amount occupied by the running job, the resource amount occupied by the job to be run itself and the reserved resource amount.

[0069] Specifically, for example, when R + P < N and R + P + E > N, where E is the reserved resource amount and E = N / 10. At this time, it means that the job of the target sub-cluster itself also occupies a large amount of resources. To ensure the normal use of its own business first, the exclusive quota is N, and no shared resources can be provided to users of other sub-clusters, so the shared resource is 0. The above steps S201 - step S202 do not distinguish the order.

[0070] Step S203, when the sum of the resource amount occupied by the running job, the resource amount occupied by the job to be run itself and the reserved resource amount is less than the total resource amount, determine that the exclusive quota is the sum of the resource amount occupied by the running job, the resource amount occupied by the job to be run itself and the reserved resource amount, and the shared quota is the difference between the total resource amount and the sum result.

[0071] Specifically, for example, when R + P + E < N, where E is the reserved resource amount and E = N / 10. At this time, it indicates that the resources occupied by the own jobs of the target sub-cluster are not large and there is still room for reservation, so the exclusive quota is R + P + E, and the shared quota is N - (R + P + E). By reserving part of the exclusive quota, it can cope with the situation of users submitting jobs suddenly and avoid affecting the business. By lending resources in the way of shared quota, the overall resource utilization rate can be improved.

[0072] By executing the above steps S201 - step S203, the embodiments of the present disclosure analyze the usage of the exclusive quota and the shared quota of the target sub-cluster by combining R, P, E, and N in the above, giving priority to considering the resource amount occupied by the own jobs of the target sub-cluster to avoid affecting the operation of its own jobs. At the same time, by reserving part of the exclusive quota, it can cope with the situation of users submitting jobs suddenly and avoid affecting the business. By lending resources in the way of shared quota, the overall resource utilization rate can be improved.

[0073] In this embodiment, a multi-cluster job scheduling method is provided, which can be used in servers, mobile terminals, such as mobile phones, tablet computers, etc. Figure 3 It is a flowchart of the multi-cluster job scheduling method according to the embodiments of the present invention, as Figure 3 shown. In step S103, based on at least one of the total resources of the target sub-cluster, the resources occupied by the currently running own jobs, the resources occupied by the jobs to be run, and the occupied resources, configure the exclusive quota and the shared quota of the target sub-cluster. This process includes the following steps:

[0074] In step 301, when a new job is added to the target sub-cluster, evict at least one job occupying the shared quota so that the exclusive quota is less than the occupied resources.

[0075] Specifically, as Figure 4 shown, the shared quota supports 2 jobs, and the exclusive quota supports 5 jobs. Evict one job occupying the shared quota so that the exclusive quota is less than M. At this time, when a user submits a job suddenly, by evicting the job occupying the shared quota, it can ensure the normal operation of the own jobs of the target sub-cluster.

[0076] In step S302, when the own job of the target sub-cluster is completed, release the resource occupation amount of the own job of the target sub-cluster, so that the exclusive quota decreases and the shared quota increases.

[0077] Specifically, as Figure 5 shown, when the user job with the exclusive quota in the target sub-cluster runs to completion, the exclusive quota of this user decreases, and the shared quota of other users increases. More jobs of other users can run in this sub-cluster to improve the resource utilization of the target sub-cluster. The above steps S301 - step S302 have no order.

[0078] Therefore, the embodiment of the present disclosure executes the above steps S301-S302. When a user submits a job suddenly, the job occupying the shared quota is expelled to ensure the normal operation of the target subcluster's own jobs. When the target subcluster's own jobs are executed, the resource occupancy of the target subcluster's own jobs is released, thereby enhancing the resource utilization rate of the shared quota, thereby improving the resource utilization rate in the multi-cluster scenario.

[0079] In some optional implementations, the above step S103, allocating the to-be-scheduled jobs according to the first job queue and the second job queue, includes:

[0080] Step a1, obtaining the job type corresponding to the job to be scheduled.

[0081] Step a2: if the job type of the job to be scheduled is a preemptible job, the job to be scheduled is allocated to the first job queue.

[0082] Step a3: if the job type of the job to be scheduled is a job that cannot be preempted, the job to be scheduled is allocated to the second job queue.

[0083] Specifically, the number of jobs to be scheduled can be one or more. The multiple jobs to be scheduled placed in the first job queue are preemptible jobs. This type of job can be run in the subcluster where the user is located, or in other subclusters, and the job can tolerate eviction operations caused by insufficient resources in other subclusters; the multiple jobs to be scheduled placed in the second job queue are non-preemptible jobs. This type of job runs in the subcluster where the user to be scheduled is located.

[0084] The embodiment of the present disclosure places corresponding types of to-be-scheduled jobs in the first job queue and the second job queue respectively, which is conducive to ensuring the normal and orderly operation of the business of the target sub-cluster itself, and is also conducive to sharing the jobs of the own and other sub-clusters.

[0085] In this embodiment, a multi-cluster job scheduling method is provided, which can be used for servers and mobile terminals, such as mobile phones and tablet computers. Figure 6 is a flowchart of a multi-cluster job scheduling method according to an embodiment of the present invention. Figure 6 As shown, step S104, based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster, schedule the job to be scheduled to the corresponding target subcluster. The process includes the following steps:

[0086] Step S601: based on the quota configuration result of the target subcluster, obtain the current remaining amount of the target subcluster's shared quota.

[0087] Step S602 : selecting a target subcluster corresponding to the job to be scheduled according to the sorting result from largest to smallest of the current remaining amount of the shared quota.

[0088] Step S603: Obtain the idle resource amount of the shared quota of the target subcluster corresponding to the job to be scheduled.

[0089] Step S604: determine whether the idle resource amount of the shared quota is sufficient to share the current resource amount of the job to be scheduled.

[0090] Step S605: If yes, based on the job allocation result of the job to be scheduled, schedule the job to be scheduled to the shared quota of the target subcluster.

[0091] Step S606: If not, continue to determine whether the next target sub-cluster among the multiple sub-clusters is sufficient to share the current resource amount of the job to be scheduled.

[0092] In a specific example, the to-be-scheduled jobs in the first job queue and the second job queue are both scheduled according to the above steps S601 to S606.

[0093] Therefore, the embodiment of the present disclosure is conducive to ensuring the full and fair use of resources by executing the above steps S601-S606, and is conducive to timely understanding of quota allocation in actual use. While smoothly handling its own business, it can also share the usage resources of other sub-groups, thereby improving the overall resource utilization rate.

[0094] Based on the same concept, the present disclosure discloses a multi-cluster job scheduling system, such as Figure 7 As shown, it includes: a main cluster 71 and multiple sub-clusters 72, wherein the main cluster 71 includes: an interface module 710, a scheduling module 711, a quota management module 712 and a data routing module 713. The job to be scheduled initiated by the user is obtained through the interface module.

[0095] The resource information of the target subcluster is obtained through the data routing module; the resource information includes: the total amount of resources of the target subcluster, the amount of resources occupied by the currently running jobs, the amount of resources occupied by the jobs to be run, and the amount of occupied resources, and the target subcluster is each subcluster in the multiple subclusters; the exclusive quota and the shared quota of the target subcluster are configured through the quota management module based on at least one of the total amount of resources of the target subcluster, the amount of resources occupied by the currently running jobs, the amount of resources occupied by the jobs to be run, and the amount of occupied resources; wherein the exclusive quota and the shared quota are divided based on the current quota of the target subcluster; the scheduling module allocates the jobs to be scheduled according to the first job queue and the second job queue; based on the job allocation result of the jobs to be scheduled and the quota configuration result of the target subcluster, the jobs to be scheduled are scheduled to the corresponding target subcluster.

[0096] Specifically, the Master Cluster, MC for short, is used to receive / respond to user requests and is responsible for managing subclusters. The Master Cluster itself does not run user jobs. Sub Clusters, SC for short, are clusters that can run user jobs and are managed by the Master Cluster. Users are the owners of subclusters.

[0097] The interface module mentioned above is the access entry for multi-cluster resources and the only entry for the entire system operation. It stores data based on the ETCD cluster, is responsible for the authentication / authorization / verification of various resources, and provides external RESTful APIs. In implementation, the native kubernetes api-server can be used, which is also the interface for users to submit / update / find / delete jobs. The scheduling module is responsible for distributing user jobs to appropriate subclusters for operation based on the subcluster information (such as cluster idle resources, cluster affinity requirements, etc.) fed back by the data routing module and the user quota information fed back by the quota management module. The data routing module is responsible for data transmission between subclusters and host groups, including: collecting subcluster information (cluster resources, job status, etc.) for decision-making by the scheduling module and the quota management module; monitoring the job status and feeding it back to the scheduling module; routing the user's operation to the corresponding subcluster and feeding back the result to the user; routing the job expulsion operation sent by the quota management module; distributing the scheduled jobs to the corresponding cluster, etc. The quota management module dynamically calculates and manages the resource usage quotas of users in each subcluster based on the subcluster information provided by the data routing module, expels running jobs outside the quota, and provides dynamic quota information for the scheduling module to make decisions.

[0098] In the multi-cluster job scheduling system of the embodiment of the present disclosure, the sub-cluster registers metadata information (access address, access signaling, user, etc.) to the data routing module, the data routing module actively connects to the sub-cluster, obtains cluster information (cluster resources, job status, user information, etc.) and caches it, the quota management module obtains all sub-cluster information from the data routing module, and dynamically calculates the available resource quota of each user in each sub-cluster. If the amount of resources used by the user exceeds its current quota, the quota management module will expel part of the user's job in the sub-cluster through the data routing module to ensure that its resource amount is less than the quota, the scheduling module allocates a suitable sub-cluster to the job submitted by the user according to the data provided by the data routing module and the quota management module, and notifies the data routing module of the scheduling result, after the data routing module receives the job expulsion command of the quota management module, it will find the corresponding sub-cluster, stop the corresponding job, and then notify the scheduling module and the quota management module of the result, after the data routing module receives the scheduling result of the scheduling module, it will actually distribute the job to the corresponding sub-cluster for operation, and notify the scheduling module of the distribution result.

[0099] In this embodiment, a multi-cluster job scheduling device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0100] This embodiment provides a multi-cluster job scheduling device, such as Figure 8 As shown, including:

[0101] The acquisition unit 81 is used to acquire the resource information of the job to be scheduled and the target sub-cluster sent by the user; the resource information includes: the total resources of the target sub-cluster, the resources occupied by the currently running job, the resources occupied by the job to be run, and the occupied resources. The target sub-cluster is each sub-cluster in the multiple sub-clusters;

[0102] The configuration unit 82 is used to configure the exclusive quota and the shared quota of the target subcluster based on at least one of the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources; wherein the exclusive quota and the shared quota are divided based on the current quota of the target subcluster;

[0103] An allocating unit 83, configured to allocate the jobs to be scheduled according to the first job queue and the second job queue;

[0104] The scheduling unit 84 is configured to schedule the job to be scheduled to the corresponding target subcluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster.

[0105] In some optional implementations, the configuration unit 82 includes:

[0106] The first configuration subunit is used to determine that the exclusive quota is the total amount of resources and the shared quota is zero when the sum of the amount of resources occupied by the running job and the amount of resources occupied by the job to be run is greater than or equal to the total amount of resources;

[0107] The second configuration subunit is used to determine that the exclusive quota is the total amount of resources and the shared quota is zero when the sum of the amount of resources occupied by the running job and the amount of resources occupied by the job to be run is less than the total amount of resources, and the sum of the amount of resources occupied by the running job, the amount of resources occupied by the job to be run and the reserved amount is greater than the total amount of resources;

[0108] The third configuration subunit is used to determine that the exclusive quota is the sum of the resources occupied by the running jobs, the resources occupied by the jobs to be run, and the reserved resources, and the shared quota is the difference between the total resources and the sum, when the sum of the resources occupied by the running jobs, the resources occupied by the jobs to be run, and the reserved resources is less than the total resources.

[0109] In some optional implementations, the configuration unit 82 further includes:

[0110] A fourth configuration subunit is configured to evict at least one job occupying the shared quota when a new job is added to the target subcluster, so that the exclusive quota is less than the occupied resource amount;

[0111] The fifth configuration subunit is used to release the resource usage of the target subcluster's own jobs when the target subcluster's own jobs are executed, so that the exclusive quota is reduced and the shared quota is increased.

[0112] In some optional implementations, the allocating unit 83 includes:

[0113] The acquisition subunit is used to obtain the job type corresponding to the job to be scheduled;

[0114] A first allocation subunit, configured to allocate the to-be-scheduled job to a first job queue if the job type of the to-be-scheduled job is a preemptible job;

[0115] The second allocating subunit is used to allocate the to-be-scheduled job to the second job queue if the job type of the to-be-scheduled job is a non-preemptible job.

[0116] In some optional implementations, the scheduling unit 84 includes:

[0117] A first acquisition subunit is used to acquire the current remaining amount of the target subcluster's shared quota based on the quota configuration result of the target subcluster;

[0118] The cluster selection subunit is used to select the target subcluster corresponding to the job to be scheduled according to the sorting result of the current remaining amount of the shared quota from large to small;

[0119] The second acquisition subunit is used to acquire the free resource amount of the shared quota of the target subcluster corresponding to the job to be scheduled;

[0120] A first judgment subunit is used to judge whether the idle resource amount of the shared quota is sufficient to share the current resource amount of the job to be scheduled;

[0121] The job scheduling subunit is used for, if yes, scheduling the job to be scheduled into the shared quota of the target subcluster based on the job allocation result of the job to be scheduled;

[0122] The second judgment subunit is used to, if not, continue to judge whether the next target subcluster among the multiple subclusters is sufficient to share the current resource amount of the job to be scheduled.

[0123] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0124] The multi-cluster job scheduling device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0125] An embodiment of the present invention further provides a computer device having the above-mentioned multi-cluster job scheduling device.

[0126] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0127] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0128] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0129] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0131] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0132] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0133] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0134] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multi-cluster job scheduling method, characterized in that: The method comprises: Obtain resource information of the job to be scheduled and the target subcluster sent by the user; the resource information includes: the total amount of resources of the target subcluster, the amount of resources occupied by the currently running job, the amount of resources occupied by the job to be run, and the amount of occupied resources. The target subcluster is each subcluster in the multiple subclusters; Based on at least one of the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources, configure the exclusive quota and the shared quota of the target subcluster; wherein the exclusive quota and the shared quota are divided based on the current quota of the target subcluster; Allocate the to-be-scheduled jobs according to the first job queue and the second job queue; Based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster, the job to be scheduled is scheduled to the corresponding target subcluster.

2. The method according to claim 1, characterized in that Based on at least one of the total resources of the target subcluster, the resources occupied by currently running jobs, the resources occupied by jobs to be run, and the occupied resources, the exclusive quota and shared quota of the target subcluster are configured, including: When the sum of the amount of resources occupied by the running job and the amount of resources occupied by the job to be run is greater than or equal to the total amount of resources, the exclusive quota is determined to be the total amount of resources, and the shared quota is zero; When the sum of the amount of resources occupied by the running job and the amount of resources occupied by the job to be run is less than the total amount of resources, and the sum of the amount of resources occupied by the running job, the amount of resources occupied by the job to be run and the reserved amount is greater than the total amount of resources, the exclusive quota is determined to be the total amount of resources, and the shared quota is zero; When the sum of the amount of resources occupied by the running job, the amount of resources occupied by the job to be run, and the reserved amount of resources is less than the total amount of resources, the exclusive quota is determined to be the sum of the amount of resources occupied by the running job, the amount of resources occupied by the job to be run, and the reserved amount of resources, and the shared quota is the difference between the total amount of resources and the sum.

3. The method according to claim 1 or 2, characterized in that: Based on at least one of the total resources of the target subcluster, the resources occupied by currently running jobs, the resources occupied by jobs to be run, and the occupied resources, the exclusive quota and shared quota of the target subcluster are configured, including: When a new job is added to the target subcluster, at least one job occupying the shared quota is evicted so that the exclusive quota is less than the occupied resource amount; When the target sub-cluster's own job is executed, the resource usage of the target sub-cluster's own job is released, so that the exclusive quota is reduced and the shared quota is increased.

4. The method according to claim 1, characterized in that: Allocating the to-be-scheduled jobs according to the first job queue and the second job queue includes: Obtaining the job type corresponding to the job to be scheduled; If the job type of the to-be-scheduled job is a preemptible job, allocating the to-be-scheduled job to the first job queue; If the job type of the job to be scheduled is a job that cannot be preempted, the job to be scheduled is allocated to the second job queue.

5. The method according to claim 1, characterized in that Scheduling the job to be scheduled to the corresponding target subcluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster includes: Based on the quota configuration result of the target subcluster, obtaining the current remaining amount of the shared quota of the target subcluster; Selecting the target subcluster corresponding to the job to be scheduled according to the sorting result from largest to smallest of the current remaining amount of the shared quota; Obtaining the amount of free resources of the shared quota of the target subcluster corresponding to the job to be scheduled; Determine whether the idle resource amount of the shared quota is sufficient to share the current resource amount of the job to be scheduled; If so, scheduling the job to be scheduled into the shared quota of the target subcluster based on the job allocation result of the job to be scheduled; If not, continue to determine whether the next target sub-cluster among the multiple sub-clusters is sufficient to share the current resource amount of the job to be scheduled.

6. A multi-cluster job scheduling device, characterized in that: The device comprises: An acquisition unit is used to acquire resource information of the job to be scheduled and the target subcluster sent by the user; the resource information includes: the total amount of resources of the target subcluster, the amount of resources occupied by the currently running job, the amount of resources occupied by the job to be run, and the amount of occupied resources. The target subcluster is each subcluster in the multiple subclusters; A configuration unit, configured to configure the exclusive quota and the shared quota of the target subcluster based on at least one of the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources; wherein the exclusive quota and the shared quota are divided based on the current quota of the target subcluster; An allocating unit, configured to allocate the to-be-scheduled jobs according to the first job queue and the second job queue; The scheduling unit is configured to schedule the job to be scheduled to the corresponding target subcluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster.

7. A multi-cluster job scheduling system, characterized in that: include: A main cluster and multiple sub-clusters, wherein the main cluster includes: an interface module, a scheduling module, a quota management module and a data routing module, Acquire the job to be scheduled initiated by the user through the interface module; Obtain resource information of the target subcluster through the data routing module; the resource information includes: the total amount of resources of the target subcluster, the amount of resources occupied by the currently running jobs, the amount of resources occupied by the jobs to be run, and the amount of occupied resources, and the target subcluster is each subcluster in the multiple subclusters; The quota management module configures the exclusive quota and shared quota of the target subcluster based on at least one of the total resources of the target subcluster, the resources occupied by the currently running jobs, the resources occupied by the jobs to be run, and the occupied resources; wherein the exclusive quota and the shared quota are divided based on the current quota of the target subcluster; The scheduling module allocates the job to be scheduled according to the first job queue and the second job queue; and schedules the job to be scheduled to the corresponding target subcluster based on the job allocation result of the job to be scheduled and the quota configuration result of the target subcluster.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-cluster job scheduling method according to any one of claims 1 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multi-cluster job scheduling method according to any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the multi-cluster job scheduling method according to any one of claims 1 to 5.

Images