Cluster resource scheduling method and device

Abstract

The invention provides a cluster resource scheduling method and device, and relates to the field of cloud computing platforms, in particular to the field of resource scheduling. According to the specific implementation scheme, a to-be-scheduled task currently created by a first user is obtained, and the resource request quantity of the to-be-scheduled task is obtained; a resource view list corresponding to the cluster pool at the current moment is obtained, and the resource view list is used for recording resource related parameters corresponding to each task queue contained in each cluster located in the cluster pool; determining a target task queue and a target to-be-scheduled resource from a cluster pool according to the resource view list; and scheduling the target to-be-scheduled resource to execute the to-be-scheduled task based on the target task queue.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing platforms, in particular to the field of resource scheduling, and specifically to a cluster resource scheduling method and device thereof. Background Art

[0002] In modern cloud computing environments, with the continuous growth of application loads, cluster management faces the problem of resource allocation and utilization efficiency. In related technologies, resources of other idle queues in the cluster are usually reused to achieve dynamic resource allocation and recycling, thereby improving the utilization of resources in the cluster. However, although this mechanism performs well in a single cluster environment, it still has limitations in multi-cluster resource management. Therefore, it is urgent to propose new solutions for multi-cluster resource management to optimize the overall resource allocation and service quality. Summary of the invention

[0003] The present disclosure provides a cluster resource scheduling method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, a cluster resource scheduling method is provided, which comprises obtaining a to-be-scheduled task currently created by a first user, and obtaining a resource request amount of the to-be-scheduled task; obtaining a resource view list corresponding to a cluster pool at a current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue contained in each cluster in the cluster pool; determining a target task queue and a target to-be-scheduled resource from the cluster pool according to the resource view list; and scheduling the target to-be-scheduled resource to execute the to-be-scheduled task based on the target task queue.

[0005] By obtaining a list of resource views for each cluster, this application can understand the resource status of each cluster from a global perspective, thereby making more reasonable scheduling decisions to avoid resource overload in one cluster while other clusters are idle; by obtaining the resource request amount of the task to be scheduled in real time, combined with the resource view of the cluster pool, resource allocation can be flexibly adjusted; in a multi-cluster environment, resources can be dynamically scheduled according to load changes, providing good elastic expansion capabilities.

[0006] According to one embodiment of the present application, a target task queue and a target resource to be scheduled are determined from a cluster pool according to a resource view list, including: obtaining first resource-related parameters related to a first user from the resource view list, wherein the first resource-related parameters correspond one-to-one to the first task queue related to the first user; judging whether the queue idle resources corresponding to each first task queue meet the resource request amount according to the first resource-related parameters; in response to the existence of at least one first task queue whose queue idle resources meet the resource request amount, using the first task queue as a candidate first task queue; determining a target task queue from the candidate first task queues, and determining a target resource to be scheduled from the queue idle resources corresponding to the target task queue.

[0007] According to one embodiment of the present application, based on the first resource-related parameters, determining whether the queue idle resources corresponding to each first task queue meet the resource request amount includes: based on the first resource-related parameters, obtaining a first difference between the current quota value and the currently used resource value recorded in each first resource-related parameter; comparing the first difference with the resource request amount to obtain a first comparison result; in response to the first comparison result corresponding to any first resource-related parameter indicating that the first difference is greater than or equal to the resource request amount, determining that the queue idle resources of the first task queue corresponding to the first resource-related parameter meet the resource request amount; in response to the first comparison result corresponding to any first resource-related parameter indicating that the first difference is less than the resource request amount, determining that the queue idle resources of the first task queue corresponding to the first resource-related parameter do not meet the resource request amount.

[0008] According to one embodiment of the present application, the cluster resource scheduling method also includes: in response to the queue idle resources of each first task queue not satisfying the resource request amount, judging whether each first task queue is eligible to preempt resources according to the first resource-related parameters; taking the cluster where the first task queues that are eligible to preempt resources are located as a candidate cluster, and determining the target cluster from the candidate clusters; obtaining the cluster idle resource amount of the target cluster at the current moment according to the resource view list; judging whether the cluster idle resource amount satisfies the resource request amount; in response to the cluster idle resource amount satisfying the resource request amount, performing resource preemption on the cluster idle resources of the target cluster to determine the target resources to be scheduled, and taking the first task queue belonging to the target cluster as the target task queue.

[0009] According to one embodiment of the present application, whether each first task queue is qualified to preempt resources is determined based on the first resource-related parameters, including: based on the first resource-related parameters, obtaining a second difference between the maximum allowed resource value and the currently used resource value recorded in each first resource-related parameter; comparing the second difference with the resource request amount to obtain a second comparison result; in response to any second comparison result indicating that the second difference is greater than or equal to the resource request amount, determining that the first task queue corresponding to the second comparison result is qualified to preempt resources; in response to any second comparison result indicating that the second difference is less than the resource request amount, determining that the first task queue corresponding to the second comparison result is not qualified to preempt resources.

[0010] According to one embodiment of the present application, determining whether the amount of idle resources in the cluster meets the amount of resource requests includes: comparing the amount of idle resources in the cluster with the amount of resource requests to obtain a third comparison result; in response to the third comparison result indicating that the amount of idle resources in the cluster is greater than or equal to the amount of resource requests, determining that the amount of idle resources in the cluster meets the amount of resource requests; in response to the third comparison result indicating that the amount of idle resources in the cluster is less than the amount of resource requests, determining that the amount of idle resources in the cluster does not meet the amount of resource requests.

[0011] According to one embodiment of the present application, the cluster resource scheduling method also includes: in response to the amount of idle resources in the cluster not satisfying the amount of resource requests, comparing the current quota value recorded in the first resource-related parameter corresponding to the first task queue located in the target cluster with the minimum allowed resource value to obtain a fourth comparison result; in response to the fourth comparison result indicating that the current quota value is less than the minimum allowed resource value, obtaining a preempted state task queue in the target cluster that is in a resource preempted state; determining the task to be deleted from the running tasks running in the preempted state task queue; pausing the running of the task to be deleted and deleting it from the target cluster; using the first task queue belonging to the target cluster as the target task queue, and determining the target resource to be scheduled based on the idle resources vacated after the task to be deleted is deleted.

[0012] According to one embodiment of the present application, tasks to be deleted are determined from running tasks running in a preempted state task queue, including: obtaining a third difference between a resource request amount and an idle resource amount of the cluster; obtaining a task running resource value, a task priority, and a task running time of each running task running in the preempted state task queue; and determining tasks to be deleted from the running tasks based on the task running resource value, the task priority, and the task running time in combination with the third difference.

[0013] According to an embodiment of the present application, after the target to-be-scheduled resource is scheduled to execute the to-be-scheduled task based on the target task queue, the method further includes: updating resource-related parameters corresponding to the target cluster to which the target task queue belongs.

[0014] According to an embodiment of the present application, the cluster resource scheduling method further includes: in response to a fourth comparison result indicating that the current quota value is greater than or equal to the minimum allowed resource value, placing the to-be-scheduled task in a task waiting list.

[0015] According to an embodiment of the present application, the cluster resource scheduling method further includes: in response to each first task queue not being qualified to preempt resources, placing the to-be-scheduled task in a task waiting list.

[0016] According to one embodiment of the present application, determining a target task queue from candidate first task queues includes: obtaining the priority of the cluster to which each candidate first task queue belongs, and taking the candidate first task queue corresponding to the cluster with the highest priority as the target task queue; or, randomly selecting a candidate first task queue from the candidate first task queues as the target task queue; or, comparing the first difference values ​​corresponding to each candidate first task queue, and taking the candidate first task queue corresponding to the first difference value with the largest value as the target task queue.

[0017] According to one embodiment of the present application, the cluster resource scheduling method also includes: obtaining the second user corresponding to the task to be deleted; obtaining the second resource-related parameters of the second task queue corresponding to the second user in other clusters from the resource view list; judging whether the queue idle resources corresponding to each second task queue meet the second resource request amount corresponding to the task to be deleted according to the second resource-related parameters; in response to the existence of at least one second task queue whose queue idle resources meet the second resource request amount, executing the task to be deleted based on the second task queue that meets the second resource request amount; in response to the queue idle resources of the second task queue not meeting the second resource request amount, placing the task to be deleted in the task waiting list.

[0018] According to another aspect of the present disclosure, a cluster resource scheduling device is provided, including: a first acquisition module, used to obtain a to-be-scheduled task currently created by a first user, and to obtain a resource request amount of the to-be-scheduled task; a second acquisition module, used to obtain a resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue contained in each cluster in the cluster pool; a determination module, used to determine a target task queue and a target to-be-scheduled resource from the cluster pool according to the resource view list; and an execution module, used to schedule the target to-be-scheduled resource to execute the to-be-scheduled task based on the target task queue.

[0019] According to one embodiment of the present application, the determination module is also used to: obtain first resource-related parameters related to the first user from the resource view list, wherein the first resource-related parameters correspond one-to-one to the first task queue related to the first user; determine whether the queue idle resources corresponding to each first task queue meet the resource request amount according to the first resource-related parameters; in response to the existence of at least one first task queue whose queue idle resources meet the resource request amount, use the first task queue as a candidate first task queue; determine the target task queue from the candidate first task queues, and determine the target to-be-scheduled resources from the queue idle resources corresponding to the target task queue.

[0020] According to one embodiment of the present application, the determination module is also used to: obtain, based on the first resource-related parameters, a first difference between the current quota value recorded in each first resource-related parameter and the currently used resource value; compare the first difference with the resource request amount to obtain a first comparison result; in response to the first comparison result corresponding to any first resource-related parameter indicating that the first difference is greater than or equal to the resource request amount, determine that the queue idle resources of the first task queue corresponding to the first resource-related parameter meet the resource request amount; in response to the first comparison result corresponding to any first resource-related parameter indicating that the first difference is less than the resource request amount, determine that the queue idle resources of the first task queue corresponding to the first resource-related parameter do not meet the resource request amount.

[0021] According to one embodiment of the present application, the determination module is also used for: in response to the queue idle resources of each first task queue not satisfying the resource request amount, judging whether each first task queue is eligible to preempt resources according to the first resource-related parameters; taking the cluster where the first task queues that are eligible to preempt resources are located as a candidate cluster, and determining the target cluster from the candidate clusters; obtaining the cluster idle resource amount of the target cluster at the current moment according to the resource view list; judging whether the cluster idle resource amount satisfies the resource request amount; in response to the cluster idle resource amount satisfying the resource request amount, performing resource preemption on the cluster idle resources of the target cluster to determine the target resources to be scheduled, and taking the first task queue belonging to the target cluster as the target task queue.

[0022] According to one embodiment of the present application, the determination module is also used to: obtain, based on the first resource-related parameters, a second difference between the maximum allowed resource value and the currently used resource value recorded in each first resource-related parameter; compare the second difference with the resource request amount to obtain a second comparison result; in response to any second comparison result indicating that the second difference is greater than or equal to the resource request amount, determine that the first task queue corresponding to the second comparison result is eligible to preempt resources; in response to any second comparison result indicating that the second difference is less than the resource request amount, determine that the first task queue corresponding to the second comparison result is not eligible to preempt resources.

[0023] According to one embodiment of the present application, the determination module is further used to: compare the amount of idle resources in the cluster and the amount of resource requests to obtain a third comparison result; in response to the third comparison result indicating that the amount of idle resources in the cluster is greater than or equal to the amount of resource requests, determine that the amount of idle resources in the cluster meets the amount of resource requests; in response to the third comparison result indicating that the amount of idle resources in the cluster is less than the amount of resource requests, determine that the amount of idle resources in the cluster does not meet the amount of resource requests.

[0024] According to one embodiment of the present application, the determination module is also used to: in response to the amount of idle resources in the cluster not satisfying the amount of resource requests, compare the current quota value recorded in the first resource-related parameter corresponding to the first task queue in the target cluster with the minimum allowed resource value to obtain a fourth comparison result; in response to the fourth comparison result indicating that the current quota value is less than the minimum allowed resource value, obtain a preempted state task queue in the target cluster that is in a resource preempted state; determine the task to be deleted from the running tasks running in the preempted state task queue; suspend the running of the task to be deleted and delete it from the target cluster; use the first task queue in the target cluster as the target task queue, and determine the target resources to be scheduled based on the idle resources vacated after the task to be deleted is deleted.

[0025] According to one embodiment of the present application, the determination module is also used to: obtain a third difference between the resource request amount and the cluster idle resource amount; obtain the task running resource value, task priority and task running time of each running task running in the preempted state task queue; determine the task to be deleted from the running tasks based on the task running resource value, task priority and task running time, combined with the third difference.

[0026] According to an embodiment of the present application, the determination module is further used to: update resource-related parameters corresponding to the target cluster to which the target task queue belongs.

[0027] According to an embodiment of the present application, the determination module is further used to: in response to the fourth comparison result indicating that the current quota value is greater than or equal to the minimum allowed resource value, place the to-be-scheduled task in a task waiting list.

[0028] According to an embodiment of the present application, the determination module is further used to: in response to each first task queue not being eligible to preempt resources, place the to-be-scheduled task in a task waiting list.

[0029] According to one embodiment of the present application, the determination module is also used to: obtain the priority of the cluster to which each candidate first task queue belongs, and use the candidate first task queue corresponding to the cluster with the highest priority as the target task queue; or, randomly select a candidate first task queue from the candidate first task queues as the target task queue; or, compare the first difference values ​​corresponding to each candidate first task queue, and use the candidate first task queue corresponding to the first difference value with the largest value as the target task queue.

[0030] According to one embodiment of the present application, the determination module is also used to: obtain the second user corresponding to the task to be deleted; obtain the second resource-related parameters of the second task queue corresponding to the second user in other clusters from the resource view list; determine whether the queue idle resources corresponding to each second task queue meet the second resource request amount corresponding to the task to be deleted based on the second resource-related parameters; in response to the existence of at least one second task queue whose queue idle resources meet the second resource request amount, execute the task to be deleted based on the second task queue that meets the second resource request amount; in response to the queue idle resources of the second task queues not meeting the second resource request amount, place the task to be deleted in the task waiting list.

[0031] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned cluster resource scheduling method.

[0032] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above cluster resource scheduling method.

[0033] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above cluster resource scheduling method when executed by a processor.

[0034] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0036] Figure 1 It is a schematic diagram of an exemplary implementation of a cluster resource scheduling method according to an exemplary embodiment of the present disclosure.

[0037] Figure 2 The diagram is a schematic diagram of a cluster pool and a resource view list corresponding to the cluster pool according to an exemplary embodiment of the present disclosure.

[0038] Figure 3 The diagram is a schematic diagram of a cluster pool and a resource view list corresponding to the cluster pool according to an exemplary embodiment of the present disclosure.

[0039] Figure 4It is a schematic diagram of a maximum allowed resource value of a task queue according to an exemplary embodiment of the present disclosure.

[0040] Figure 5 It is a schematic diagram of an exemplary implementation of a cluster resource scheduling method according to an exemplary embodiment of the present disclosure.

[0041] Figure 6 It is a schematic diagram of an exemplary implementation of a cluster resource scheduling method according to an exemplary embodiment of the present disclosure.

[0042] Figure 7 It is a schematic diagram of an exemplary implementation of a cluster resource scheduling method according to an exemplary embodiment of the present disclosure.

[0043] Figure 8 is a schematic diagram of a cluster resource scheduling device according to an exemplary embodiment of the present disclosure.

[0044] Fig. 9 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0046] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0047] Figure 1 is a schematic diagram of an exemplary implementation of a cluster resource scheduling method shown in the present application, such as Figure 1 As shown, the cluster resource scheduling method includes the following steps:

[0048] S101, obtaining a task to be scheduled currently created by a first user, and obtaining a resource request amount of the task to be scheduled.

[0049] In the present application, after receiving a task to be scheduled, the user who created the task to be scheduled is regarded as the first user, and the resource request amount of the task to be scheduled is obtained.

[0050] The resource request amount refers to the amount of resources required to execute the task to be scheduled, wherein the resources may be resources such as a graphics processing unit (GPU), a central processing unit (CPU), etc.

[0051] S102, obtaining a resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue included in each cluster in the cluster pool.

[0052] It is not difficult to understand that a cluster pool can contain multiple clusters, each cluster can contain multiple task queues, and the same user can have task queues in one or more clusters in the cluster pool.

[0053] This application proposes a concept of resource view list on multiple clusters. For ease of understanding, Figure 2 is a schematic diagram of a cluster pool and a resource view list corresponding to the cluster pool shown in the present application. For example, the cluster pool includes 3 clusters, and user 1 and user 2 each have a corresponding task queue in the 3 clusters. Figure 2 As shown, user 1 corresponds to task queue 1-1, task queue 2-1, and task queue 3-1 in cluster 1, cluster 2, and cluster 3 respectively; user 2 corresponds to task queue 1-2, task queue 2-2, and task queue 3-2 in cluster 1, cluster 2, and cluster 3 respectively. The resource view list is used to record the resource-related parameters corresponding to each of the six task queues. Each task queue corresponds to one resource-related parameter in the resource view list, that is, the task queues correspond to the resource-related parameters in the resource view list one by one.

[0054] Figure 3 : is a schematic diagram of a cluster pool and a resource view list corresponding to the cluster pool shown in the present application. For example, the cluster pool includes 3 clusters, and user 1, user 2, and user 3 each have a corresponding task queue in the 3 clusters. Figure 3 As shown, user 1 corresponds to task queue 1-1, task queue 2-1, and task queue 3-1 in cluster 1, cluster 2, and cluster 3 respectively; user 2 corresponds to task queue 1-2, task queue 2-2, and task queue 3-2 in cluster 1, cluster 2, and cluster 3 respectively; user 3 corresponds to task queue 1-3, task queue 2-3, and task queue 3-3 in cluster 1, cluster 2, and cluster 3 respectively. The resource view list is used to record the resource-related parameters corresponding to each of the nine task queues. Each task queue corresponds to one resource-related parameter in the resource view list, that is, the task queues correspond to the resource-related parameters in the resource view list one by one.

[0055] Among them, in this application, each resource-related parameter may include the following five fields:

[0056] 1. Minimum allowed resource value (MIN), which is used to indicate the minimum number of resources that need to be guaranteed if the task queue corresponding to the resource-related parameter has enough tasks to run. The sum of the minimum allowed resource values ​​corresponding to all task queues contained in the same cluster is equal to the total number of resources contained in the cluster.

[0057] 2. The maximum allowed resource value (MAX) is used to indicate the maximum amount of resources allowed when the task queue corresponding to the resource-related parameter operates in excess.

[0058] 3. The current quota value is used to indicate the current resource quota of the task queue corresponding to the resource-related parameter. It will change dynamically when resource preemption and resource recycling occur. The initial value of the current quota value is the minimum allowed resource value contained in the resource-related parameter.

[0059] 4. The currently used resource value is used to indicate the current amount of used resources of the task queue corresponding to the resource-related parameters.

[0060] 5. The identification field is used to indicate the cluster and user corresponding to the resource-related parameters.

[0061] For ease of understanding, Figure 4 is a schematic diagram of the maximum allowed resource value of a task queue shown in this application, such as Figure 4 As shown in the figure, the amount of resources that can be expanded for the task queue is between the minimum allowed resource value and the maximum allowed resource value. For example, if the minimum allowed resource value of a task queue is 6 and the maximum allowed resource value is 10, that is, when the task queue has tasks to be scheduled, if the 6 resources corresponding to the task queue are not enough, at most 4 resources can be temporarily seized from the resources corresponding to other queues in the cluster.

[0062] The capacity of each task queue is set to the maximum allowed resource value in the resource-related parameters corresponding to the task queue.

[0063] S103: Determine a target task queue and a target resource to be scheduled from the cluster pool according to the resource view list.

[0064] Since the resource view list clearly records the current resource parameters of each task queue, in this application, the target task queue and the target resources to be scheduled are determined from the cluster pool according to the resource view list.

[0065] S104, scheduling the target resources to be scheduled to execute the tasks to be scheduled based on the target task queue.

[0066] The embodiment of the present application proposes a cluster resource scheduling method, and the specific scheme is: obtain the to-be-scheduled task currently created by the first user, and obtain the resource request amount of the to-be-scheduled task; obtain the resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record the resource-related parameters corresponding to each task queue contained in each cluster in the cluster pool; determine the target task queue and the target to-be-scheduled resource from the cluster pool according to the resource view list; and schedule the target to-be-scheduled resource based on the target task queue to execute the to-be-scheduled task. By obtaining the resource view list of each cluster, the present application can understand the resource status of each cluster from a global perspective, thereby making more reasonable scheduling decisions to avoid overloading of a cluster resource while other clusters are idle; obtain the resource request amount of the to-be-scheduled task in real time, and flexibly adjust the resource allocation in combination with the resource view of the cluster pool; in a multi-cluster environment, it can dynamically schedule resources according to load changes, providing good elastic expansion capabilities.

[0067] Figure 5 is a schematic diagram of an exemplary implementation of a cluster resource scheduling method shown in the present application, such as Figure 5 As shown, the cluster resource scheduling method includes the following steps:

[0068] S501, obtaining a task to be scheduled currently created by a first user, and obtaining a resource request amount of the task to be scheduled.

[0069] S502: Obtain a resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue included in each cluster in the cluster pool.

[0070] Regarding the specific implementation of steps S501 to S502, reference may be made to the specific introduction of the relevant parts in the above embodiments, which will not be elaborated here.

[0071] S503: Acquire first resource-related parameters related to the first user from the resource view list, wherein the first resource-related parameters correspond one-to-one to the first task queue related to the first user.

[0072] by Figure 2 For example, if the first user is user 1, the resource related parameters corresponding to task queue 1-1, task queue 2-1, and task queue 3-1 related to user 1 in the resource view list are determined according to the identification field as the first resource related parameters.

[0073] by Figure 3 For example, if the first user is user 3, the resource-related parameters corresponding to task queues 1-3, task queues 2-3, and task queues 3-3 related to user 3 in the resource view list are determined according to the identification field as the first resource-related parameters.

[0074] S504: Determine, based on the first resource-related parameters, whether the idle queue resources corresponding to each first task queue meet the resource request amount.

[0075] According to the first resource-related parameters, the difference between the current quota value recorded in each first resource-related parameter and the currently used resource value is obtained as the first difference; the first difference and the resource request amount are compared to obtain a first comparison result; if the first comparison result corresponding to any first resource-related parameter indicates that the first difference is greater than or equal to the resource request amount, it is determined that the queue idle resources of the first task queue corresponding to the first resource-related parameter meet the resource request amount; if the first comparison result corresponding to any first resource-related parameter indicates that the first difference is less than the resource request amount, it is determined that the queue idle resources of the first task queue corresponding to the first resource-related parameter do not meet the resource request amount.

[0076] Continue with Figure 2 For example, if the resource request amount is 1 resource and the first user is user 1, after determining the first resource-related parameter corresponding to the task queue 1-1 related to user 1, obtain the difference between the current quota value recorded by the first resource-related parameter corresponding to the task queue 1-1 and the currently used resource value. If the current quota value recorded by the first resource-related parameter corresponding to the task queue 1-1 is 6, and the currently used resource value is 2, that is, the first difference is 4. Since 4 is greater than 1 resource required by the resource request amount, it is determined that the queue idle resources of the task queue 1-1 meet the resource request amount.

[0077] Similarly, it is determined whether the idle resources of the task queue 2 - 1 and the task queue 3 - 1 related to the user 1 meet the resource request amount.

[0078] S505 : In response to the existence of at least one first task queue whose queue idle resources satisfy the resource request amount, the first task queue is selected as a candidate first task queue.

[0079] For example, continue with Figure 2 For example, if the first user is user 1, the queue idle resources of task queue 1-1 and task queue 2-1 related to user 1 meet the resource request amount, while the queue idle resources of task queue 3-1 do not meet the resource request amount, then task queue 1-1 and task queue 2-1 are used as candidate first task queues.

[0080] S506: Determine a target task queue from the candidate first task queues, and determine a target resource to be scheduled from the queue idle resources corresponding to the target task queue.

[0081] As an achievable manner, the priority of the cluster to which each candidate first task queue belongs is obtained, and the candidate first task queue corresponding to the cluster with the highest priority is used as the target task queue.

[0082] As another implementable manner, a candidate first task queue is randomly selected from the candidate first task queues as the target task queue.

[0083] As another achievable manner, the first difference values ​​corresponding to each candidate first task queue are compared, and the candidate first task queue corresponding to the first difference value with the largest value is taken as the target task queue.

[0084] For example, continue with Figure 2 For example, if the first user is user 1, if task queue 1-1 is determined to be the target task queue, since the idle resources of the target task queue itself can meet the resource request, the target resource to be scheduled is determined from the idle resources of the queue corresponding to the target task queue.

[0085] S507: Schedule the target to-be-scheduled resources to execute the to-be-scheduled tasks based on the target task queue.

[0086] The embodiment of the present application can optimize resource allocation and improve resource utilization by obtaining a cluster resource view; secondly, it dynamically determines whether the idle resources of the task queue meet the request, thereby ensuring efficient scheduling and reducing waiting time; in addition, if there are multiple candidate task queues, the method for determining the target task queue from them is flexible and diverse, such as priority selection and random selection, which helps to achieve the optimal scheduling strategy in different scenarios.

[0087] Furthermore, since resource scheduling has occurred as described above, it is necessary to update resource-related parameters corresponding to the target cluster to which the target task queue belongs.

[0088] It is not difficult to understand that the minimum allowed resource value, maximum allowed resource value and identification field in the resource-related parameters corresponding to each task queue generally do not change after being set. Therefore, in this application, the update of the current quota value and the currently used resource value contained in the resource-related parameters is focused on.

[0089] For example, continue with Figure 2For example, if the first user is user 1, and if task queue 1-1 is determined to be the target task queue, since there is no resource preemption or resource recovery in the above resource scheduling process, it is only necessary to update the current used resource value corresponding to task queue 1-1. For example, if the resource request amount is 1 resource, if the current quota value recorded in the first resource-related parameter corresponding to task queue 1-1 is 6, and the current used resource value is 2, then the current used resource value in the first resource-related parameter corresponding to task queue 1-1 is increased from 2 to 3 by 1, and the current quota value in the first resource-related parameter corresponding to task queue 1-1 remains unchanged at 6.

[0090] Figure 6 is a schematic diagram of an exemplary implementation of a cluster resource scheduling method shown in the present application, such as Figure 6 As shown, the cluster resource scheduling method includes the following steps:

[0091] S601, obtaining a task to be scheduled currently created by a first user, and obtaining a resource request amount of the task to be scheduled.

[0092] S602, obtaining a resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue included in each cluster in the cluster pool.

[0093] S603: Acquire first resource-related parameters related to the first user from the resource view list, wherein the first resource-related parameters correspond one-to-one to the first task queue related to the first user.

[0094] S604: Determine, based on the first resource-related parameters, whether the queue idle resources corresponding to each first task queue meet the resource request amount.

[0095] Regarding the specific implementation of steps S601 to S604, reference may be made to the specific introduction of the relevant parts in the above embodiments, which will not be elaborated here.

[0096] S605 , in response to the idle queue resources of each first task queue not satisfying the resource request amount, determining whether each first task queue is qualified to seize the resource according to first resource-related parameters.

[0097] If the idle resources of each first task queue do not meet the resource request amount, it means that the idle resources of the first task queue alone are not enough to schedule the task to be scheduled. At this time, it should be considered whether the idle resources of other users can be preempted so that the task to be scheduled can be executed.

[0098] When judging whether each first task queue is qualified to preempt resources, it is first necessary to obtain the second difference between the maximum allowed resource value and the currently used resource value recorded in each first resource-related parameter based on the first resource-related parameter; compare the second difference with the resource request amount to obtain a second comparison result; if any second comparison result indicates that the second difference is greater than or equal to the resource request amount, it is determined that the first task queue corresponding to the second comparison result is qualified to preempt resources; if any second comparison result indicates that the second difference is less than the resource request amount, it is determined that the first task queue corresponding to the second comparison result is not qualified to preempt resources.

[0099] S606: The cluster where the first task queue having the resource preemption qualification is located is taken as a candidate cluster, and a target cluster is determined from the candidate clusters.

[0100] If only one first task queue has the qualification to preempt resources, the cluster where the first task queue is located is directly used as the target cluster.

[0101] If there are multiple first task queues that are eligible to seize resources, it means there are multiple candidate clusters. Figure 2 For example, if the first user is user 1, if the idle resources of task queues 1-1, 2-1, and 3-1 related to user 1 do not meet the resource request amount, and after judgment, it is found that only task queues 1-1 and 2-1 are eligible to preempt resources, then the clusters where task queues 1-1 and 2-1 are located are used as candidate clusters, that is, Figure 2 Cluster 1 and cluster 2 in are selected as candidate clusters.

[0102] As an implementable manner, when determining the target cluster from the candidate clusters, the candidate cluster with the highest cluster priority may be used as the target cluster.

[0103] As another achievable manner, the amount of cluster idle resources of each candidate cluster may be first obtained, and the candidate cluster with the largest amount of cluster idle resources may be used as the target cluster.

[0104] As another achievable manner, a candidate cluster may be randomly designated from the candidate clusters as the target cluster.

[0105] S607: Obtain the current idle resource amount of the target cluster according to the resource view list.

[0106] by Figure 2Taking cluster 1 in as the target cluster as an example, the total resources contained in cluster 1 can be obtained (the sum of the minimum allowed resource values ​​in the resource-related parameters corresponding to all task queues in cluster 1), and then the sum of the currently used resource values ​​of all task queues in cluster 1 can be obtained according to the resource-related parameters corresponding to each task queue in cluster 1. The total resources contained in cluster 1 are subtracted from the sum of the currently used resource values ​​of all task queues in cluster 1 to obtain the cluster free resources of cluster 1 at the current moment.

[0107] S608: Determine whether the amount of idle resources in the cluster meets the resource request amount.

[0108] Compare the amount of idle resources in the cluster and the amount of resource requests to obtain a third comparison result; if the third comparison result indicates that the amount of idle resources in the cluster is greater than or equal to the amount of resource requests, determine that the amount of idle resources in the cluster meets the amount of resource requests; if the third comparison result indicates that the amount of idle resources in the cluster is less than the amount of resource requests, determine that the amount of idle resources in the cluster does not meet the amount of resource requests.

[0109] S609 , in response to the idle cluster resources satisfying the resource request amount, resource preemption is performed on the idle cluster resources of the target cluster to determine the target resources to be scheduled, and the first task queue belonging to the target cluster is used as the target task queue.

[0110] If the current idle resources of the target cluster meet the resource request, it means that resources can be obtained to schedule the tasks to be scheduled by preempting resources from other task queues in the target cluster. The preempted resources can be determined based on the resource request and the idle resources of the target task queue itself. For example, if the resource request is 3 resources and the target task queue itself has 2 idle resources, then only 1 resource needs to be preempted from other queues to obtain a total of 3 resources to execute the tasks to be scheduled.

[0111] For example, Figure 2 For example, if the first user is user 1, if task queue 1-1 is determined to be the target task queue, the current quota value of task queue 1-1 is 6, and the current used resource value is also 6, then it means that if the task queue 1-1 is not expanded, the task to be scheduled cannot be executed based on the task queue 1-1. If the resource request amount of the task to be scheduled is 1, the above-mentioned task queue 1-1 is qualified to preempt resources and the cluster idle resources of cluster 1 meet the resource request amount. Since cluster 1 only has task queue 1-2 except task queue 1-1, 1 resource is preempted from the resources corresponding to task queue 1-2 as the target resource to be scheduled.

[0112] Similarly, Figure 3For example, if the first user is user 1, if task queue 1-1 is determined to be the target task queue, the current quota value of task queue 1-1 is 6, and the current used resource value is also 6, then it means that if the task queue 1-1 is not expanded, the task to be scheduled cannot be executed based on the task queue 1-1. If the resource request amount of the task to be scheduled is 1, the above-mentioned task queue 1-1 is qualified to preempt resources and the cluster idle resources of cluster 1 meet the resource request amount. Since cluster 1 has task queue 1-2 and task queue 1-3 in addition to task queue 1-1, 1 resource is preempted from the resources corresponding to task queue 1-2 or task queue 1-3 as the target resource to be scheduled.

[0113] S610: Schedule the target to-be-scheduled resources to execute the to-be-scheduled tasks based on the target task queue.

[0114] The embodiment of the present application can optimize resource allocation and improve resource utilization by obtaining a cluster resource view; secondly, if the amount of idle resources in the cluster meets the amount of resource requests, resources of other queues are preempted, thereby increasing the utilization of cluster resources.

[0115] Furthermore, since resource preemption and resource scheduling have occurred as described above, it is necessary to update resource-related parameters corresponding to the target cluster to which the target task queue belongs.

[0116] It is not difficult to understand that the minimum allowed resource value, maximum allowed resource value and identification field in the resource-related parameters corresponding to each task queue generally do not change after being set. Therefore, in this application, the update of the current quota value and the currently used resource value contained in the resource-related parameters is focused on.

[0117] It is not difficult to understand that the current quota value of the target task queue increases, and the increase is the number of resources preempted from other queues to its own queue; the current used resource value of the target task queue increases, and the increase is the value corresponding to the resource request amount. Correspondingly, the current quota value of the queue of the preempted resources decreases, and the decrease is the number of preempted resources.

[0118] For example, Figure 2For example, if the first user is user 1, if task queue 1-1 is determined to be the target task queue, the current quota value of task queue 1-1 is 6, and the current used resource value is also 6. If the resource request amount of the task to be scheduled is 1, and task queue 1-1 has preempted 1 resource from the resources corresponding to task queue 1-2, then the current quota value of task queue 1-1 is modified to 7 (the current quota value plus the number of preempted resources), and the current used resource value of task queue 1-1 is also modified to 7 (the current used resource value plus the resource request amount), and the current quota value of task queue 1-2 is correspondingly subtracted from the number of preempted resources. For example, if the current quota value of task queue 1-2 is 8, then the current quota value of task queue 1-2 is updated from 8 minus 1 to 7, and the current used resource value of task queue 1-2 remains unchanged.

[0119] For example, Figure 3 For example, if the first user is user 1, if task queue 1-1 is determined to be the target task queue, the current quota value of task queue 1-1 is 6, and the current used resource value is also 6. If the resource request amount of the task to be scheduled is 1, it means that task queue 1-1 needs to seize 1 resource (since in cluster 1, in addition to task queue 1-1, there are task queues 1-2 and 1-3 left. Here, it is necessary to determine whose 1 resource to seize between task queues 1-2 and 1-3. Specifically, if both task queues 1-2 and 1-3 have equal to or greater than 1 resource left, task queues 1-2 or 1-3 can be randomly assigned to the task queues 1-2 and 1-3. Determined as the queue that has been preempted by one resource, if only task queue 1-2 has equal to or greater than one resource left, task queue 1-2 is determined as the queue that has been preempted by one resource). After determining the queue that has been preempted by one resource, resource preemption is performed, and the current quota value of task queue 1-1 is modified to 7 (the current quota value plus the number of preempted resources), and the current used resource value of task queue 1-1 is also modified to 7 (the current used resource value plus the resource request amount), and the current quota value of the queue that has been preempted by one resource is correspondingly subtracted by the number of preempted resources, and the current used resource value of the queue that has been preempted by one resource remains unchanged. In addition, the resource-related parameters of the remaining queues except the target task queue and the queue that has been preempted by one resource remain unchanged.

[0120] Furthermore, if each first task queue is not qualified to preempt resources, it means that the first user has no ability to process the to-be-scheduled task in the first task queue corresponding to each cluster, and the to-be-scheduled task is directly placed in the task waiting list.

[0121] Figure 7 is a schematic diagram of an exemplary implementation of a cluster resource scheduling method shown in the present application, such as Figure 7As shown, the cluster resource scheduling method includes the following steps:

[0122] S701, obtaining a task to be scheduled currently created by a first user, and obtaining a resource request amount of the task to be scheduled.

[0123] S702, obtaining a resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue included in each cluster in the cluster pool.

[0124] S703: Acquire first resource-related parameters related to the first user from the resource view list, wherein the first resource-related parameters correspond one-to-one to the first task queue related to the first user.

[0125] S704: Determine, based on the first resource-related parameters, whether the queue idle resources corresponding to each first task queue meet the resource request amount.

[0126] S705 , in response to the idle queue resources of each first task queue not satisfying the resource request amount, determining whether each first task queue is qualified to preempt resources according to first resource-related parameters.

[0127] S706 , taking the cluster where the first task queue having the qualification to seize resources is located as a candidate cluster, and determining a target cluster from the candidate clusters.

[0128] S707: Obtain the current idle resource amount of the target cluster according to the resource view list.

[0129] S708: Determine whether the amount of idle resources in the cluster meets the resource request amount.

[0130] Regarding the specific implementation of steps S701 to S708, reference may be made to the specific introduction of the relevant parts in the above embodiments, which will not be elaborated here.

[0131] S709, in response to the cluster idle resource amount not satisfying the resource request amount, compare the current quota value recorded in the first resource-related parameter corresponding to the first task queue in the target cluster with the minimum allowed resource value to obtain a fourth comparison result.

[0132] If the current idle resources of the target cluster do not meet the resource request, it means that sufficient resources cannot be obtained to execute the scheduled tasks based on resource preemption. At this time, it is necessary to determine whether the first task queue in the target cluster has its own resources preempted by other queues at the current moment.

[0133] For example, Figure 2For example, if the first user is user 1 and the target cluster is cluster 1, the first task queue in the target cluster is task queue 1-1. The current quota value recorded in the first resource-related parameter of task queue 1-1 is compared with the minimum allowed resource value to obtain a fourth comparison result.

[0134] S710: In response to a fourth comparison result indicating that the current quota value is less than the minimum allowed resource value, obtain a preempted state task queue in the target cluster that is in a preempted resource state.

[0135] As mentioned above, the current quota value in the resource-related parameters corresponding to each task queue is set to the minimum allowed resource value corresponding to the task queue during initialization. If the current quota value of the task queue is less than the minimum allowed resource value, it means that the resources that the task queue should have are now preempted by other queues.

[0136] S711, determining a task to be deleted from the running tasks running in the preempted state task queue.

[0137] First, it is necessary to determine which queues in the target cluster have preempted the resources of other queues. Specifically, the current quota value and the minimum allowed resource value of each queue in the target cluster except the first task queue related to the first user can be obtained. If the current quota value of a queue is greater than its corresponding minimum allowed resource value, it means that the queue is a preemptive task queue in the resource preemption state.

[0138] Then, obtain the third difference between the resource request amount and the cluster's idle resources (for example, if the resource request amount of the task to be scheduled requires 3 resources, and the cluster has 2 idle resources at this time, that is, the third difference is 1, which means that in addition to the idle resources of the cluster that need to be used at this time, 1 resource needs to be recovered to run the task to be scheduled); obtain the task running resource value, task priority and task running time of each running task running in the preempted state task queue; determine the task to be deleted from the running tasks based on the task running resource value, task priority and task running time, combined with the third difference.

[0139] Specifically, the task running resource value, task priority and task running time of each running task running in all preempted task queues are obtained, and the running task with the lowest task priority is preferentially determined as the task to be deleted (if multiple running tasks are of the lowest priority, the task with the shortest running time among the running tasks with the lowest priority is preferentially determined as the task to be deleted), and then it is determined whether the resources vacated after deleting the task to be deleted (that is, the task running resource value of the task to be deleted) is greater than or equal to the third difference between the resource request amount and the cluster idle resource amount. If it is greater than or equal to, it means that deleting this one task to be deleted can provide sufficient resources to the task to be scheduled in combination with the cluster idle resources (here it means that the total number of resources vacated by deleting the task to be deleted plus the cluster idle resources meets the resource request amount of the task to be scheduled). If it is less than, it means that it is necessary to continue to find the next task to be deleted, and then it is determined whether the resources vacated after deleting the two tasks to be deleted are greater than or equal to the third difference between the resource request amount and the cluster idle resource amount, and so on, until it is determined that the resources vacated after deleting the task to be deleted are greater than or equal to the third difference between the resource request amount and the cluster idle resource amount.

[0140] S712: Pause the task to be deleted and delete it from the target cluster.

[0141] S713: The first task queue in the target cluster is used as the target task queue, and the target resources to be scheduled are determined based on the idle resources freed up after the tasks to be deleted are deleted.

[0142] For example, Figure 2 For example, if the first user is user 1 and the target cluster is cluster 1, then the task queue 1-1 in the target cluster is the target task queue.

[0143] In the present application, the target resources to be scheduled are determined from the resources freed up after the tasks to be deleted are deleted and the idle resources of the cluster.

[0144] S714, scheduling the target to-be-scheduled resources to execute the to-be-scheduled tasks based on the target task queue.

[0145] The embodiment of the present application mainly introduces the situation where if the idle resources in the target cluster cannot meet the resource request amount, it is necessary to recycle the resources belonging to the target task queue. In a multi-cluster environment, resources can be dynamically scheduled according to load changes to provide good elastic expansion capabilities.

[0146] Furthermore, since resource recycling and resource scheduling have occurred as described above, it is necessary to update resource-related parameters corresponding to the target cluster to which the target task queue belongs.

[0147] It is not difficult to understand that the minimum allowed resource value, maximum allowed resource value and identification field in the resource-related parameters corresponding to each task queue generally do not change after being set. Therefore, in this application, the update of the current quota value and the currently used resource value contained in the resource-related parameters is focused on.

[0148] It is not difficult to understand that the current quota value of the target task queue increases by the amount of increase: the value of the resources allocated to itself from the original idle resources of other queues and the idle resources after the to-be-deleted tasks are deleted; the current used resource value of the target task queue increases by the amount of increase corresponding to the resource request amount; correspondingly, the current quota value of the queue of recycled resources decreases by the amount of decrease: the number of resources allocated to the target task queue from the idle resources after recycling, and the current used resource value of the queue of recycled resources decreases by the amount of running resources corresponding to the deleted tasks in the queue.

[0149] For example, Figure 2 For example, if the resource request amount of the task to be scheduled is 1, if the first user is user 1, if task queue 1-1 is determined to be the target task queue, the current quota value of task queue 1-1 is 6 (the initial quota value is 7), and the current quota value of task queue 1-2 is 4 (the initial quota value is 3), that is, task queue 1-2 has seized 1 resource from task queue 1-1, then the task to be deleted is determined from task queue 1-2. Taking the task to be deleted occupying 1 resource as an example, the resources vacated after deleting the task to be deleted are recycled and called by task queue 1-1 to execute the task to be scheduled. At this time, the current quota value and the current used resource value of task queue 1-1 are both increased by 1, and the current quota value and the current used resource value of task queue 1-2 are both reduced by 1.

[0150] Furthermore, if the fourth comparison result indicates that the current quota value is greater than or equal to the minimum allowed resource value, it means that the idle resources of the cluster cannot meet the resource request at the current moment, and there is no resource preemption in the target task queue. At this moment, there are no resources to execute the task to be scheduled, so the task to be scheduled is placed in the task waiting list.

[0151] Furthermore, the second user corresponding to the task to be deleted is obtained; the second resource-related parameters of the second task queues corresponding to the second user in other clusters are obtained from the resource view list; based on the second resource-related parameters, it is determined whether the queue idle resources corresponding to each second task queue meet the second resource request amount corresponding to the task to be deleted (this is because during the period when the task to be deleted is running in the target cluster before it is deleted, the task queue resources corresponding to the second user in other clusters may be idle); in response to the existence of at least one second task queue whose queue idle resources meet the second resource request amount, the task to be deleted is executed based on the second task queue that meets the second resource request amount (if there are multiple second task queues whose queue idle resources meet the second resource request amount, one second task queue is selected from them to execute the task to be deleted); in response to the fact that the queue idle resources of all second task queues do not meet the second resource request amount, the task to be deleted is placed in a task waiting list.

[0152] Figure 8 is a schematic diagram of a cluster resource scheduling device shown in the present application, such as Figure 8 As shown, the cluster resource scheduling device 800 includes a first acquisition module 801, a second acquisition module 802, a determination module 803 and an execution module 804, wherein:

[0153] The first acquisition module 801 is used to acquire the task to be scheduled currently created by the first user, and acquire the resource request amount of the task to be scheduled;

[0154] The second acquisition module 802 is used to acquire a resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue included in each cluster in the cluster pool;

[0155] A determination module 803 is used to determine a target task queue and a target resource to be scheduled from the cluster pool according to the resource view list;

[0156] The execution module 804 is used to schedule the target to-be-scheduled resources to execute the to-be-scheduled tasks based on the target task queue.

[0157] By obtaining the resource view list of each cluster, this device can understand the resource status of each cluster from a global perspective, so as to make more reasonable scheduling decisions and avoid overloading of resources in a certain cluster while other clusters are idle; the resource request amount of the task to be scheduled is obtained in real time, and combined with the resource view of the cluster pool, resource allocation can be flexibly adjusted; in a multi-cluster environment, resources can be dynamically scheduled according to load changes, providing good elastic expansion capabilities.

[0158] According to one embodiment of the present application, the determination module 803 is also used to: obtain first resource-related parameters related to the first user from the resource view list, wherein the first resource-related parameters correspond one-to-one to the first task queue related to the first user; determine whether the queue idle resources corresponding to each first task queue meet the resource request amount according to the first resource-related parameters; in response to the existence of at least one first task queue whose queue idle resources meet the resource request amount, use the first task queue as a candidate first task queue; determine the target task queue from the candidate first task queues, and determine the target to-be-scheduled resources from the queue idle resources corresponding to the target task queue.

[0159] According to one embodiment of the present application, the determination module 803 is also used to: obtain, based on the first resource-related parameters, a first difference between the current quota value recorded in each first resource-related parameter and the currently used resource value; compare the first difference with the resource request amount to obtain a first comparison result; in response to the first comparison result corresponding to any first resource-related parameter indicating that the first difference is greater than or equal to the resource request amount, determine that the queue idle resources of the first task queue corresponding to the first resource-related parameter meet the resource request amount; in response to the first comparison result corresponding to any first resource-related parameter indicating that the first difference is less than the resource request amount, determine that the queue idle resources of the first task queue corresponding to the first resource-related parameter do not meet the resource request amount.

[0160] According to one embodiment of the present application, the determination module 803 is also used to: in response to the queue idle resources of each first task queue not satisfying the resource request amount, determine whether each first task queue is eligible to preempt resources according to the first resource-related parameters; use the cluster where the first task queue that is eligible to preempt resources is located as a candidate cluster, and determine the target cluster from the candidate clusters; obtain the cluster idle resource amount of the target cluster at the current moment according to the resource view list; determine whether the cluster idle resource amount satisfies the resource request amount; in response to the cluster idle resource amount satisfying the resource request amount, perform resource preemption on the cluster idle resources of the target cluster to determine the target resources to be scheduled, and use the first task queue belonging to the target cluster as the target task queue.

[0161] According to one embodiment of the present application, the determination module 803 is also used to: obtain, based on the first resource-related parameters, a second difference between the maximum allowed resource value and the currently used resource value recorded in each first resource-related parameter; compare the second difference with the resource request amount to obtain a second comparison result; in response to any second comparison result indicating that the second difference is greater than or equal to the resource request amount, determine that the first task queue corresponding to the second comparison result is eligible to preempt resources; in response to any second comparison result indicating that the second difference is less than the resource request amount, determine that the first task queue corresponding to the second comparison result is not eligible to preempt resources.

[0162] According to one embodiment of the present application, the determination module 803 is also used to: compare the amount of idle resources in the cluster and the amount of resource requests to obtain a third comparison result; in response to the third comparison result indicating that the amount of idle resources in the cluster is greater than or equal to the amount of resource requests, determine that the amount of idle resources in the cluster meets the amount of resource requests; in response to the third comparison result indicating that the amount of idle resources in the cluster is less than the amount of resource requests, determine that the amount of idle resources in the cluster does not meet the amount of resource requests.

[0163] According to one embodiment of the present application, the determination module 803 is also used to: in response to the amount of idle resources in the cluster not satisfying the amount of resource request, compare the current quota value recorded in the first resource-related parameter corresponding to the first task queue in the target cluster with the minimum allowed resource value to obtain a fourth comparison result; in response to the fourth comparison result indicating that the current quota value is less than the minimum allowed resource value, obtain the preempted state task queue in the target cluster that belongs to the resource preempted state; determine the task to be deleted from the running tasks running in the preempted state task queue; suspend the running of the task to be deleted and delete it from the target cluster; use the first task queue in the target cluster as the target task queue, and determine the target resource to be scheduled based on the idle resources vacated after the task to be deleted is deleted.

[0164] According to one embodiment of the present application, the determination module 803 is also used to: obtain a third difference between the resource request amount and the cluster idle resource amount; obtain the task running resource value, task priority and task running time of each running task running in the preempted state task queue; determine the task to be deleted from the running tasks based on the task running resource value, task priority and task running time, combined with the third difference.

[0165] According to an embodiment of the present application, the determination module 803 is further used to: update resource-related parameters corresponding to the target cluster to which the target task queue belongs.

[0166] According to an embodiment of the present application, the determination module 803 is further used to: in response to the fourth comparison result indicating that the current quota value is greater than or equal to the minimum allowed resource value, place the to-be-scheduled task in a task waiting list.

[0167] According to an embodiment of the present application, the determination module 803 is further used to: in response to each first task queue not being eligible to preempt resources, place the to-be-scheduled task in a task waiting list.

[0168] According to one embodiment of the present application, the determination module 803 is also used to: obtain the priority of the cluster to which each candidate first task queue belongs, and use the candidate first task queue corresponding to the cluster with the highest priority as the target task queue; or, randomly select a candidate first task queue from the candidate first task queues as the target task queue; or, compare the first difference values ​​corresponding to each candidate first task queue, and use the candidate first task queue corresponding to the first difference value with the largest value as the target task queue.

[0169] According to one embodiment of the present application, the determination module 803 is also used to: obtain the second user corresponding to the task to be deleted; obtain the second resource-related parameters of the second task queue corresponding to the second user in other clusters from the resource view list; determine whether the queue idle resources corresponding to each second task queue meet the second resource request amount corresponding to the task to be deleted based on the second resource-related parameters; in response to the existence of at least one second task queue whose queue idle resources meet the second resource request amount, execute the task to be deleted based on the second task queue that meets the second resource request amount; in response to the queue idle resources of the second task queues not meeting the second resource request amount, place the task to be deleted in the task waiting list.

[0170] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0171] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0172] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0173] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0174] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the cluster resource scheduling method. For example, in some embodiments, the cluster resource scheduling method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the cluster resource scheduling method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the cluster resource scheduling method in any other appropriate manner (e.g., by means of firmware).

[0175] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0177] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0179] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0180] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0181] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0182] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A cluster resource scheduling method, comprising: Obtaining a task to be scheduled currently created by the first user, and obtaining a resource request amount of the task to be scheduled; Obtain a resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue included in each cluster in the cluster pool; Determine a target task queue and a target resource to be scheduled from the cluster pool according to the resource view list; The target to-be-scheduled resource is scheduled based on the target task queue to execute the to-be-scheduled task.

2. The method according to claim 1, wherein: The determining the target task queue and the target resource to be scheduled from the cluster pool according to the resource view list includes: Acquire a first resource-related parameter related to the first user from the resource view list, wherein the first resource-related parameter corresponds one-to-one to a first task queue related to the first user; According to the first resource-related parameters, determining whether the queue idle resources corresponding to each of the first task queues meet the resource request amount; In response to the existence of at least one queue idle resource of the first task queue satisfying the resource request amount, taking the first task queue as a candidate first task queue; A target task queue is determined from the candidate first task queues, and a target resource to be scheduled is determined from queue idle resources corresponding to the target task queue.

3. The method according to claim 2, wherein: The determining, according to the first resource-related parameter, whether the queue idle resources corresponding to each of the first task queues meet the resource request amount includes: According to the first resource-related parameters, obtaining a first difference between a current quota value and a currently used resource value recorded in each of the first resource-related parameters; Comparing the first difference with the resource request amount to obtain a first comparison result; In response to the first comparison result corresponding to any of the first resource-related parameters indicating that the first difference is greater than or equal to the resource request amount, determining that queue idle resources of the first task queue corresponding to the first resource-related parameter meet the resource request amount; In response to the first comparison result corresponding to any of the first resource-related parameters indicating that the first difference is less than the resource request amount, it is determined that queue idle resources of the first task queue corresponding to the first resource-related parameter do not meet the resource request amount.

4. The method according to claim 3, wherein: The method further comprises: In response to the idle queue resources of each of the first task queues not satisfying the resource request amount, determining whether each of the first task queues is eligible to preempt resources according to the first resource-related parameters; The cluster where the first task queue having the preemptive resource qualification is located is taken as a candidate cluster, and a target cluster is determined from the candidate clusters; According to the resource view list, obtain the cluster idle resource amount of the target cluster at the current moment; Determine whether the idle resources of the cluster meet the resource request amount; In response to the cluster idle resource amount satisfying the resource request amount, resource preemption is performed on the cluster idle resources of the target cluster to determine the target resources to be scheduled, and the first task queue belonging to the target cluster is used as the target task queue.

5. The method according to claim 4, wherein: The determining, according to the first resource-related parameters, whether each of the first task queues is qualified to seize resources includes: According to the first resource-related parameters, obtaining a second difference between the maximum allowed resource value and the currently used resource value recorded in each of the first resource-related parameters; Comparing the second difference with the resource request amount to obtain a second comparison result; In response to any of the second comparison results indicating that the second difference is greater than or equal to the resource request amount, determining that the first task queue corresponding to the second comparison result is eligible to preempt resources; In response to any second comparison result indicating that the second difference is smaller than the resource request amount, it is determined that the first task queue corresponding to the second comparison result is not eligible to preempt resources.

6. The method according to claim 5, wherein: The determining whether the amount of idle resources in the cluster meets the amount of resource requests includes: Comparing the idle resource amount of the cluster with the resource request amount to obtain a third comparison result; In response to the third comparison result indicating that the amount of idle resources in the cluster is greater than or equal to the amount of requested resources, determining that the amount of idle resources in the cluster satisfies the amount of requested resources; In response to the third comparison result indicating that the idle resource amount of the cluster is less than the requested resource amount, it is determined that the idle resource amount of the cluster does not satisfy the requested resource amount.

7. The method according to claim 6, wherein: The method further comprises: In response to the idle resource amount of the cluster not satisfying the resource request amount, comparing the current quota value recorded in the first resource-related parameter corresponding to the first task queue in the target cluster with the minimum allowed resource value to obtain a fourth comparison result; In response to the fourth comparison result indicating that the current quota value is less than the minimum allowed resource value, obtaining a preempted state task queue in the target cluster that is in a preempted resource state; Determine the task to be deleted from the running tasks running in the preempted state task queue; Pause the task to be deleted and delete it from the target cluster; The first task queue belonging to the target cluster is used as the target task queue, and the target to-be-scheduled resources are determined based on the idle resources vacated after the to-be-deleted tasks are deleted.

8. The method according to claim 7, wherein: The step of determining the task to be deleted from the running tasks in the preempted state task queue includes: Obtaining a third difference between the resource request amount and the cluster idle resource amount; Obtaining the task running resource value, task priority and task running time of each running task running in the preempted state task queue; According to the task running resource value, the task priority and the task running time, combined with the third difference, a task to be deleted is determined from the running tasks.

9. The method according to any one of claims 2 to 8, wherein: After scheduling the target resource to be scheduled to execute the task to be scheduled based on the target task queue, the method further includes: The resource-related parameters corresponding to the target cluster to which the target task queue belongs are updated.

10. The method according to claim 7, wherein: The method further comprises: In response to the fourth comparison result indicating that the current quota value is greater than or equal to the minimum allowed resource value, the to-be-scheduled task is placed in a task waiting list.

11. The method according to claim 5, wherein: The method further comprises: In response to each of the first task queues not being eligible to preempt resources, the to-be-scheduled tasks are placed in a task waiting list.

12. The method according to claim 3, wherein: The determining of the target task queue from the candidate first task queues includes: Obtaining the priority of the cluster to which each candidate first task queue belongs, and taking the candidate first task queue corresponding to the cluster with the highest priority as the target task queue; or, Randomly select a candidate first task queue from the candidate first task queues as the target task queue; or, The first difference values ​​corresponding to each of the candidate first task queues are compared, and the candidate first task queue corresponding to the first difference value with the largest value is taken as the target task queue.

13. The method according to claim 8, wherein: The method further comprises: Obtaining a second user corresponding to the task to be deleted; Acquire, from the resource view list, second resource related parameters of the second task queue corresponding to the second user in other clusters; According to the second resource related parameters, determining whether the queue idle resources corresponding to each of the second task queues meet the second resource request amount corresponding to the task to be deleted; In response to the existence of at least one queue idle resource of the second task queue that satisfies the second resource request amount, executing the to-be-deleted task based on the second task queue that satisfies the second resource request amount; In response to the fact that the idle resources of the second task queue do not satisfy the second resource request amount, the to-be-deleted task is placed in a task waiting list.

14. A cluster resource scheduling device, comprising: A first acquisition module is used to acquire a task to be scheduled currently created by a first user, and to acquire a resource request amount of the task to be scheduled; A second acquisition module is used to acquire a resource view list corresponding to the cluster pool at the current moment, wherein the resource view list is used to record resource-related parameters corresponding to each task queue included in each cluster in the cluster pool; A determination module, used for determining a target task queue and a target resource to be scheduled from the cluster pool according to the resource view list; An execution module is used to schedule the target resource to be scheduled to execute the task to be scheduled based on the target task queue.

15. The device according to claim 14, wherein: The determining module is further used for: Acquire a first resource-related parameter related to the first user from the resource view list, wherein the first resource-related parameter corresponds one-to-one to a first task queue related to the first user; According to the first resource-related parameters, determining whether the queue idle resources corresponding to each of the first task queues meet the resource request amount; In response to the existence of at least one queue idle resource of the first task queue satisfying the resource request amount, taking the first task queue as a candidate first task queue; A target task queue is determined from the candidate first task queues, and a target resource to be scheduled is determined from queue idle resources corresponding to the target task queue.

16. The device according to claim 15, wherein: The determining module is further used for: According to the first resource-related parameters, obtaining a first difference between a current quota value and a currently used resource value recorded in each of the first resource-related parameters; Comparing the first difference with the resource request amount to obtain a first comparison result; In response to the first comparison result corresponding to any of the first resource-related parameters indicating that the first difference is greater than or equal to the resource request amount, determining that queue idle resources of the first task queue corresponding to the first resource-related parameter meet the resource request amount; In response to the first comparison result corresponding to any of the first resource-related parameters indicating that the first difference is less than the resource request amount, it is determined that queue idle resources of the first task queue corresponding to the first resource-related parameter do not meet the resource request amount.

17. The device according to claim 16, wherein: The determining module is further used for: In response to the idle queue resources of each of the first task queues not satisfying the resource request amount, determining whether each of the first task queues is eligible to preempt resources according to the first resource-related parameters; The cluster where the first task queue having the preemptive resource qualification is located is taken as a candidate cluster, and a target cluster is determined from the candidate clusters; According to the first resource-related parameter, obtaining the cluster idle resource amount of the target cluster at the current moment; Determine whether the idle resources of the cluster meet the resource request amount; In response to the cluster idle resource amount satisfying the resource request amount, resource preemption is performed on the cluster idle resources of the target cluster to determine the target resources to be scheduled, and the first task queue belonging to the target cluster is used as the target task queue.

18. The device according to claim 17, wherein: The determining module is further used for: According to the resource view list, obtaining a second difference between the maximum allowed resource value and the currently used resource value recorded in each of the first resource-related parameters; Comparing the second difference with the resource request amount to obtain a second comparison result; In response to any of the second comparison results indicating that the second difference is greater than or equal to the resource request amount, determining that the first task queue corresponding to the second comparison result is eligible to preempt resources; In response to any second comparison result indicating that the second difference is smaller than the resource request amount, it is determined that the first task queue corresponding to the second comparison result is not eligible to preempt resources.

19. The device according to claim 18, wherein: The determining module is further used for: Comparing the idle resource amount of the cluster with the resource request amount to obtain a third comparison result; In response to the third comparison result indicating that the amount of idle resources in the cluster is greater than or equal to the amount of requested resources, determining that the amount of idle resources in the cluster satisfies the amount of requested resources; In response to the third comparison result indicating that the idle resource amount of the cluster is less than the requested resource amount, it is determined that the idle resource amount of the cluster does not satisfy the requested resource amount.

20. The device according to claim 19, wherein The determining module is further used for: In response to the idle resource amount of the cluster not satisfying the resource request amount, comparing the current quota value recorded in the first resource-related parameter corresponding to the first task queue in the target cluster with the minimum allowed resource value to obtain a fourth comparison result; In response to the fourth comparison result indicating that the current quota value is less than the minimum allowed resource value, obtaining a preempted state task queue in the target cluster that is in a preempted resource state; Determine the task to be deleted from the running tasks running in the preempted state task queue; Pause the task to be deleted and delete it from the target cluster; The first task queue belonging to the target cluster is used as the target task queue, and the target to-be-scheduled resources are determined based on the idle resources vacated after the to-be-deleted tasks are deleted.

21. The device according to claim 20, wherein: The determining module is further used for: Obtaining a third difference between the resource request amount and the cluster idle resource amount; Obtaining the task running resource value, task priority and task running time of each running task running in the preempted state task queue; According to the task running resource value, the task priority and the task running time, combined with the third difference, a task to be deleted is determined from the running tasks.

22. The device according to any one of claims 15 to 21, wherein: The determining module is further used for: The resource-related parameters corresponding to the target cluster to which the target task queue belongs are updated.

23. The device according to claim 20, wherein: The determining module is further used for: In response to the fourth comparison result indicating that the current quota value is greater than or equal to the minimum allowed resource value, the to-be-scheduled task is placed in a task waiting list.

24. The device according to claim 18, wherein The determining module is further used for: In response to each of the first task queues not being eligible to preempt resources, the to-be-scheduled tasks are placed in a task waiting list.

25. The device according to claim 16, wherein: The determining module is further used for: Obtaining the priority of the cluster to which each candidate first task queue belongs, and taking the candidate first task queue corresponding to the cluster with the highest priority as the target task queue; or, Randomly select a candidate first task queue from the candidate first task queues as the target task queue; or, The first difference values ​​corresponding to each of the candidate first task queues are compared, and the candidate first task queue corresponding to the first difference value with the largest value is taken as the target task queue.

26. The device according to claim 21, wherein The determining module is further used for: Obtaining a second user corresponding to the task to be deleted; Acquire, from the resource view list, second resource related parameters of the second task queue corresponding to the second user in other clusters; According to the second resource related parameters, determining whether the queue idle resources corresponding to each of the second task queues meet the second resource request amount corresponding to the task to be deleted; In response to the existence of at least one queue idle resource of the second task queue that satisfies the second resource request amount, executing the to-be-deleted task based on the second task queue that satisfies the second resource request amount; In response to the fact that the idle resources of the second task queue do not satisfy the second resource request amount, the to-be-deleted task is placed in a task waiting list.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images