Method, apparatus and system for dynamically adjusting task allocation

Abstract

The application discloses a kind of dynamically adjusted task allocation method, device and system, it is related to computer technical field.The specific embodiment of the method includes: to each host included in cluster, the performance parameter of host is obtained regularly;According to preset rule and performance parameter, judge whether host is to be scheduled host;In the case where host is to be scheduled host, from the task that host is executing, determine redistribution task, end redistribution task and add redistribution task to task queue to re-perform task allocation.The embodiment can realize the optimal allocation of resource to cluster according to the actual demand of task, make the use rate of CPU, memory and disk of entire cluster host, etc. at a balanced level, make the flow data task distribution in cluster more balanced and can stably run, simultaneously, improve cluster CPU resource utilization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method, device and system for dynamically adjusting task allocation. BACKGROUND

[0002] Since the emergence of Apache Hadoop, a distributed cluster composed of multiple scattered Worker hosts has become the mainstream architecture of big data processing. In this architecture, a Master host is generally responsible for the scheduling of computing tasks, and multiple Worker hosts are responsible for the execution of actual tasks.

[0003] At present, in such a cluster, the mainstream task scheduling scheme is based on fixed slots as the basic unit of task processing resources. Taking Apache Flink, a distributed processing engine for streaming data, as an example, the TaskManager task management module sets a fixed number of slots according to the performance of the Worker node when starting, and each slot can start a task Task. Each slot holds a certain amount of memory resources. After receiving the Task to be deployed from the TaskManager, each slot establishes a connection with its upstream, receives data and processes it.

[0004] Most of the existing task scheduling schemes in the cluster are based on static scheduling of slots. The number of slots in the cluster and the resources held are determined at the start of the cluster and will not change. Moreover, the tasks in the cluster are only scheduled once at the initialization of the task, and after the task is allocated to a slot on a Worker node, no second allocation will be made unless the task fails. However, in actual cluster operation, different tasks have different demands for CPU, memory and other computing resources. In the static scheduling scheme, in order to meet the runtime needs of most tasks, the resources of the slots need to be set according to the maximum resource demand of all tasks throughout the task lifecycle, which results in a great waste of computing resources. SUMMARY

[0005] Therefore, the embodiments of the present application provide a method, device and system for dynamically adjusting task allocation, which can dynamically schedule the tasks being executed during the task execution phase, realize the optimal allocation of resources in the cluster according to the actual needs of the tasks, make the usage rate of CPU, memory and disk resources of the entire cluster host at a balanced level, make the distribution of streaming data tasks in the cluster more balanced and stable, and improve the CPU resource utilization of the cluster.

[0006] To achieve the above object, according to an aspect of the embodiments of the present application, a method for dynamically adjusting task allocation is provided.

[0007] The method comprises: acquiring performance parameters of each host included in the cluster at a timing; judging whether the host is a to-be-scheduled host according to preset rules and the performance parameters; in the case that the host is a to-be-scheduled host, determining a reassignment task from tasks being executed by the host, ending the reassignment task and adding the reassignment task to a task queue for re-tasking.

[0008] Optionally, the task allocation comprises: acquiring a to-be-allocated task from the task queue, and selecting a host for the to-be-allocated task according to task information of the to-be-allocated task, and allocating the to-be-allocated task to the host.

[0009] Optionally, the selecting a host for the to-be-allocated task according to the task information of the to-be-allocated task comprises: judging whether the to-be-allocated task is a CPU reassignment task according to the task information; if yes, sorting the hosts included in the cluster according to available CPU resource sizes; otherwise, sorting the hosts included in the cluster according to available process number proportions; and selecting a host from the sorted hosts.

[0010] Optionally, the selecting a host from the sorted hosts comprises: traversing the sorted hosts in sequence, and selecting a host according to available process numbers, available disk spaces, CPU loads and residual memories of the hosts.

[0011] Optionally, the selecting a host according to available process numbers, available disk spaces, CPU loads and residual memories of the hosts comprises: for each host, judging whether an available process number of the host is greater than a process number required by the to-be-allocated task; if yes, judging whether the to-be-allocated task is a disk consumption task; if yes, judging whether a CPU load of the host is less than a second threshold value when an available disk space of the host is greater than a first threshold value; otherwise, directly judging whether the CPU load of the host is less than the second threshold value; if yes, taking the host as the selected host when a residual memory of the host is greater than a memory required by the to-be-allocated task; otherwise, adding the host to a to-be-scheduled host list; and if no host is selected after the traversal is completed, taking a host with a lowest CPU load in the to-be-scheduled host list as the selected host.

[0012] Optionally, the performance parameter comprises CPU load and available disk space; and the determining whether the host is a to-be-scheduled host according to the preset rule and the performance parameter comprises: determining whether the host is a to-be-scheduled host according to the CPU load and the available disk space of the host, wherein if the CPU load of the host is greater than a third threshold value and the duration reaches a first time threshold value, the host is regarded as a to-be-scheduled host; and if the available disk space of the host is less than a fourth threshold value and the duration reaches a second time threshold value, the host is regarded as a to-be-scheduled host.

[0013] Optionally, the state type of the to-be-scheduled host comprises disk high load and CPU high load; and the determining the re-allocated task from the tasks being executed by the host comprises: determining whether the state type of the host is disk high load; if yes, determining the task with the largest disk occupancy being executed by the host as the re-allocated task; otherwise, determining the re-allocated task according to the CPU usage of the tasks being executed by the host.

[0014] Optionally, the determining the re-allocated task according to the CPU usage of the tasks being executed by the host comprises: obtaining the task with the largest CPU usage, and determining whether the CPU usage of the task exceeds a fifth threshold value; if yes, for each task being executed by the host except the task with the largest CPU usage, if the task is not scheduled in a recent period, determining the task as the re-allocated task, and adding a CPU re-allocated task identifier to the task; otherwise, sequentially traversing each task being executed by the host, and performing the following operation on each task until the CPU usage of the host is less than the fifth threshold value: if the task is not scheduled in a recent period, determining the task as the re-allocated task, and adding a CPU re-allocated task identifier to the task.

[0015] According to another aspect of the embodiment of the present application, a device for dynamically adjusting task allocation is provided.

[0016] A device for dynamically adjusting task allocation comprises: a host parameter acquisition module, configured to acquire performance parameters of each host included in a cluster at a timing; a host state determination module, configured to determine whether the host is a to-be-scheduled host according to a preset rule and the performance parameters; and a task allocation adjustment module, configured to, in a case where the host is a to-be-scheduled host, determine a re-allocated task from tasks being executed by the host, end the re-allocated task, and add the re-allocated task to a task queue for re-performing task allocation.

[0017] Optionally, the task allocation adjustment module is further configured to, when performing task allocation: acquire a to-be-allocated task from the task queue, and select a host for the to-be-allocated task according to task information of the to-be-allocated task, and allocate the to-be-allocated task to the host.

[0018] Optionally, the task allocation adjustment module is further configured to: determine, according to the task information, whether the task to be allocated is a CPU reallocation task; if yes, sort the hosts included in the cluster according to available CPU resource sizes; otherwise, sort the hosts included in the cluster according to available process number proportions; and select a host from the sorted hosts.

[0019] Optionally, when selecting a host from the sorted hosts, the task allocation adjustment module is further configured to: traverse the sorted hosts in sequence, and select a host according to available process numbers, available disk spaces, CPU loads and residual memories of each host.

[0020] Optionally, when selecting a host from the sorted hosts, the task allocation adjustment module is further configured to: for each host, determine whether an available process number of the host is greater than a process number required by the task to be allocated; if yes, determine whether the task to be allocated is a disk consumption task; if yes, when an available disk space of the host is greater than a first threshold, determine whether a CPU load of the host is less than a second threshold; otherwise, directly determine whether the CPU load of the host is less than the second threshold; if yes, when a residual memory of the host is greater than a memory required by the task to be allocated, select the host as the selected host; otherwise, add the host to a list of hosts to be scheduled; and if no host is selected after the traversal is completed, select a host with a lowest CPU load in the list of hosts to be scheduled as the selected host.

[0021] Optionally, the performance parameters include a CPU load and an available disk space; and the host state determination module is further configured to: determine, according to the CPU load and the available disk space of the host, whether the host is a host to be scheduled, wherein if the CPU load of the host is greater than a third threshold and a duration reaches a first time threshold, the host is determined as a host to be scheduled; and if the available disk space of the host is less than a fourth threshold and a duration reaches a second time threshold, the host is determined as a host to be scheduled.

[0022] Optionally, the state types of the hosts to be scheduled include a disk high load and a CPU high load; and the task allocation adjustment module is further configured to: determine whether a state type of the host is the disk high load; if yes, determine a task with a largest disk occupancy amount being executed by the host as a reallocation task; otherwise, determine a reallocation task according to a CPU usage rate of a task being executed by the host.

[0023] Optionally, the task allocation adjustment module is further configured to: acquire a task with the largest CPU usage, and determine whether the CPU usage of the task exceeds a fifth threshold; if yes, for each task being executed by the host except the task with the largest CPU usage, if the task has not been scheduled in a recent period, the task is determined as a reallocation task, and a CPU reallocation task identifier is added to the task; otherwise, each task being executed by the host is sequentially traversed, and the following operation is performed on each task until the CPU usage of the host is less than the fifth threshold: if the task has not been scheduled in a recent period, the task is determined as a reallocation task, and a CPU reallocation task identifier is added to the task.

[0024] According to a further aspect of the embodiments of the present application, a system for dynamically adjusting task allocation is provided.

[0025] A system for dynamically adjusting task allocation comprises: a task scheduling node configured to acquire performance parameters of each host included in a cluster at a time; determine whether the host is a to-be-scheduled host according to a preset rule and the performance parameters; in a case where the host is the to-be-scheduled host, determine a reallocation task from tasks being executed by the host, end the reallocation task, and add the reallocation task to a task queue for re-task allocation; and a host configured to execute allocated tasks.

[0026] Optionally, the host is further configured to: acquire current resource usage data of itself at a time, and report the resource usage data to the task scheduling node; and the task scheduling node is further configured to: determine a type of the host according to the resource usage data.

[0027] Optionally, the host is further configured to: acquire resource occupation data of tasks being executed at a time, and report the resource occupation data to the task scheduling node; and the task scheduling node is further configured to: determine a type of each task according to the resource occupation data.

[0028] According to a further aspect of the embodiments of the present application, an electronic device for dynamically adjusting task allocation is provided.

[0029] An electronic device for dynamically adjusting task allocation comprises: one or more processors; and a storage configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a method for dynamically adjusting task allocation provided by the embodiments of the present application.

[0030] According to a further aspect of the embodiments of the present application, a computer readable medium is provided.

[0031] A computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method for dynamically adjusting task allocation provided by the embodiments of the present application.

[0032] An embodiment of the above-mentioned application has the following advantages or beneficial effects: the performance parameters of each host included in the cluster are acquired at a timing; whether the host is a to-be-scheduled host is determined according to a preset rule and the performance parameters; in the case that the host is a to-be-scheduled host, a reallocation task is determined from the tasks being executed by the host, the reallocation task is ended and added to a task queue for re-task allocation. The technical solution uses a task scheduling algorithm that still monitors the running status of the hosts in the cluster in the process of task execution and dynamically adjusts and allocates tasks, realizes the optimal allocation of the resources of the cluster according to the actual needs of the tasks, makes the usage rates of the CPU, memory and disk resources of the entire cluster host at a balanced level, makes the distribution of the stream data tasks in the cluster more balanced and stable, and improves the CPU resource utilization rate of the cluster.

[0033] The further effects of the above-mentioned non-conventional optional mode will be described in the following in combination with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application. Among them:

[0035] Figure 1 is a main step schematic diagram of the method for dynamically adjusting task allocation according to the embodiments of the present application;

[0036] Figure 2 is a system architecture diagram of the task allocation system according to the embodiments of the present application;

[0037] Figure 3 is an implementation flow schematic diagram of the task allocation stage according to the embodiments of the present application;

[0038] Figure 4 is an implementation flow schematic diagram of the task execution stage for finding a to-be-scheduled host according to the embodiments of the present application;

[0039] Figure 5 is an implementation flow schematic diagram of the task execution stage for determining a reallocation task according to the embodiments of the present application;

[0040] Figure 6 is a main module schematic diagram of the device for dynamically adjusting task allocation according to the embodiments of the present application;

[0041] Figure 7 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0042] Figure 8 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0043] Exemplary embodiments of the present application are described herein below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding them. These should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize various changes, modifications, and implementations of the embodiments described herein, without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0044] In the prior art, most of the task scheduling schemes in a cluster are based on a slot static scheduling scheme. This static scheduling is mainly reflected in that the number of slots in the cluster and the resources held are determined at the start of the cluster and do not change; the tasks in the cluster are only scheduled once at task initialization, and after a task is assigned to a certain slot on a Worker node, no second assignment is made unless the task execution fails.

[0045] However, in actual cluster operation, different tasks have different requirements for core processors CPU, disks, memory, and other computing resources. In the static scheduling scheme, in order to meet the runtime requirements of most tasks, the resources of the slots need to be set according to the maximum resource requirement of all tasks in the entire task life cycle, which results in a great waste of computing resources. For example, there are 10 tasks in the cluster, each of which needs to be executed for 10 minutes, of which 9 tasks only need 1G of memory in the entire life cycle, and another task A needs 4G of memory for 1 minute in the 10-minute execution life cycle and only needs 1G of memory at other times. In order to meet the requirements of task A, all slots of the cluster must be set to 4G of memory, so that the cluster ultimately wastes 27G of memory.

[0046] In order to solve the technical problems existing in the prior art, the present application proposes a method, device, and system for dynamically adjusting task allocation, which can dynamically schedule tasks, and still monitor the operation of the hosts in the cluster in real time after allocation is completed, and dynamically adjust and allocate tasks, so that the host resources of the entire cluster reach a balanced state, and tasks will not appear "starvation" due to being unable to be allocated.

[0047] Figure 1 is a main step schematic diagram of the method for dynamically adjusting task allocation according to an embodiment of the present application. As shown in Figure 1 the method for dynamically adjusting task allocation according to an embodiment of the present application mainly includes the following steps S101 to S103.

[0048] Step S101: for each host included in the cluster, acquire performance parameters of the host in a timely manner;

[0049] Step S102: according to a preset rule and the performance parameters, determine whether the host is a to-be-scheduled host;

[0050] Step S103: in the case where the host is the to-be-scheduled host, determine a re-allocated task from tasks being executed by the host, end the re-allocated task, and add the re-allocated task to a task queue for re-task allocation.

[0051] According to the steps S101 to S103, the task scheduling algorithm is adopted to monitor the running status of the host in the cluster in a real-time manner during task execution, and dynamically adjust and allocate the tasks, so as to realize the optimal allocation of the resources of the cluster according to the actual demand of the tasks, and balance the use of the CPU, memory, disk and other resources of the entire cluster host.

[0052] In order to realize the method for dynamically adjusting task allocation, the application provides a system for dynamically adjusting task allocation, which comprises a task scheduling node and a host. The task scheduling node is configured to acquire performance parameters of each host included in a cluster in a timely manner, determine whether the host is a to-be-scheduled host according to a preset rule and the performance parameters, determine a re-allocated task from tasks being executed by the host in the case where the host is the to-be-scheduled host, end the re-allocated task, and add the re-allocated task to a task queue for re-task allocation. The host is configured to execute the allocated task.

[0053] According to the embodiment of the application, in the specific application process, the host can also be configured to collect resource usage data of the host in a timely manner and report the resource usage data to the task scheduling node. The task scheduling node can also be configured to determine the type of the host according to the resource usage data. On the other hand, the host can also be configured to calculate resource occupation data of the task being executed in a timely manner and report the resource occupation data to the task scheduling node. The task scheduling node can also be configured to determine the type of each task according to the resource occupation data.

[0054] The system for dynamically adjusting task allocation of the embodiment of the application will be described below with reference to the accompanying drawings. Figure 2 is a system architecture diagram of the task allocation system of the embodiment of the application. In the embodiment, the task scheduling node is specifically implemented as a Master node, and the host is specifically implemented as a Worker node. As shown in Figure 2As shown, the task allocation system of the embodiment of the application mainly consists of a Master node and a plurality of Worker nodes. Each Worker node can be run by a single host. The Master node mainly consists of a cluster resource management module, a cluster task scheduling module and a cluster task management module, and the Master node is mainly responsible for allocation and scheduling of cluster tasks. The Worker node mainly consists of a node resource acquisition module, a task resource calculation module and a task execution module, and the Worker node is responsible for execution of tasks and calculation and reporting of resource conditions. According to the technical solution of the application, the main dynamic task scheduling process is as follows:

[0055] 1. After the cluster is started, the node resource acquisition module of the Worker node acquires current resource usage data of the CPU, memory, disk and the like of the Worker node included in the cluster in a timely manner, and reports the resource usage data to the cluster resource management module of the Master node. The cluster resource management module of the Master node stores the latest resource usage data of all Worker nodes of the cluster, and can divide the Worker nodes into CPU type (with more CPU cores), memory type (with more memory), disk type (with more disk) and general type according to the available resource usage data. The general type Worker node is, for example, a host corresponding to the most frequently configured resource among the hosts included in the cluster. The CPU type, memory type and disk type are all divided relative to the general type, for example, the CPU type is a host with more CPU cores than the CPU cores of the general type host, and the memory type and disk type are the same division principle;

[0056] 2. The task resource calculation module of the Worker node is responsible for calculating resource occupation data of the tasks running on the node in a timely manner (for example, 10 seconds, which can be flexibly set according to application needs), and reporting the resource occupation data to the cluster task management module of the Master node. The cluster task management module of the Master node is responsible for maintaining the latest resource occupation data of the tasks running on the Worker node, and dividing the tasks into CPU consumption type, memory consumption type, disk consumption type and balanced type according to the resource occupation data. When dividing the task types, a pre-set threshold value can be used for division, for example, a CPU consumption type task can be defined as a task whose host CPU usage rate exceeds 10% during execution, and the like;

[0057] 3. When the Master node obtains a task from a task queue, the cluster task scheduling module of the Master node uses a first-stage (task allocation stage) task scheduling algorithm to select an optimal Worker node for the task, and allocates the task to the Worker node for execution;

[0058] 4. The cluster task scheduling module of the master node uses a second stage (task execution stage) task scheduling algorithm at regular intervals (for example, 3 minutes, which can be flexibly set as required) to find a set of worker nodes with tight resources, and re-allocates tasks of these worker nodes to make the use of the entire cluster resources reach a relatively balanced state.

[0059] The dynamic allocation of tasks using the task allocation system of the embodiment of the present application not only adopts a two-stage task scheduling algorithm of a task allocation stage and a task execution stage, but also realizes optimal allocation of cluster resources according to actual requirements of tasks, so that the use rates of CPU, memory, and disk resources of the entire cluster host are at a balanced level. At the same time, the most suitable running node can be selected for different task types (for example, including CPU consumption type, memory consumption type, disk consumption type, etc.), so as to achieve the effect of efficient running of tasks, higher utilization of cluster resources, and more stable running state of the cluster.

[0060] According to one embodiment of the present application, whether a task that has not been allocated before or a task that has been allocated but is determined to be re-allocated, needs to be allocated in the following manner. Specifically, when allocating a task, a to-be-allocated task can be obtained from a task queue, and a host is selected for the to-be-allocated task according to task information of the to-be-allocated task, and the to-be-allocated task is allocated to the host. Wherein, when selecting a host for the to-be-allocated task according to the task information of the to-be-allocated task, the following steps can be included:

[0061] According to the task information, it is determined whether the to-be-allocated task is a CPU re-allocated task;

[0062] If yes, the hosts included in the cluster are sorted according to available CPU resource size; otherwise, the hosts included in the cluster are sorted according to available process number proportion;

[0063] A host is selected from the sorted hosts.

[0064] In another embodiment of the present application, when selecting a host from the sorted hosts, the sorted hosts can be traversed in sequence, and a host is selected according to available process number, available disk space, CPU load, and remaining memory of each host.

[0065] In yet another embodiment of the present application, a host is selected according to available process number, available disk space, CPU load, and remaining memory of each host, which can include the following steps:

[0066] For each host, it is determined whether the available process number of the host is greater than the required process number of the to-be-allocated task;

[0067] If yes, it is judged whether the to-be-allocated task is a disk consumption type task;

[0068] If yes, it is judged whether the CPU load of the host is less than a second threshold value when the available disk space of the host is greater than a first threshold value; otherwise, it is directly judged whether the CPU load of the host is less than the second threshold value;

[0069] If yes, the host is selected as a selected host when the remaining memory of the host is greater than the required memory of the to-be-allocated task; otherwise, the host is added to a to-be-scheduled host list.

[0070] If no host is selected after the traversal, the host with the lowest CPU load in the to-be-scheduled host list is selected as the selected host.

[0071] In the above embodiments, the first threshold value and the second threshold value can be flexibly set according to application requirements, which does not affect the implementation of the technical solution of the present application.

[0072] The implementation process of the task allocation stage of the embodiment of the present application will be described below. Figure 3 The implementation process of the task allocation stage of the embodiment of the present application will be described below. Figure 3 is a schematic diagram of the implementation process of the task allocation stage of the embodiment of the present application. As shown in Figure 3 After a to-be-allocated task is obtained from the task queue, the task allocation is performed according to the following process:

[0073] 1. The resource usage data corresponding to the hosts in the cluster where the task is located is obtained according to the task, and each host is a Worker node.

[0074] 2. The hosts are sorted according to the task information. The sorting manner of the hosts corresponding to different task information is different. Specifically:

[0075] (1) If the CPU reallocation task identifier is included in the task information (i.e., the task is a CPU reallocation task), it means that the task is a reallocation task triggered by the cluster scheduling mechanism for dynamically adjusting task allocation, and is a reallocation task determined due to the high CPU load of the host. At this time, the hosts need to be sorted according to the available CPU resources;

[0076] (2) If the CPU reallocation task identifier is not contained in the task information (i.e., the task is not a CPU reallocation task), it is indicated that the task is a task submitted for the first time, or a task automatically retried due to execution failure for other reasons, or a disk-consuming task (the information is contained in the task information) with large disk occupancy, at this time, the host needs to be sorted according to the available process number ratio. The available process number ratio of the host is the available slot ratio, each slot can be understood as a process, and each slot can execute a task, but a task may need multiple slots to execute, i.e., a task needs one or more processes to execute;

[0077] 3. Iterating the sorted hosts;

[0078] 4. Taking out a host, and executing step 5, if there is no host, it is indicated that all hosts have been iterated and no host meeting the requirements is found, at this time, step 12 is executed;

[0079] 5. Judging whether the available process number of the host is greater than the process number required by the task, if yes, step 6 is executed; otherwise, step 4 is executed;

[0080] 6. Judging whether the task is a disk-consuming task, if yes, step 7 is executed, otherwise, step 8 is executed;

[0081] 7. Checking whether the available disk space of the host is greater than a first threshold, usually, for a host, when the available disk space is less than a certain threshold (for example, 20%), the task should not be allocated, otherwise, the operation of the host will be affected. If yes, step 8 is executed, otherwise, step 4 is executed;

[0082] 8. Checking whether the CPU load of the host is less than a second threshold, wherein the second threshold can be represented as: core number*f1, f1 is a definable threshold, usually, for a host, when the CPU load reaches a certain threshold (for example, core number*0.6), the task should not be allocated, otherwise, the operation of the host will be affected. If yes, step 9 is executed, otherwise, step 11 is executed;

[0083] 9. Checking whether the remaining memory of the host is greater than the required memory of the task, if yes, step 10 is executed, otherwise, step 4 is executed;

[0084] 10. Task allocation is performed, and the process ends;

[0085] 11. The host is added to the list of hosts to be scheduled, and then step 4 is executed;

[0086] 12. Sort the list of hosts to be scheduled and select the host with the lowest load from the list for task allocation. This can prevent the task from "starving" due to excessive cluster pressure and the inability to be allocated tasks.

[0087] according to Figure 3 The illustrated embodiment allows for the selection of a more suitable host based on the different types of tasks, thereby achieving efficient task execution, higher cluster resource utilization, and a more stable cluster operation. Furthermore, since the cluster host's memory is not pre-allocated, the remaining memory on the host can be checked to dynamically adjust the memory allocated to tasks based on the minimum and maximum memory requirements configured for each process, thus improving the cluster's memory resource utilization.

[0088] According to the technical solution of the present invention, after the task allocation is completed, during the task execution process, it is still necessary to periodically search for the set of hosts to be scheduled in the cluster, determine the reassigned task from the tasks being executed by each host to be scheduled, end the reassignment task, and add the reassigned task to the task queue for reassignment.

[0089] In one embodiment of the present invention, for each host in the cluster, performance parameters of the host are periodically acquired, wherein the performance parameters include, for example, the host's CPU load and available disk space. Then, based on preset rules and the performance parameters, it is determined whether the host is a host to be scheduled. Specifically, the determination of whether a host is a host to be scheduled can be performed according to the following steps:

[0090] The host is determined to be a host to be scheduled based on its CPU load and available disk space. Specifically, if the CPU load of the host is greater than a third threshold and the duration reaches a first time threshold, then the host is designated as a host to be scheduled. If the available disk space of the host is less than a fourth threshold and the duration reaches a second time threshold, then the host is designated as a host to be scheduled.

[0091] According to one embodiment of the present invention, the state type of the host to be scheduled includes high disk load and high CPU load. Furthermore, determining the reallocation task from the tasks being executed by the host may specifically include:

[0092] Determine whether the host's status type is disk high load; where host status type refers to the specific overload status type when the current host is in an overload state;

[0093] If so, the task with the highest disk usage currently being executed by the host is identified as the reallocation task;

[0094] Otherwise, the step of determining the reallocation task according to the CPU usage of the task being executed by the host.

[0095] According to still another embodiment of the present application, the step of determining the reallocation task according to the CPU usage of the task being executed by the host can specifically include:

[0096] acquiring the task with the largest CPU usage and judging whether the CPU usage of the task exceeds a fifth threshold value;

[0097] If yes, for each task being executed by the host except the task with the largest CPU usage, if the task has not been scheduled in a recent period, the task is determined as a reallocation task, and a CPU reallocation task identifier is added to the task.

[0098] Otherwise, each task being executed by the host is traversed in sequence, and for each task, the following operation is performed until the CPU usage of the host is less than the fifth threshold value: if the task has not been scheduled in a recent period, the task is determined as a reallocation task, and a CPU reallocation task identifier is added to the task.

[0099] The implementation process of the task execution stage of the embodiment of the present application will be described below in combination with Figure 4 and Figure 5 The implementation process of the task execution stage of the embodiment of the present application will be described below in combination with Figure 4 is a flowchart of the implementation process of the task execution stage of the embodiment of the present application. Figure 5 is a flowchart of the implementation process of the task execution stage of the embodiment of the present application. The cluster resource management module of the Master node will perform load checking and determine the reallocation task at regular time intervals.

[0100] As shown in Figure 4 , the main process of the cluster resource management module of the Master node for regularly searching for the set of host to be scheduled from the cluster is as follows:

[0101] 1. Traversing the hosts included in the cluster;

[0102] 2. Taking out a host, and performing step 3, if there is no host to be traversed, performing step 7;

[0103] 3. Checking whether the CPU load of the host is greater than a third threshold value (the third threshold value can be the same as the second threshold value, or can be different) and the duration reaches a first time threshold value (for example, 3 minutes), wherein the third threshold value can be represented as core number*f2, f2 is a definable threshold value, if yes, performing step 5, otherwise, performing step 4;

[0104] 4. Check if the available disk space of the host is less than a fourth threshold (which can be the same as the first threshold or different) and the duration reaches a second time threshold (for example, 3 minutes). If yes, execute step 6, otherwise execute step 2;

[0105] 5. Add the host to the list of hosts to be scheduled and set the state type as CPU high load. At this time, since the host is in the state of CPU load overload, the state type corresponding to the host is recorded as CPU high load;

[0106] 6. Add the host to the list of hosts to be scheduled and set the state type as disk high load. At this time, since the host is in the state of disk load overload, the state type corresponding to the host is recorded as disk high load;

[0107] 7. Check end.

[0108] As shown in Figure 5 After obtaining the hosts to be scheduled, the cluster task scheduling module of the Master node can analyze the tasks being executed by the hosts to be scheduled to determine the tasks that need to be redistributed. The main process is as follows:

[0109] 1. Traverse the hosts to be scheduled;

[0110] 2. Take out a host to be scheduled and execute step 3. If there is no host to be traversed, execute step 14;

[0111] 3. Determine whether the state type of the host to be scheduled is disk high load. If yes, execute step 4, otherwise execute step 5;

[0112] 4. Find the task with the largest disk occupancy of the host to be scheduled and determine it as a task to be redistributed. End the task (disk space can only be recycled by ending the process to release disk space), and put it into the task queue. Then execute step 2;

[0113] 5. Sort the tasks being executed by the host to be scheduled according to CPU usage;

[0114] 6. Get the task with the highest CPU usage and determine whether it meets the condition of CPU usage > fifth threshold, where the fifth threshold is for example 50%. It should be noted that if the CPU usage of the task is greater than 50%, placing it on any host to execute will cause the load of the host to be high, so the task will not be killed, but other tasks will be processed. If yes, execute step 7, otherwise execute step 10;

[0115] 7. Traverse all the remaining tasks except the task with the highest CPU usage. If there is no task to be traversed, execute step 2;

[0116] 8. judging whether the task is scheduled in the recent period, wherein the recent period is for example in the last 10 minutes, and being scheduled means that the task is originally executed on a host A and is reassigned to be executed on the host B. If the task is scheduled in the recent period, the scheduling is not performed again, preventing some tasks from being frequently scheduled and affecting the execution efficiency and wasting system resources. If yes, step 7 is performed, otherwise step 9 is performed;

[0117] 9. determining the task as a reassignment task, adding a CPU reassignment task identifier to the task, ending the task and putting the task into a task queue, and then performing step 7;

[0118] 10. sequentially traversing all tasks being executed on the host to be scheduled, and if there is no task to be traversed, step 2 is performed;

[0119] 11. judging whether the task taken out is scheduled in the recent period, if yes, step 10 is performed, otherwise step 12 is performed;

[0120] 12. determining the task as a reassignment task, adding a CPU reassignment task identifier to the task, ending the task and putting the task into a task queue, and then performing step 13;

[0121] 13. judging whether the CPU usage of the host to be scheduled is less than the fifth threshold, if yes, step 14 is performed, otherwise step 10 is performed;

[0122] 14. ending the scheduling.

[0123] According to the embodiments shown in Figure 4 and Figure 5 , the running data of the cluster host can be dynamically obtained and the dynamic scheduling of the task can be performed in the task execution stage, so that the flow data tasks in the cluster are more evenly distributed and can be stably run, and the CPU resource utilization of the cluster is improved.

[0124] In all the embodiments of the present application, the thresholds appeared can be flexibly set according to the application needs, which does not affect the implementation of the technical solutions of the present application.

[0125] According to another aspect of the present application, a device for dynamically adjusting task allocation is provided. Figure 6 is a schematic diagram of main modules of the device for dynamically adjusting task allocation according to the embodiments of the present application. As shown in Figure 6 , the device 600 for dynamically adjusting task allocation according to the embodiments of the present application mainly includes a host parameter obtaining module 601, a host state judging module 602 and a task allocation adjusting module 603.

[0126] The host parameter acquisition module 601 is configured to acquire performance parameters of each host included in the cluster in a timely manner.

[0127] The host state determination module 602 is configured to determine whether the host is a to-be-scheduled host according to a preset rule and the performance parameters.

[0128] The task allocation adjustment module 603 is configured to determine a reallocation task from tasks being executed by the host in a case where the host is the to-be-scheduled host, end the reallocation task, and add the reallocation task to a task queue to perform task allocation again.

[0129] According to an embodiment of the present application, the task allocation adjustment module 603 can also be configured to acquire a to-be-allocated task from the task queue and select a host for the to-be-allocated task according to task information of the to-be-allocated task when performing task allocation, and allocate the to-be-allocated task to the host.

[0130] According to an embodiment of the present application, the task allocation adjustment module 603 can also be configured to:

[0131] determine whether the to-be-allocated task is a CPU reallocation task according to the task information;

[0132] if yes, sort the hosts included in the cluster according to available CPU resource sizes; or if not, sort the hosts included in the cluster according to available process number proportions;

[0133] select a host from the sorted hosts.

[0134] According to another embodiment of the present application, the task allocation adjustment module 603 can also be configured to:

[0135] traverse the sorted hosts in sequence and select a host according to available process numbers, available disk spaces, CPU loads, and remaining memories of each host.

[0136] According to still another embodiment of the present application, the task allocation adjustment module 603 can also be configured to:

[0137] for each host, determine whether an available process number of the host is greater than a process number required by the to-be-allocated task;

[0138] if yes, determine whether the to-be-allocated task is a disk consumption task;

[0139] if yes, determine whether a CPU load of the host is less than a second threshold value when an available disk space of the host is greater than a first threshold value; or if not, directly determine whether the CPU load of the host is less than the second threshold value.

[0140] If yes, the host is selected as the selected host if the remaining memory of the host is greater than the memory required by the to-be-allocated task; otherwise, the host is added to the to-be-scheduled host list.

[0141] If no host is selected after the traversal, the host with the lowest CPU load in the to-be-scheduled host list is selected as the selected host.

[0142] According to still another embodiment of the present application, the performance parameter includes CPU load and available disk space; the host state judging module 602 can be further configured to:

[0143] determine whether the host is a to-be-scheduled host according to the CPU load and the available disk space of the host, wherein the host is determined as a to-be-scheduled host if the CPU load of the host is greater than a third threshold value and the duration reaches a first time threshold value, or if the available disk space of the host is less than a fourth threshold value and the duration reaches a second time threshold value.

[0144] According to still another embodiment of the present application, the state type of the to-be-scheduled host includes disk high load and CPU high load; the task allocation adjusting module 603 can be further configured to:

[0145] determine whether the state type of the host is disk high load.

[0146] If yes, the task with the largest disk occupancy being executed by the host is determined as the reallocation task.

[0147] Otherwise, the reallocation task is determined according to the CPU usage of the task being executed by the host.

[0148] According to still another embodiment of the present application, the task allocation adjusting module 603 can be further configured to:

[0149] obtain the task with the largest CPU usage and determine whether the CPU usage of the task exceeds a fifth threshold value;

[0150] If yes, for each task being executed by the host except the task with the largest CPU usage, if the task has not been scheduled in a recent period, the task is determined as the reallocation task and a CPU reallocation task identifier is added to the task.

[0151] Otherwise, each task being executed by the host is traversed in sequence, and for each task, the following operations are performed until the CPU usage of the host is less than the fifth threshold value: if the task has not been scheduled in a recent period, the task is determined as the reallocation task and a CPU reallocation task identifier is added to the task.

[0152] According to the technical scheme of the embodiment of the present application, the performance parameters of each host included in the cluster are acquired in time, and whether the host is a to-be-scheduled host is determined according to the preset rule and the performance parameters. In the case that the host is a to-be-scheduled host, the reassignment task is determined from the task being executed by the host, the reassignment task is ended and added to the task queue for re-tasking, the task scheduling algorithm that still monitors the running of the host in the cluster in real time during the task execution and dynamically adjusts and allocates the task is adopted, the optimal allocation of the resources of the cluster according to the actual demand of the task is realized, the use rate of the CPU, the memory and the disk of the entire cluster host is at a balanced level, the flow data task distribution in the cluster is more balanced and can be stably run, and the CPU resource utilization of the cluster is improved.

[0153] Figure 7 An exemplary system architecture 700 to which the method for dynamically adjusting task allocation or the apparatus for dynamically adjusting task allocation of the embodiments of the present application can be applied is shown.

[0154] As shown in Figure 7 The system architecture 700 can include terminal devices 701, 702, 703, a network 704 and a server 705. The network 704 is used to provide a communication link medium between the terminal devices 701, 702, 703 and the server 705. The network 704 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0155] The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 701, 702, 703, such as data processing applications, search applications, flow data processing tools, task scheduling tools, instant messaging tools, mailbox clients, social platform software, etc. (only as examples).

[0156] The terminal devices 701, 702, 703 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0157] The server 705 can be a server providing various services, for example, a background management server providing support for a website browsed by a user using the terminal device 701, 702, 703 (only as an example). The background management server can perform the following processing on received task allocation request data and the like: acquiring performance parameters of each host included in a cluster in a time manner; determining whether the host is a to-be-scheduled host according to a preset rule and the performance parameters; in a case where the host is the to-be-scheduled host, determining a re-allocation task from tasks being executed by the host, ending the re-allocation task, adding the re-allocation task to a task queue for re-task allocation, and the like, and feeding back a processing result (for example, a task allocation result--only as an example) to the terminal device.

[0158] It should be noted that the method for dynamically adjusting task allocation provided by the embodiment of the present application is generally executed by the server 705, and accordingly, the device for dynamically adjusting task allocation is generally arranged in the server 705.

[0159] It should be understood that, Figure 7 The number of terminal devices, networks and servers in the above-mentioned system is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

[0160] Reference will be made to Figure 8 which shows a structural schematic diagram of a computer system 800 suitable for implementing the terminal device or server of the embodiment of the present application. Figure 8 The terminal device or server shown is only an example, and should not bring any limitation to the function and use range of the embodiment of the present application.

[0161] As shown in Figure 8 , the computer system 800 includes a central processing unit (CPU) 801 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or programs loaded from a storage portion 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802 and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0162] The following components are connected to the I / O interface 805: an input part 806 including a keyboard, a mouse, etc.; an output part 807 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 808 including a hard disk, etc.; and a communication part 809 including a network interface card such as a LAN card, a modem, etc. The communication part 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 810 as necessary, so that a computer program read out therefrom is installed in the storage part 808 as necessary.

[0163] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-described functions defined in the system of the present disclosure are executed.

[0164] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0165] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0166] The units or modules described in the embodiments of the present application can be implemented by software or hardware. The described units or modules can also be arranged in a processor, for example, a processor can be described as including a host parameter acquisition module, a host state determination module and a task allocation adjustment module. In some cases, the names of these units or modules do not constitute a limitation on the units or modules themselves, for example, the host parameter acquisition module can also be described as a module for periodically acquiring performance parameters of each host included in a cluster.

[0167] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to include: periodically acquiring performance parameters of each host included in a cluster; determining whether the host is a to-be-scheduled host according to a preset rule and the performance parameters; in the case that the host is a to-be-scheduled host, determining a reallocation task from a task being executed by the host, ending the reallocation task and adding the reallocation task to a task queue for re-task allocation.

[0168] According to the technical solution of the embodiments of the present application, by periodically acquiring performance parameters of each host included in a cluster, determining whether the host is a to-be-scheduled host according to a preset rule and the performance parameters, and in the case that the host is a to-be-scheduled host, determining a reallocation task from a task being executed by the host, ending the reallocation task and adding the reallocation task to a task queue for re-task allocation, the task scheduling algorithm that still monitors the running status of the hosts in the cluster in the process of task execution and dynamically adjusts and allocates tasks is adopted, the optimal allocation of resources of the cluster according to actual needs of tasks is realized, the use rates of CPU, memory and disk resources of the entire cluster host are at a balanced level, the distribution of stream data tasks in the cluster is more balanced and can be stably run, and the CPU resource utilization rate of the cluster is improved.

[0169] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made depending on design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method of dynamically adjusting task allocation, characterized by, The method comprises: acquiring performance parameters of each host included in the cluster in a timely manner; judging whether the host is a to-be-scheduled host according to preset rules and the performance parameters; in the case that the host is a to-be-scheduled host, determining a reassignment task from tasks being executed by the host, ending the reassignment task and adding the reassignment task to a task queue for re-task scheduling; wherein the state type of the to-be-scheduled host comprises disk high load and CPU high load, and the state type of the host is a type of overload state to which the host has been in; determining the reassignment task from the tasks being executed by the host comprises: judging whether the state type of the host is disk high load; if yes, determining a task with the largest disk occupancy being executed by the host as the reassignment task; otherwise, determining the reassignment task according to CPU usage of the tasks being executed by the host; determining the reassignment task according to the CPU usage of the tasks being executed by the host comprises: acquiring a task with the largest CPU usage and judging whether the CPU usage of the task is greater than a fifth threshold; if yes, for each task being executed by the host except the task with the largest CPU usage, if the task has not been scheduled in a recent period, determining the task as the reassignment task and adding a CPU reassignment task identifier to the task, wherein the task has been scheduled refers to that the task was originally executed on a host other than the host and has been reassigned to the host, and the CPU reassignment task identifier is used to identify that the task is a reassignment task determined due to excessive CPU load of the host; otherwise, sequentially traversing each task being executed by the host and performing the following operation on each task until the CPU usage of the host is less than the fifth threshold: if the task has not been scheduled in the recent period, determining the task as the reassignment task and adding the CPU reassignment task identifier to the task.

2. The method of claim 1, wherein, performing task scheduling comprises: acquiring a to-be-scheduled task from the task queue and selecting a host for the to-be-scheduled task according to task information of the to-be-scheduled task, and scheduling the to-be-scheduled task to the host.

3. The method of claim 2, wherein, selecting the host for the to-be-scheduled task according to the task information of the to-be-scheduled task comprises: judging whether the to-be-scheduled task is a CPU reassignment task according to the task information; if yes, sorting the hosts included in the cluster according to available CPU resource size; otherwise, sorting the hosts included in the cluster according to available process number proportion; selecting a host from the sorted hosts.

4. The method of claim 3, wherein, selecting the host from the sorted hosts comprises: sequentially traversing the sorted hosts and selecting a host according to available process number, available disk space, CPU load and remaining memory of each host.

5. The method of claim 4, wherein, selecting the host according to available process number, available disk space, CPU load and remaining memory of each host comprises: for each host, judging whether the available process number of the host is greater than the process number required by the to-be-scheduled task; if yes, judging whether the to-be-scheduled task is a disk-consuming task; If yes, when available disk space of the host is greater than a first threshold, it is judged whether CPU load of the host is less than a second threshold; otherwise, it is directly judged whether CPU load of the host is less than the second threshold. If yes, when remaining memory of the host is greater than memory required by the task to be allocated, the host is selected as a selected host; otherwise, the host is added to a list of hosts to be scheduled. If no host is selected after the traversal, a host with the lowest CPU load in the list of hosts to be scheduled is selected as the selected host.

6. The method of claim 1, wherein, The performance parameters include CPU load and available disk space. According to the preset rule and the performance parameters, it is judged whether the host is a host to be scheduled, comprising: According to CPU load and available disk space of the host, it is judged whether the host is a host to be scheduled, wherein if CPU load of the host is greater than a third threshold and a duration reaches a first time threshold, the host is selected as a host to be scheduled; if available disk space of the host is less than a fourth threshold and a duration reaches a second time threshold, the host is selected as a host to be scheduled.

7. An apparatus for dynamically adjusting task allocation, the apparatus comprising: Comprising: A host parameter acquisition module is configured to acquire performance parameters of each host included in a cluster in a timely manner; A host state judgment module is configured to judge whether the host is a host to be scheduled according to a preset rule and the performance parameters; A task allocation adjustment module is configured to, in a case where the host is a host to be scheduled, determine a reallocation task from tasks being executed by the host, end the reallocation task and add the reallocation task to a task queue for task allocation again; wherein a state type of a host to be scheduled includes disk high load and CPU high load, and a state type of a host is a type of an overload state to which the host has been in currently; The task allocation adjustment module is further configured to judge whether the state type of the host is disk high load; if yes, a task with the largest disk occupancy being executed by the host is determined as the reallocation task; otherwise, a reallocation task is determined according to CPU usage of tasks being executed by the host. The task allocation adjustment module is further configured to: acquire a task with the largest CPU usage, and determine whether the CPU usage of the task exceeds a fifth threshold; if yes, for each task being executed by the host except the task with the largest CPU usage, if the task has not been scheduled in a recent period, the task is determined as a reallocation task, and a CPU reallocation task identifier is added to the task, wherein the task has been scheduled refers to that the task was originally executed on a host other than the host and has been reallocated to the host, and the CPU reallocation task identifier is used to identify that the task is a reallocation task determined due to excessive CPU load of the host; otherwise, each task being executed by the host is sequentially traversed, and the following operations are performed on each task until the CPU usage of the host is less than the fifth threshold: if the task has not been scheduled in a recent period, the task is determined as a reallocation task, and a CPU reallocation task identifier is added to the task.

8. A system for dynamically adjusting task allocation, characterized by The method comprises: a task scheduling node configured to acquire, for each host included in the cluster, a performance parameter of the host; determine, according to a preset rule and the performance parameter, whether the host is a to-be-scheduled host; in a case where the host is the to-be-scheduled host, determine a reallocation task from tasks being executed by the host, end the reallocation task, and add the reallocation task to a task queue for task allocation again; wherein a state type of the to-be-scheduled host includes disk high load and CPU high load, and the state type of the host is a type of overload state to which the host currently has been in; the determining of the reallocation task from the tasks being executed by the host includes: determining whether the state type of the host is the disk high load; if yes, determining a task with the largest disk occupancy from the tasks being executed by the host as the reallocation task; otherwise, determining the reallocation task according to CPU usage of the tasks being executed by the host; the determining of the reallocation task according to the CPU usage of the tasks being executed by the host includes: acquiring a task with the largest CPU usage, and determining whether the CPU usage of the task exceeds a fifth threshold; if yes, for each task being executed by the host except the task with the largest CPU usage, if the task has not been scheduled in a recent period, the task is determined as a reallocation task, and a CPU reallocation task identifier is added to the task, wherein the task has been scheduled refers to that the task was originally executed on a host other than the host and has been reallocated to the host, and the CPU reallocation task identifier is used to identify that the task is a reallocation task determined due to excessive CPU load of the host; otherwise, each task being executed by the host is sequentially traversed, and the following operations are performed on each task until the CPU usage of the host is less than the fifth threshold: if the task has not been scheduled in a recent period, the task is determined as a reallocation task, and a CPU reallocation task identifier is added to the task; a host configured to execute the allocated task.

9. The system of claim 8, wherein, the host is further configured to collect current resource usage data of itself at a regular time and report the data to the task scheduling node; the task scheduling node is further configured to determine the type of the host according to the resource usage data.

10. The system of claim 8, wherein, the host is further configured to calculate resource occupation data of the task being executed at a regular time and report the data to the task scheduling node; the task scheduling node is further configured to determine the type of each task according to the resource occupation data.

11. An electronic device for dynamically adjusting task allocation, the electronic device comprising: comprising: one or more processors; a memory device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-6.

12. A computer readable medium having stored thereon a computer program, characterized in that, the program is executed by the processor to implement the method of any one of claims 1-6.

Images