A task scheduling method and device

Abstract

The application provides a task scheduling method and device, electronic equipment and computer program product, relates to the field of information technology, and the method comprises the following steps: acquiring the resource usage of a host; predicting the available host resources at a future target time; acquiring a task to be scheduled, constructing a prediction model based on a host resource weight set, and acquiring the required host resource proportion of the task to be scheduled by using the prediction model; calculating the task weight of the task to be scheduled according to the host resource weight set; according to the required host resource proportion and the task weight, calculating the first residual resource after resource allocation of the current available host resources and the second residual resource after resource allocation of the available host resources at a future time; when it is judged that the second residual resource is large, scheduling the task according to the time limit requirement of each task to be scheduled. The application can ensure the balanced use of host resources while ensuring the consistency of the task scheduling result, and effectively improves the utilization rate of resources.

Description

Technical Field

[0001] This application relates to the field of information technology, specifically to a task scheduling method, apparatus, electronic device, and computer program product. Background Technology

[0002] As service providers modernize their networks, their workloads and services are moving from the core network (located within the data center) to the network edge. In NFV's distributed task scheduling, consistency in task scheduling is crucial. Common solutions include relative balancing based on the number of hosts and tasks, and comprehensive balancing based on resources and tasks. However, these solutions mostly rely on distributed scheduling schemes using management nodes and scheduling agents, such as Kubernetes, which requires application adaptation. Another approach involves assigning the hash values ​​of execution units and tasks to a consistent hash ring. This ring then finds the hash values ​​of the n nearest corresponding execution units to the task's hash value, enabling flexible matching between execution units and tasks. The same task can be assigned to different execution units, satisfying the requirement for multiple executions. However, this task scheduling scheme doesn't consider the host resources of each host node, leading to resource imbalances after task startup.

[0003] In summary, existing technologies have the following drawbacks: the scheduling task algorithm is complex, and the use of specific hash mixing operations to randomly assign scheduling task host nodes or the development of closed scheduling strategies based on open source products (such as ZooKeeper and Kubernetes) makes it impossible to predict the scheduling task results, thus failing to guarantee the balanced use of host resources. Summary of the Invention

[0004] This application provides a task scheduling method, apparatus, electronic device, and computer program product to solve the above-mentioned technical problems, and can ensure the consistency of task scheduling results and the balanced use of host resources.

[0005] In a first aspect, embodiments of this application provide a task scheduling method, including:

[0006] The system acquires real-time resource usage information reported by each host in the distributed system; wherein the resource usage information includes the available host resources at the current moment.

[0007] Based on the resource usage and current task execution information, the available host resources at the future target time are predicted;

[0008] All tasks to be scheduled within the time period between the current time and the future target time are obtained. A prediction model is constructed based on the pre-obtained set of host resource weights, and the required host resource ratio for each task to be scheduled is obtained using the prediction model.

[0009] Calculate the task weight of the task to be scheduled based on the host resource weight set;

[0010] Based on the required host resource ratio and the task weight, calculate the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time.

[0011] When it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled.

[0012] In one embodiment, predicting the available host resources at a future target time based on the resource usage and current task execution information includes:

[0013] Based on the current task running information, tasks that will end before the future target time are selected, and the resource usage of the tasks is recorded.

[0014] Based on the resource occupancy and usage, the available host resources at the future target time are predicted.

[0015] In one embodiment, the method for obtaining the host resource weight set includes:

[0016] Obtain the observation values ​​of historical tasks within a preset historical time range, as well as the resource utilization rate of each host corresponding to the historical tasks, and construct a mapping matrix between the resource utilization rate of each host and the historical tasks;

[0017] Based on the mapping matrix, the predicted resource values ​​of each host at the current moment are calculated using a preset exponential smoothing algorithm.

[0018] The host resource weight set is obtained by weighted averaging the predicted values ​​of each host resource and the preset weight values ​​of each host resource.

[0019] In one embodiment, calculating the task weight of the task to be scheduled based on the host resource weight set includes:

[0020] The secondary weight of each host is obtained by calculating the weight of each host resource in the host resource weight set using a preset secondary algorithm.

[0021] The maximum weight of the cluster is obtained by summing the quadratic weights of each host.

[0022] The task weight of the task to be scheduled is calculated by combining the maximum weight of the cluster and the random result obtained by the host.

[0023] In one embodiment, when it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled, specifically including:

[0024] When it is determined that the first priority resource in the second remaining resources is greater than the first priority resource in the first remaining resources, task scheduling is performed according to the time limit requirements of each task to be scheduled.

[0025] In one embodiment, before obtaining the host resource weights, the method further includes:

[0026] The weight values ​​of each host resource are adaptively adjusted based on the available host resources at the future target time.

[0027] In one embodiment, the host's resource types include one or more of MEM resources, CPU resources, memory resources, IO resources, file handle resources, and file descriptor resources.

[0028] Secondly, embodiments of this application provide a task scheduling device, comprising:

[0029] The data acquisition module is used to acquire the resource usage information reported by each host in the distributed system in real time; wherein, the resource usage information includes the available host resources at the current moment;

[0030] The resource prediction module is used to predict the available host resources at a future target time based on the resource usage and current task running information.

[0031] The demand forecasting module is used to obtain all scheduled tasks within the time period between the current time and the future target time, construct a forecasting model based on the pre-obtained set of host resource weights, and use the forecasting model to obtain the required host resource ratio for each scheduled task.

[0032] The weight calculation module is used to calculate the task weight of the task to be scheduled based on the host resource weight set;

[0033] The resource calculation module is used to calculate, based on the required host resource ratio and the task weight, the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time.

[0034] The task scheduling module is used to schedule tasks according to the time limit requirements of each task to be scheduled when it is determined that the second remaining resource is greater than the first remaining resource.

[0035] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the task scheduling method described in the first aspect.

[0036] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the task scheduling method described in the first aspect.

[0037] The task scheduling method, apparatus, electronic device, and computer program product provided in this application predict the available resources at a future target time based on the resources reported by the host in real time and the task running information. At the same time, it predicts the resources required by the task to be scheduled, and then allocates resources by combining the calculated task weights and predicts the remaining resources in the future. Then, it schedules the task based on the remaining resources and the time limit requirements of the task to be scheduled. This can ensure the consistency of task scheduling results while ensuring the balanced use of host resources, and effectively improve the utilization rate of resources. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is one of the flowcharts illustrating the task scheduling method provided in the embodiments of this application;

[0040] Figure 2 This is a second flowchart illustrating the task scheduling method provided in the embodiments of this application;

[0041] Figure 3 This is a schematic diagram of the structure of the task scheduling device provided in the embodiments of this application;

[0042] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] Figure 1 This is a flowchart illustrating the task scheduling method. (Refer to...) Figure 1 This application provides a task scheduling method, which may include the following steps:

[0045] S1. Obtain the resource usage information reported by each host in the distributed system in real time; wherein, the resource usage information includes the available host resources at the current moment. The host resource types include one or more of MEM resources, CPU resources, memory resources, IO resources, file handle resources, and file descriptor resources.

[0046] Understandably, in a distributed system, each host can periodically report its resource usage at minute intervals, including information such as host IP, CPU utilization, memory utilization, IO utilization, and file handle utilization. This data can be stored in a database, with one data point collected every minute.

[0047] S2. Based on the resource usage and current task operation information, predict the available host resources at the future target time.

[0048] Understandably, based on real-time resource usage data and the running information of currently running tasks, it is possible to predict the available host resources at a future target time. For example, each task includes information such as start time, current resource usage percentage, running cycle, and expected end time.

[0049] S3. Obtain all tasks to be scheduled within the time period between the current time and the future target time, construct a prediction model based on the pre-obtained set of host resource weights, and use the prediction model to obtain the required host resource ratio for each task to be scheduled.

[0050] It should be noted that, for all tasks to be scheduled in the future, the required proportion of each resource for each task can be determined by using a predictive model built based on the pre-acquired set of host resource rights.

[0051] S4. Calculate the task weight of the task to be scheduled based on the host resource weight set.

[0052] It should be noted that, based on the host resource weight set, the task weight of the scheduled task can be further obtained using a preset algorithm model. Since each host uses the same calculation model, each task can obtain the same task weight on each host.

[0053] S5. Based on the required host resource ratio and the task weight, calculate the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time.

[0054] It should be noted that, based on the available host resources at the current moment, the first remaining resource can be calculated by allocating host resources to multiple tasks to be scheduled according to the required host resource ratio and task weight. Similarly, the second remaining resource can be calculated based on the available host resources at the future target time.

[0055] S6. When it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled.

[0056] Compare the size of the first remaining resource and the second remaining resource. If the second remaining resource is larger, determine the urgency of the tasks to be scheduled. If one or more tasks to be scheduled can withstand the corresponding delay, the allocation method based on the available host resources at the future target time can be adopted to allocate resources to multiple tasks to be scheduled, and then schedule the tasks according to the resource allocation.

[0057] The task scheduling method provided in this application predicts the available resources at a future target time based on the resources reported by the host in real time and the task running information. At the same time, it predicts the resources required by the task to be scheduled, and then allocates resources by combining the calculated task weights and predicts the remaining resources in the future. Then, it schedules the task based on the remaining resources and the time limit requirements of the task to be scheduled. This can ensure the consistency of task scheduling results while ensuring the balanced use of host resources, and effectively improve the utilization rate of resources.

[0058] In one embodiment, step S2 may specifically include:

[0059] Based on the current task running information, tasks that will end before the future target time are selected, and the resource usage of the tasks is recorded.

[0060] Based on the resource occupancy and usage, the available host resources at the future target time are predicted.

[0061] It should be noted that, based on the running information of currently running tasks, tasks that will end before the future target time can be identified and filtered. Based on the resource usage ratio of these tasks, the available host resources at the future target time can be calculated.

[0062] The task scheduling method provided in this application can predict the available host resources at a future target time, further predict the scheduling result at the future target time, and then allocate resources and schedule tasks based on the scheduling result, thereby effectively improving the utilization rate of resources.

[0063] In one embodiment, the method for obtaining the host resource weight set may include:

[0064] Obtain the observation values ​​of historical tasks within a preset historical time range, as well as the resource utilization rate of each host corresponding to the historical tasks, and construct a mapping matrix between the resource utilization rate of each host and the historical tasks;

[0065] Based on the mapping matrix, the predicted resource values ​​of each host at the current moment are calculated using a preset exponential smoothing algorithm.

[0066] The host resource weight set is obtained by weighted averaging the predicted values ​​of each host resource and the preset weight values ​​of each host resource.

[0067] It should be noted that when obtaining the host resource weight set, a mapping matrix is ​​first constructed based on the observation values ​​of historical tasks and resource usage. Then, the exponential smoothing algorithm is used to predict the resource prediction value at the current moment. Finally, the weighted average algorithm is used to calculate the resource weight of all hosts in the host set, forming the host resource weight set.

[0068] In one embodiment, step S4 may specifically include:

[0069] The secondary weight of each host is obtained by calculating the weight of each host resource in the host resource weight set using a preset secondary algorithm.

[0070] The maximum weight of the cluster is obtained by summing the quadratic weights of each host.

[0071] The task weight of the task to be scheduled is calculated by combining the maximum weight of the cluster and the random result obtained by the host.

[0072] It should be noted that, in order to calculate the weights of each host resource according to a percentage system and make the values ​​of the host resource weight set more concentrated, a two-level algorithm can be used to calculate the values ​​of the host resource weight set to obtain the secondary weights of each host. After accumulating the secondary weights, the maximum weight of the cluster is obtained. Then, random results are calculated based on the random seed of the host. Combining the maximum weight of the cluster with the same random results, the same task weight of each host for the same scheduled task can be obtained.

[0073] The task scheduling method provided in this application uses a two-level algorithm to calculate the host resource weights to obtain secondary weights, ensuring the concentration of the weight set and preventing algorithm anomalies caused by excessively large or small host resource weights, thus effectively improving the reliability of resource prediction.

[0074] In one embodiment, when it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled, specifically including:

[0075] When it is determined that the first priority resource in the second remaining resources is greater than the first priority resource in the first remaining resources, task scheduling is performed according to the time limit requirements of each task to be scheduled.

[0076] It should be noted that when comparing the size of the second and first remaining resources, resources can be prioritized. For example, resources essential for task execution, such as CPU and MEN, can be set to first priority, while resources affecting task execution speed, such as I / O utilization, can be set to second priority. When comparing sizes, first compare the size of the first-priority resources. If the sizes of the first-priority resources are comparable, then compare the size of the second-priority resources. Alternatively, the first-priority and second-priority resources can be weighted and summed before comparison. For example, for the first remaining resource, the weighted sum of the first-priority and second-priority resources in a 2:1 ratio can be used as the comparison value for the first remaining resource. Similarly, the comparison value for the second remaining resource can be obtained, and then the comparison values ​​for the first and second remaining resources can be compared.

[0077] The task scheduling method provided in this application compares the size of the first and second remaining resources by dividing them into first and second priority resources. This allows for a more accurate comparison result that better reflects the actual scenario, thereby effectively improving the reliability and accuracy of resource prediction and task scheduling, and further enhancing resource utilization.

[0078] In one embodiment, before obtaining the host resource weights, the method further includes:

[0079] The weight values ​​of each host resource are adaptively adjusted based on the available host resources at the future target time.

[0080] In this embodiment of the application, before calculating the host resource weight, the host resource weight value used to calculate the host resource weight is first adjusted. The adjustment is based on the predicted available host resources at the future target time. For example, if the current CPU and MEM resources are abundant, but the IO resources are scarce, the weight value of the IO resources can be appropriately increased when predicting the resource weight of each host, so that the resource weight obtained by the prediction can meet the requirement of calling fewer IO resources.

[0081] The task scheduling method provided in this application effectively improves the prediction accuracy of subsequent host resource weights, avoids task execution failure caused by the predicted value of a certain scheduling factor being lower than the actual usage value, thereby effectively improving the reliability and accuracy of resource prediction and task scheduling, and further improving resource utilization.

[0082] Please see Figure 2 Based on the above scheme, and to facilitate a better understanding of the task scheduling method provided in the embodiments of this application, the following detailed explanation is provided:

[0083] This application provides a task scheduling method based on a resource priority consistency algorithm to solve the problem of how to ensure the consistency of task scheduling results when the host nodes in a distributed system do not communicate with each other, and how to ensure full utilization of host resources. It is applicable to scenarios where one or more tasks are started on one or more host nodes.

[0084] The embodiments of this application mainly include the following steps:

[0085] Step A101: Calculate the resource weight of each host based on the host resources;

[0086] Step A102: Predict the resources that each host may release in the near future based on the resource factors of the scheduling tasks;

[0087] Step A103: Predict the host resources required for tasks to be scheduled in the near future based on the exponential smoothing algorithm;

[0088] Step A104: Calculate the proportionally reduced resource weights based on the basic resource weights using a formatted method.

[0089] Step A105: Calculate the maximum resource weight of the algorithm;

[0090] Step A106: Calculate the scheduling task weight of the current task based on the maximum weight of the scheduling task and the scheduling task name;

[0091] Step A107: Based on the available host resources in the future, the host resources required by each scheduled task, and the time limit requirements of each scheduled task, schedule the tasks.

[0092] Based on the above solution, the specific implementation process is described in detail below:

[0093] Step S101: Report resources regularly.

[0094] Scheduling units can be deployed on various distributed system hosts to report information such as host IP, CPU usage, memory usage, IO usage, and file handle usage on a minute-by-minute basis (this data can be stored in a database, with one record per minute).

[0095] Step S102: Predict the host resources that may be released during the future time period t1, to obtain the available host resources at the current time + t1. The prediction method may specifically include:

[0096] Step S1021: Obtain the host resource utilization rate S at time t through the fusion scheduling module. t The system retrieves the execution results of currently running tasks, including keywords, post-run resource usage percentages, and execution cycle (stored in a database, one record per scheduled task, with configurable resource usage percentages such as MEM, CPU, and IO). This embodiment retrieves the start time, current resource usage percentage, and estimated end time of already running schedulers based on their execution keywords.

[0097] Step S1022: Filter out the currently executing tasks that will end within the next t1 time period, and record the current resource usage ratio of these tasks as C. i,j Where i represents the resource type, including CPU utilization, memory utilization, IO utilization, file handle utilization, etc., and j represents the task number.

[0098] Step S1023: Using the formula Predict the resource utilization of each host at time t+t1, where S i,t S represents the utilization rate of the i-th type of host resources at time t. i,t+t1 Let be the utilization rate of the i-th type of host resources at time t+t1.

[0099] Taking MEM resource election as an example: The current host MEM utilization rate has reached 75%, and the MEM occupancy rate of the current scheduling module is 15%. If the host MEM will reach 90% after the current scheduling module starts, exceeding the safety setting of 85%, but according to the algorithm prediction, after the original scheduling task finishes running in the next 5 minutes, the host will release 20% of the MEM utilization rate. Therefore, we determine that the current host can schedule this module. In order to ensure that the operating system does not forcibly kill processes with high MEM utilization rates based on security protection, the current host resource utilization rate plus the resource utilization rate after scheduling is greater than 90%, and the current host cannot schedule this task.

[0100] Step S103: Obtain all tasks to be scheduled within the future time period t1, and calculate the proportion of host resources required by the tasks.

[0101] After resource reporting is completed, task scheduling is performed. The name and number of the task to be scheduled are passed to the computing module. Based on the task name and task number, the proportion of various host resources required by the task to be scheduled is predicted, and the scheduling factors of various host resources are obtained (stored in the database; the number of factors should preferably not exceed 9. In this embodiment, it is configured as CPU, MEM, IO, and FD. The specific number of factors can be added and deleted through database configuration. Each data record records one resource factor, configured according to the importance of the resource factor. The larger the value, the greater the weight. The value of the resource factor can be adjusted in real time according to the distribution of the currently available resources).

[0102] Based on the task name and task number, predict the proportion of various host resources required by the task to be scheduled, specifically including:

[0103] Step S1031: Obtain historical task information and construct a model based on neural network algorithm to predict the usage ratio of each resource through historical task information.

[0104] The neural network algorithm can be an exponential smoothing algorithm, and the specific method for constructing the prediction model is as follows:

[0105] Obtain multiple observations such as task name, task number, and host resource information within a certain historical period; obtain the resource usage of each host corresponding to each task and calculate the actual utilization rate of each resource; construct a mapping matrix between observations and resource utilization rates;

[0106] The resource weight A of all hosts in the host set is predicted using machine learning (the embodiment of this application uses the exponential smoothing algorithm to achieve intelligent prediction); the resource weight A is calculated based on 10 resource information reports submitted in the last 10 minutes, and the mathematical formula is S. t =a*y t +(1-a)*S t-1Where 'a' is the first exponential smoothing value of the prediction algorithm, which can be set to 0.5 in this embodiment, and y t Let S be the resource utilization rate reported at time t. t-1 Given the predicted value from the previous moment, we sequentially predict the resource values ​​for the current moment, denoted as CPU. t MEM t IO t FD t The host resource weights are obtained using a weighted average algorithm, as shown in the formula:

[0107] A = (CPU) t CPU sa2 +MEM t *MEM sa2 +IO t *IO sa2 +FD t *FD sa2 ) / (CPU sa2 +MEM sa2 +IO sa2 +FD sa2 The resource weights A1 to Am of each host are calculated, and a prediction model is then constructed. Among these, CPU... sa2 MEM sa2 IO sa2 and FD sa2 This is the weight value of each resource. This weight value can be adjusted in real time according to the utilization rate of various resources at the future time t1 predicted in step S102. For example, if the current CPU and MEM resources are abundant, but the IO resources are scarce, the weight value of the IO resources can be appropriately increased when predicting the resource weight values ​​of each host, so that the resource weight values ​​obtained by the prediction can meet the requirements of calling fewer IO resources.

[0108] Step S1032: Based on the above prediction model, predict the proportion of each resource required for the task to be scheduled.

[0109] Step S104: Format cluster host weights B.

[0110] The resource weights A of each host obtained in the previous steps are used to obtain the secondary weights B1 to Bm of each host according to the second-level algorithm. The value selection algorithm is calculated according to the normalization algorithm, and the mathematical calculation formula is as follows: Where Ai represents the corresponding values ​​of A1 to Am, and Ai is the current normalized element;

[0111] The addition of a secondary weighting algorithm is to perform proportional calculations on the original data A1 to Am according to a percentage system, ensuring that the weight B is between 0 and 100. Without the secondary weighting algorithm, the values ​​in the first weight A would become more discrete, for example, A might have a weight of 20, 80, or 100. The secondary weighting algorithm sets the value of B to 10, 40, or 50 to prevent the weight A from being too large or too small, which could cause algorithm errors.

[0112] Step S105: Obtain the maximum weight C of the cluster.

[0113] The maximum weight C of the cluster is obtained by summing the values ​​of B1 to Bm from the previous steps.

[0114] Step S106: Obtain the task weight D.

[0115] First, a random seed is obtained for each host. This seed is generated by randomly calculating the timestamp of the previous minute of the scheduling time, the task name, and the task number. The ASCII values ​​of the timestamp, task name, and task number are then converted into corresponding numbers using the BKDRhash algorithm. The task name and task number are passed in from the fusion scheduling module. Since all hosts have the same random seed, the random result obtained by each host is identical. After obtaining the random result D1, the random task weight D is obtained according to the algorithm D = C * D1 / (double)(RAND_MAX), where D1 = rand(atoi(time_stamp) + to_number(application name + task number)). The to_number algorithm can use the BKDRhash algorithm, and atoi() is a function that converts a string to an integer. At this point, each task obtains the same calculated task weight D on each host.

[0116] Step S107: Task filtering.

[0117] The specific steps involved in task filtering are as follows:

[0118] S1071: Based on the available host resources at the current moment, allocate host resources to the multiple tasks to be scheduled, and calculate the remaining resource count S1 after task allocation;

[0119] S1072: Based on the predicted available host resources at time t1, allocate host resources to the multiple tasks to be scheduled, and calculate the remaining resource count S2 after task allocation. At this time, only the optimization of resource allocation for multiple tasks is considered, without considering the execution order of each task;

[0120] S1073: Compare S1 and S2. If S2 > S1, determine the urgency of the tasks to be scheduled in the queue. If one or more tasks whose execution order needs to be adjusted can withstand the delay of t1, then use the method of allocating resources based on the available host resources at the predicted time of t1 to allocate resources to multiple tasks to be scheduled.

[0121] When comparing S1 and S2, resources can be classified. For example, resources necessary for task execution, such as CPU and MEN, can be set as the first priority, while resources that affect task execution speed, such as IO utilization, can be set as the second priority. First, compare the size of each type of resource with the first priority, and then compare the size of each type of resource with the second priority.

[0122] S1074: Schedule tasks for which resources have been allocated.

[0123] It should be noted that the embodiments of this application provide a consensus algorithm for distributed hosts, which also takes into account the optimized use of host resources in distributed mode. The algorithm can be extended according to actual needs, and can also be evolved based on this, making it highly practical. The scheduling algorithm used is simple and lightweight, can be directly simulated according to the examples, can predict scheduling results, and can reproduce scheduling result trends according to specific scenarios, exhibiting strong scalability. The calculation algorithms for each resource weight A and secondary weight B involved can be adjusted for rationality and adaptability, such as adjusting CPU weights and increasing SOCKET connection weights.

[0124] The task scheduling method provided in this application predicts the host resources that may be released in the future, uses an exponential smoothing algorithm to predict the host resources required by the tasks to be scheduled in the future, and adjusts the weight of each resource in the algorithm in real time according to the distribution of host resources obtained from the prediction. This ensures that sufficient margin is given to the key resources or those with less remaining resources when predicting resources, and avoids situations where tasks cannot be executed due to a small deviation in the prediction results.

[0125] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0126] 1. The embodiments of this application not only provide a consensus algorithm for distributed hosts, but also take into account the optimized use of host resources in distributed mode. The algorithm can be extended according to actual needs, and can also be evolved on this basis, making it highly practical.

[0127] 2. The scheduling algorithm used in this application embodiment is simple and lightweight, can be directly simulated according to the sample, can predict the scheduling result, can reproduce the trend of the scheduling result according to the specific scenario, has strong scalability, and the calculation algorithms of each resource weight A and B involved can be adjusted reasonably and adaptably, such as adjusting the CPU weight and increasing the SOCKET connection number weight.

[0128] 3. The embodiments of this application can solve the problem of "difficulty in changing the scheduling algorithm and inability to add other scheduling factors". The scheduling factors can be added and deleted in the database to ensure that the number of scheduling factors and reported resources are consistent. When performing index prediction, dynamic calculation can be performed based on the configured parameters.

[0129] The task scheduling apparatus provided in the embodiments of this application is described below. The task scheduling apparatus described below and the task scheduling method described above can be referred to each other.

[0130] Please see Figure 3 This application provides a task scheduling device, including:

[0131] Data acquisition module 1 is used to acquire the resource usage information reported by each host in the distributed system in real time; wherein, the resource usage information includes the available host resources at the current moment;

[0132] Resource prediction module 2 is used to predict the available host resources at a future target time based on the resource usage and current task running information;

[0133] The demand forecasting module 3 is used to obtain all the tasks to be scheduled within the time period between the current time and the future target time, construct a forecasting model based on the pre-obtained set of host resource weights, and use the forecasting model to obtain the required host resource ratio of each task to be scheduled.

[0134] Weight calculation module 4 is used to calculate the task weight of the task to be scheduled based on the host resource weight set;

[0135] Resource calculation module 5 is used to calculate, based on the required host resource ratio and the task weight, the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time.

[0136] The task scheduling module 6 is used to schedule tasks according to the time limit requirements of each task to be scheduled when it is determined that the second remaining resource is greater than the first remaining resource.

[0137] In one embodiment, the resource prediction module 2 is specifically used for:

[0138] Based on the current task running information, tasks that will end before the future target time are selected, and the resource usage of the tasks is recorded.

[0139] Based on the resource occupancy and usage, the available host resources at the future target time are predicted.

[0140] In one embodiment, the method for obtaining the host resource weight set includes:

[0141] Obtain the observation values ​​of historical tasks within a preset historical time range, as well as the resource utilization rate of each host corresponding to the historical tasks, and construct a mapping matrix between the resource utilization rate of each host and the historical tasks;

[0142] Based on the mapping matrix, the predicted resource values ​​of each host at the current moment are calculated using a preset exponential smoothing algorithm.

[0143] The host resource weight set is obtained by weighted averaging the predicted values ​​of each host resource and the preset weight values ​​of each host resource.

[0144] In one embodiment, the weight calculation module 4 is specifically used for:

[0145] The secondary weight of each host is obtained by calculating the weight of each host resource in the host resource weight set using a preset secondary algorithm.

[0146] The maximum weight of the cluster is obtained by summing the quadratic weights of each host.

[0147] The task weight of the task to be scheduled is calculated by combining the maximum weight of the cluster and the random result obtained by the host.

[0148] In one embodiment, the task scheduling module 6 is specifically used for:

[0149] When it is determined that the first priority resource in the second remaining resources is greater than the first priority resource in the first remaining resources, task scheduling is performed according to the time limit requirements of each task to be scheduled.

[0150] In one embodiment, a weight adjustment module is further included, for:

[0151] Before obtaining the host resource weights, the weights of each host resource are adaptively adjusted based on the available host resources at the future target time.

[0152] In one embodiment, the host's resource types include one or more of MEM resources, CPU resources, memory resources, IO resources, file handle resources, and file descriptor resources.

[0153] It is understood that the above-described device embodiments correspond to the method embodiments of this application. The task scheduling device provided in the embodiments of this application can implement the task scheduling method provided in any one of the method embodiments of this application.

[0154] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call a computer program in the memory 430 to execute the steps of a task scheduling method, such as including:

[0155] S1. Obtain the resource usage information reported by each host in the distributed system in real time; wherein, the resource usage information includes the available host resources at the current moment;

[0156] S2. Based on the resource usage and current task operation information, predict the available host resources at the future target time.

[0157] S3. Obtain all tasks to be scheduled within the time period between the current time and the future target time, construct a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain the required host resource ratio of each task to be scheduled.

[0158] S4. Calculate the task weight of the task to be scheduled based on the host resource weight set;

[0159] S5. Based on the required host resource ratio and the task weight, calculate the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time.

[0160] S6. When it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled.

[0161] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the task scheduling method provided in the above embodiments, such as including:

[0163] S1. Obtain the resource usage information reported by each host in the distributed system in real time; wherein, the resource usage information includes the available host resources at the current moment;

[0164] S2. Based on the resource usage and current task operation information, predict the available host resources at the future target time.

[0165] S3. Obtain all tasks to be scheduled within the time period between the current time and the future target time, construct a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain the required host resource ratio of each task to be scheduled.

[0166] S4. Calculate the task weight of the task to be scheduled based on the host resource weight set;

[0167] S5. Based on the required host resource ratio and the task weight, calculate the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time.

[0168] S6. When it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled.

[0169] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including:

[0170] S1. Obtain the resource usage information reported by each host in the distributed system in real time; wherein, the resource usage information includes the available host resources at the current moment;

[0171] S2. Based on the resource usage and current task operation information, predict the available host resources at the future target time.

[0172] S3. Obtain all tasks to be scheduled within the time period between the current time and the future target time, construct a prediction model based on the pre-obtained host resource weight set, and use the prediction model to obtain the required host resource ratio of each task to be scheduled.

[0173] S4. Calculate the task weight of the task to be scheduled based on the host resource weight set;

[0174] S5. Based on the required host resource ratio and the task weight, calculate the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time.

[0175] S6. When it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled.

[0176] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0177] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A task scheduling method, characterized in that, include: The system acquires real-time resource usage information reported by each host in the distributed system; wherein the resource usage information includes the available host resources at the current moment. Based on the resource usage and current task execution information, the available host resources at the future target time are predicted; All tasks to be scheduled within the time period between the current time and the future target time are obtained. A prediction model is constructed based on a pre-obtained set of host resource weights, and the required host resource ratio for each task to be scheduled is obtained using the prediction model. Calculate the task weight of the task to be scheduled based on the host resource weight set; Based on the required host resource ratio and the task weight, calculate the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time. When it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled.

2. The task scheduling method according to claim 1, characterized in that, The step of predicting available host resources at a future target time based on the resource usage and current task execution information includes: Based on the current task running information, tasks that will end before the future target time are filtered out, and the resource usage of the tasks that will end before the future target time is recorded. Based on the resource occupancy and usage, the available host resources at the future target time are predicted.

3. The task scheduling method according to claim 1, characterized in that, The methods for obtaining the host resource weight set include: Obtain the observation values ​​of historical tasks within a preset historical time range, as well as the resource utilization rate of each host corresponding to the historical tasks, and construct a mapping matrix between the resource utilization rate of each host and the historical tasks; Based on the mapping matrix, the predicted resource values ​​of each host at the current moment are calculated using a preset exponential smoothing algorithm. The resource weight of each host is calculated by weighted averaging the predicted resource values ​​of each host and the preset resource weight values ​​of each host, and the resource weights of each host are combined into the host resource weight set.

4. The task scheduling method according to claim 1, characterized in that, The step of calculating the task weight of the task to be scheduled based on the host resource weight set includes: The secondary weight of each host is obtained by calculating the weight of each host resource in the host resource weight set using a preset secondary algorithm. The maximum weight of the cluster is obtained by summing the quadratic weights of each host. The task weight of the task to be scheduled is calculated by combining the maximum weight of the cluster and the random result obtained by the host.

5. The task scheduling method according to claim 1, characterized in that, When it is determined that the second remaining resource is greater than the first remaining resource, task scheduling is performed according to the time limit requirements of each task to be scheduled, specifically including: When it is determined that the first priority resource in the second remaining resources is greater than the first priority resource in the first remaining resources, task scheduling is performed according to the time limit requirements of each task to be scheduled.

6. The task scheduling method according to claim 3, characterized in that, Before obtaining the host resource weights, the process also includes: The weight values ​​of each host resource are adaptively adjusted based on the available host resources at the future target time.

7. The task scheduling method according to any one of claims 1-6, characterized in that, The host's resource types include one or more of the following: MEM resources, CPU resources, memory resources, IO resources, file handle resources, and file descriptor resources.

8. A task scheduling device, characterized in that, include: The data acquisition module is used to acquire the resource usage information reported by each host in the distributed system in real time; wherein, the resource usage information includes the available host resources at the current moment; The resource prediction module is used to predict the available host resources at a future target time based on the resource usage and current task running information. The demand forecasting module is used to obtain all scheduled tasks within the time period between the current time and the future target time, construct a forecasting model based on the pre-obtained host resource weight set, and use the forecasting model to obtain the required host resource ratio of each scheduled task. The weight calculation module is used to calculate the task weight of the task to be scheduled based on the host resource weight set; The resource calculation module is used to calculate, based on the required host resource ratio and the task weight, the first remaining resource after resource allocation based on the available host resources at the current time, and the second remaining resource after resource allocation based on the available host resources at the future target time. The task scheduling module is used to schedule tasks according to the time limit requirements of each task to be scheduled when it is determined that the second remaining resource is greater than the first remaining resource.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the task scheduling method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the task scheduling method according to any one of claims 1 to 7.

Images