A method and system for efficient resource scheduling of massive tasks

Abstract

The application provides a kind of method and system for efficiently resource scheduling to mass task, including central dispatcher, grouping dispatcher and resource scheduler, central dispatcher and grouping dispatcher realize the order of customer resource, reasonable allocation;Resource scheduler realizes the reasonable management of elastic host, task and resource separate scheduling, in intelligent finance and tax platform, build two-layer structure "task-resource" scheduling system, improve the efficiency of task and resource mobilization, greatly save the task processing time;In addition, central dispatcher combines "weighted maximum minimum fair algorithm", grouping dispatcher combines "high priority algorithm" to calculate "priority coefficient", respectively complete the first resource allocation and secondary resource allocation to overall customer and group customer, there is no "one dominant" resource tilt allocation phenomenon, significantly improve the fairness and rationality of task-resource scheduling.

Description

Technical Field

[0001] This invention relates to the field of financial and tax information technology and data processing, and in particular to a method and system for efficient resource scheduling of massive tasks. Background Technology

[0002] With the large-scale application of RPA (Robotic Process Automation) technology in the financial and tax field, intelligent financial and tax platforms execute millions of automated financial and tax tasks submitted by hundreds of thousands of customers every day. These massive automated tasks run on elastic hosts on cloud platforms. Due to network resource limitations, cloud platforms can only provide a maximum of tens of thousands of elastic host resources.

[0003] Currently, intelligent financial and tax platforms primarily use a First-In-First-Out (FIFO) algorithm to allocate elastic host resources for tasks, prioritizing tasks submitted earlier for execution. This can lead to a situation where a few clients submit a large number of tasks at once, consuming significant host resources and causing subsequent clients to submit tasks without available resources, resulting in long queues and poor user experience. Consequently, it fails to guarantee that massive amounts of tasks will have sufficient host resources to complete quickly, and it cannot ensure the equitable distribution of host resources among different clients, leading to a resource imbalance where one client dominates. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method and system for efficient resource scheduling of massive tasks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A system for efficient resource scheduling of massive tasks includes a central scheduler, a group scheduler, and a resource scheduler.

[0006] The central scheduler, upon receiving a task submitted by an external client, first allocates it to different task groups according to preset dimensions, with each task group having a manually set weight value; then it queries the resource scheduler for the number of currently available resources and uses a "weighted max-min fair algorithm" to allocate the available resources to each task group; it also receives task execution results reported by the elastic host and maintains the task status.

[0007] The group scheduler consists of one scheduler for each task group. The group scheduler calculates the "priority coefficient" of the tasks within the group and uses a "high priority algorithm" to allocate the group's idle resources to the tasks. Then, it sends the tasks to the corresponding elastic hosts for execution. During task execution, it monitors the task running status and sends the task running results back to the client.

[0008] The resource scheduler is responsible for managing the elastic host resources in the cloud, receiving the busy / idle status reported by the hosts, calculating the idle host resources, predicting the host demand in the future period, applying to the cloud platform in advance to open new hosts, and shutting down and releasing idle hosts.

[0009] The central scheduler is located upstream of the group scheduler, which operates independently.

[0010] A method for efficient resource scheduling of massive tasks includes the following steps: S1: Start the central scheduler process. The central scheduler creates several task groups based on different dimensions and sets a group weight for each task group. S2: For each initiated group scheduler process, the group scheduler process queries the average runtime of the tasks in the previous period under the task group dimension as the estimated runtime of the tasks in the current period for that group. S3: After receiving the task submitted by the client, the central scheduler puts the task into the corresponding group according to the corresponding grouping dimension; S4: The resource scheduler queries the current number of idle resources, periodically calculates the number of host resource changes, deploys elastic hosts based on the number of host resource changes, starts and stops elastic hosts in real time, forms a host resource list, and reports it to the central scheduler in real time. S5: The central scheduler uses the "weighted maximum-minimum fair algorithm" to allocate resources from the host resource list to each task group, completing one allocation of resources to the group, confirming the number of hosts and the host list of each task group, and distributing the host list. S6: The group scheduler for several task groups queries the task list of the corresponding group and calculates the task waiting time; S7: The group scheduler calculates the "task priority coefficient" using the "high priority algorithm" based on the estimated runtime and waiting time of the tasks in the task group. Based on the "task priority coefficient", the resources in the host list are re-allocated for the tasks in the group, the host corresponding to the task is confirmed, and the task is sent to the corresponding host for execution. S8: The host corresponding to the task executes the received task and reports the execution result to the central scheduler. The central scheduler receives the task execution result and changes the task status.

[0011] Furthermore, step S4 specifically includes: S41: The resource scheduler queries the current number of available resources; S42: The resource scheduler uses a time series forecasting algorithm to predict the number of unrunned tasks in a future period; S43: The resource scheduler estimates the number of tasks that can be completed in the next cycle based on the average running time of historical tasks, i.e. the amount of host resources to be released. S44: The resource scheduler calculates the number of host resource changes within a future period, i.e.: Number of host resources changed = Number of inactive tasks - Number of currently idle resources - Number of host resources released; S45: If the number of host resource changes > 0, call the cloud platform (Tencent Cloud Platform, Alibaba Cloud Platform, Huawei Cloud Platform, etc.) interface to enable and deploy the elastic host; if the number of resource changes < 0, call the cloud platform interface to disable and release the elastic host; the number of elastic hosts enabled or disabled is equal to the number of host resource changes. S46: The resource scheduler starts and stops elastic hosts based on the actual start and completion of tasks; S47: Based on the number of elastic hosts deployed or released in S45 and the number of elastic hosts started or stopped in S46, the resource scheduler obtains a list of host resources and reports it to the central scheduler in real time.

[0012] Furthermore, the weighted max-min fairness algorithm specifically includes the following steps: S51: Calculate the number of resources that can be allocated to each group. The calculation formula is: number of resources that can be allocated to this group = total number of remaining resources * weight of this group / ∑(sum of weights of all groups), rounded to the nearest integer. The total number of remaining resources is the number of host resources in the current host resource list; S52: When the number of resources allocated to this group is greater than or equal to the number of unrunned tasks in this group, adjust the number of resources allocated to this group to equal the number of unrunned tasks in this group, and release the excess resources; S53: When resource allocation conflicts occur among multiple groups, if the weights are different, the resources are allocated in descending order of weight; if the weights are the same, the resources are allocated in descending order of demand. S54: If all tasks in all groups have resources to run, the allocation ends. If there are still unallocated resources, the allocation will proceed to S51 for a second round until the remaining resources are 0.

[0013] Furthermore, the "high-priority algorithm" calculates the "task priority coefficient" by including the following steps: S71: The group scheduler calculates the task priority coefficient based on the estimated runtime and waiting time of each task in the task group. Task priority coefficient = (task waiting time + estimated task runtime) / estimated task runtime; S72: Sort each task in the task group from highest to lowest priority according to the task priority coefficient. The host resources in the host list of the tasks in the group are allocated according to the sorting.

[0014] A financial and tax platform system that utilizes a system for efficient resource allocation of massive tasks.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The system groups tasks according to different dimensions and adopts a two-layer task scheduling structure to schedule tasks-resources. The first-layer central scheduler is responsible for managing and allocating host resources for task groups, and the second-layer group scheduler is responsible for secondary allocation of host resources for tasks in different groups, which significantly improves the fairness and rationality of task-resource scheduling.

[0016] (2) The "weighted maximum-minimum fair algorithm" and "high priority scheduling algorithm" are introduced in the scheduling process, which can dynamically adjust the queuing order of tasks and avoid the phenomenon of "one dominant" resource allocation. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the central scheduler, group scheduler, resource scheduler and external connections in the system of Embodiment 1 of the present invention. Implementation

[0018] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments. Example

[0019] A system for efficient resource scheduling of massive tasks, including a central scheduler, a group scheduler, and a resource scheduler; Central Scheduler: After receiving tasks submitted by external clients, it first allocates them to different task groups according to different dimensions (including region, client information, task type, etc.), and manually sets weight values ​​for each task group; then it queries the resource scheduler for the number of currently available resources and uses the "weighted max-min fair algorithm" to allocate the available resources to each task group; it receives task execution results reported by elastic hosts and maintains the task status.

[0020] The central scheduler groups customer resources according to different dimensions and sets weights, dividing the customer groups according to attributes. In the initial allocation of host resources, the priority of customer host resources can be reasonably considered, allowing groups with higher weights to obtain more host resources, while maintaining the execution results of elastic hosts.

[0021] Group Scheduler: Each task group corresponds to a group scheduler. The group scheduler calculates the "priority coefficient" of the tasks in the group and uses the "high priority algorithm" to allocate the idle resources of the group to the tasks. Then, the tasks are sent to the corresponding elastic hosts for execution. During the task execution, the scheduler monitors the task running status and feeds back the task running results to the client.

[0022] The group scheduler is responsible for the host resources within its group, ensuring that the hosts are deployed within the group and that tasks are assigned to them for execution. This secondary allocation process ensures that the allocated host resources within the group are utilized more rationally and fairly.

[0023] Resource scheduler: Responsible for managing elastic host resources in the cloud, receiving busy / idle status reports from hosts, calculating idle host resources, predicting host demand in the future cycle, applying to the cloud platform in advance to open new hosts, and shutting down and releasing idle hosts.

[0024] The host scheduler is solely responsible for the resource scheduling module. It manages host resources and is independent of the allocation of client resources, but together they complete the task of host allocation. It has its own independent brain and can more rationally handle the dynamic management of host busy and idle periods.

[0025] The central scheduler is located upstream of the group scheduler, which operates independently.

[0026] Tasks and resources are scheduled separately. A two-layer "task-resource" scheduling system is built in the intelligent financial and tax platform. This ensures that massive tasks can be completed as quickly as possible with host resources, while also ensuring that host resources are reasonably allocated among different customers, avoiding the phenomenon of "one company dominating" resource allocation. Example

[0027] A method for efficient resource scheduling of massive tasks includes the following steps: S1: Start the central scheduler process. The central scheduler creates several task groups based on different dimensions (including region, customer information, task type, etc.) and sets a group weight for each task group.

[0028] In the initial customer allocation process, customers are not only grouped according to different dimensions, which is based on objective factors, but also weighted according to subjective factors. Therefore, the resources that a customer can be allocated are simultaneously matched by the objective dimensions of the customer and the subjective circumstances, which is more scientific.

[0029] S2: For each initiated group scheduler process, the group scheduler process queries the average runtime of tasks in the previous period (usually the previous day) under the task group dimension as the estimated runtime of tasks in the current period (usually the current day).

[0030] The duration of tasks within the group is periodically statistically analyzed and used as the estimated task runtime to facilitate the secondary allocation described below.

[0031] S3: After receiving the task submitted by the client, the central scheduler puts the task into the corresponding group according to the corresponding grouping dimension.

[0032] S4: The resource scheduler queries the current number of idle resources, periodically calculates the number of host resource changes, deploys elastic hosts based on the number of host resource changes, starts or shuts down elastic hosts in real time, forms a host resource list, and reports it to the central scheduler in real time.

[0033] The calculation of host resource changes more reasonably combines three important data points: the number of non-running tasks, the current number of idle resources, and the number of released host resources. This allows for reasonable coordination of host resources and task load, enabling direct management of existing hosts and pre-management of cloud platform hosts.

[0034] S5: The central scheduler uses the "weighted maximum-minimum fair algorithm" to allocate resources from the host resource list to each task group, completing one allocation of resources to the group, confirming the number of hosts and the host list for each task group, and distributing the host list.

[0035] The weighted max-min fairness algorithm fully considers both subjective and objective factors, achieving a reasonable and fair allocation in one go.

[0036] S6: The group scheduler for several task groups queries the task list of the corresponding group and calculates the task waiting time.

[0037] S7: The group scheduler calculates the "task priority coefficient" using the "high priority algorithm" based on the estimated runtime and waiting time of the tasks in the task group. Based on the "task priority coefficient", the resources in the host list are re-allocated for the tasks in the group, the host corresponding to the task is confirmed, and the task is sent to the corresponding host for execution.

[0038] The calculation system of task priority coefficient and high priority algorithm allows the elastic hosts in this group to "roll" organically. Tasks that run fast can release host resources as soon as possible, while tasks with estimated long runtimes will have their priority increasing as their waiting time increases. This makes the allocation process dynamic and has the attribute of continuous learning. The time when the last task was completed becomes an important reference data.

[0039] S8: The host corresponding to the task executes the received task and reports the execution result to the central scheduler. The central scheduler receives the task execution result and changes the task status.

[0040] Furthermore, step S4 specifically includes: S41: The resource scheduler queries the number of currently available resources.

[0041] S42: The resource scheduler uses a time series forecasting algorithm to predict the number of unrunning tasks in a future period (default is 5 minutes, configurable).

[0042] S43: The resource scheduler estimates the number of tasks that can be completed in the next cycle (default is 5 minutes, configurable) based on the average runtime of historical tasks, which is the amount of host resources to be released.

[0043] S44: The resource scheduler calculates the number of host resource changes within a future period (default 5 minutes, configurable), i.e.: Number of changes to host resources = Number of non-running tasks - Number of currently idle resources - Number of released host resources.

[0044] S45: If the number of host resource changes is greater than 0, call the cloud platform interface to enable and deploy the elastic host; if the number of resource changes is less than 0, call the cloud platform interface to disable and release the elastic host; the number of elastic hosts enabled or disabled is equal to the number of host resource changes.

[0045] S46: The resource scheduler starts and shuts down elastic hosts based on the actual start and completion of tasks.

[0046] S47: Based on the number of elastic hosts deployed or released in S45 and the number of elastic hosts started or stopped in S46, the resource scheduler obtains a list of host resources and reports it to the central scheduler in real time.

[0047] The host resource change count, as a calculated reference feature, takes into account both the existing number of hosts and the estimated number of hosts needed in the future. This allows for the rational use of currently deployed resources and the estimation of the required elastic hosts on the cloud platform, enabling preparation and full mobilization of elastic host resources to prepare logistically for task completion.

[0048] Furthermore, the weighted max-min fairness algorithm has the following principles: 1) Achieve fair and standardized allocation through weighted averaging of weights; 2) No user receives more resources than they need; 3) Users whose needs are not met will share resources according to their weight. 4) If the weights are different, resources are allocated from largest to smallest according to the weights; 5) If the weights are the same, resources are allocated from largest to smallest according to demand; 6) The allocation ends when all resources have been allocated or all requirements have been met.

[0049] Specifically, the following steps are included: S51: Calculate the number of resources that can be allocated to each group. The calculation formula is: number of resources that can be allocated to this group = total number of remaining resources * weight of this group / ∑(sum of weights of all groups), rounded to the nearest integer. The total number of remaining resources is the number of host resources in the current host resource list; S52: When the number of resources allocated to this group is greater than or equal to the number of unrunned tasks in this group, adjust the number of resources allocated to this group to equal the number of unrunned tasks in this group, and release the excess resources; S53: When resource allocation conflicts occur among multiple groups, if the weights are different, the resources are allocated in descending order of weight; if the weights are the same, the resources are allocated in descending order of demand. S54: If all tasks in all groups have resources to run, the allocation ends. If there are still unallocated resources, the allocation will proceed to S51 for a second round until the remaining resources are 0.

[0050] To illustrate the algorithm, consider an example with 13 idle host resources to be allocated to 4 task groups. The number of tasks and their weights in each group are as follows: 1) Group A has 2 tasks with a weight of 3; 2) Group B has 5 tasks, each with a weight of 1; 3) Group C has 3 tasks, each with a weight of 1; 4) Group D has 9 tasks with a weight of 2.

[0051] The resource allocation process is shown in Table 1 below:

[0052] Table 1 According to the above allocation plan, the final result is: 1) Group A is assigned 2 hosts to run 2 tasks, and 0 tasks are not running; 2) Group B was assigned 3 hosts, running 3 tasks, and 2 tasks were not running; 3) Group C was assigned 2 hosts to run 2 tasks, and 1 task was not running; 4) Group D is assigned 6 hosts to run 6 tasks, and 3 tasks are not running.

[0053] Furthermore, the "high-priority algorithm" calculates the "task priority coefficient" by including the following steps: S71: The group scheduler calculates the task priority coefficient based on the estimated runtime and waiting time of each task in the task group. Task priority coefficient = (task waiting time + estimated task runtime) / estimated task runtime; S72: Sort each task in the task group from highest to lowest priority according to the task priority coefficient. The host resources in the host list of the tasks in the group are allocated according to the sorting.

[0054] As can be seen from the formula above, tasks with shorter estimated runtimes are executed first because faster task execution releases host resources more quickly. Conversely, tasks with longer estimated runtimes have increasing priority as their waiting time increases, and they can also obtain host resources to run. The estimated runtime of a task can be estimated by using the historical average runtime of similar tasks. Example

[0055] A financial and tax platform system that utilizes a system for efficient resource allocation of massive tasks.

[0056] It includes a central scheduler, a group scheduler, and a resource scheduler to process customer resources entered into the financial and tax platform system.

[0057] The present invention has been described in the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.

Claims

1. A system for efficient resource scheduling of massive tasks, characterized in that: This includes a central scheduler, a group scheduler, and a resource scheduler; The central scheduler, upon receiving a task submitted by an external client, first assigns it to different task groups according to preset dimensions, with each task group having a manually set weight value. Then, query the resource scheduler for the number of currently available resources and use the "weighted max-min fair algorithm" to allocate the available resources to each task group; Receive task execution results reported by the elastic host and maintain task status; The dimensions include: region, customer, and task type; The weighted maximum-minimum fairness algorithm specifically includes the following steps: S51: Calculate the number of resources that can be allocated to each group, using the following formula: Round to the nearest integer. The total number of remaining resources is the number of host resources in the current host resource list; S52: When the number of resources allocated to this group is greater than or equal to the number of unrunned tasks in this group, adjust the number of resources allocated to this group to equal the number of unrunned tasks in this group, and release the excess resources; S53: When resource allocation conflicts occur among multiple groups, if the weights are different, the resources are allocated in descending order of weight; if the weights are the same, the resources are allocated in descending order of demand. S54: If all tasks in all groups have resources to run, the allocation ends. If there are still unallocated resources, the allocation will start from S51 for a second round of allocation until the remaining resources are 0. The group scheduler: Each task group corresponds to a group scheduler. The group scheduler calculates the "priority coefficient" of the tasks in the group, uses the "high priority algorithm" to allocate the idle resources of the group to the tasks, and then sends the tasks to the corresponding elastic hosts for execution. During the task execution, the scheduler monitors the task running status and feeds back the task running results to the client. The resource scheduler is responsible for managing the elastic host resources in the cloud, receiving the busy / idle status reported by the hosts, calculating the idle host resources, predicting the host demand in the future period, applying to the cloud platform in advance to open new hosts, and shutting down and releasing idle hosts. The central scheduler is located upstream of the packet scheduler, and the packet scheduler operates independently. The "high-priority algorithm" calculates the "task priority coefficient" by including the following steps: S71: The group scheduler calculates the task priority coefficient based on the estimated runtime and waiting time of each task in the task group. ； S72: Sort each task in the task group from high to low according to the task priority coefficient, and allocate host resources in the host list of the tasks in the group according to the sorting.

2. A method for efficient resource scheduling of massive tasks, characterized in that: Specifically, the following steps are included: S1: Start the central scheduler process. The central scheduler first creates several task groups according to different preset dimensions and sets a group weight for each task group. S2: Start a group scheduler process for each task group. The group scheduler process queries the average runtime of the tasks in the previous period under the task group dimension as the estimated runtime of the tasks in the current period. S3: After receiving the task submitted by the client, the central scheduler places the task into the corresponding group according to the grouping dimension of the task; S4: The resource scheduler queries the current number of idle resources, periodically calculates the number of host resource changes, deploys elastic hosts based on the number of host resource changes, starts or shuts down elastic hosts in real time, forms a host resource list and reports it to the central scheduler in real time. S5: The central scheduler uses the "weighted maximum-minimum fair algorithm" to allocate resources from the host resource list to each task group, completing one allocation of resources to the group, confirming the number of hosts and the host list of each task group, and distributing the host list. The weighted maximum-minimum fairness algorithm specifically includes the following steps: S51: Calculate the number of resources that can be allocated to each group, using the following formula: Round to the nearest integer. The total number of remaining resources is the number of host resources in the current host resource list; S52: When the number of resources allocated to this group is greater than or equal to the number of unrunned tasks in this group, adjust the number of resources allocated to this group to equal the number of unrunned tasks in this group, and release the excess resources; S53: When resource allocation conflicts occur among multiple groups, if the weights are different, the resources are allocated in descending order of weight; if the weights are the same, the resources are allocated in descending order of demand. S54: If all tasks in all groups have resources to run, the allocation ends. If there are still unallocated resources, proceed to S51 for a second round of allocation until the remaining resources are 0. S6: The group scheduler for several task groups queries the task list of the corresponding group and calculates the task waiting time; S7: The group scheduler calculates the "task priority coefficient" using the "high priority algorithm" based on the estimated runtime and waiting time of the tasks in the task group. Based on the "task priority coefficient", the resources in the host list are re-allocated for the tasks in the group, the host corresponding to the task is confirmed, and the task is sent to the corresponding host for execution. The "high-priority algorithm" calculates the "task priority coefficient" by including the following steps: S71: The group scheduler calculates the task priority coefficient based on the estimated runtime and waiting time of each task in the task group. ； S72: Sort each task in the task group from highest to lowest priority according to the task priority coefficient. Allocate host resources according to the sorting of the host list in the tasks within the group; S8: The host corresponding to the task executes the received task and reports the execution result to the central scheduler. The central scheduler receives the task execution result and changes the task status.

3. The method for efficient resource scheduling of massive tasks as described in claim 2, characterized in that: Step S4 specifically includes: S41: The resource scheduler queries the current number of available resources; S42: The resource scheduler uses a time series forecasting algorithm to predict the number of unrunned tasks in a future period; S43: The resource scheduler estimates the number of tasks that can be completed in the next cycle based on the average running time of historical tasks, i.e. the amount of host resources to be released. S44: The resource scheduler calculates the number of host resource changes within a future period, i.e.: ; S45: If the number of host resource changes is greater than 0, call the cloud platform interface to enable and deploy the elastic host; if the number of resource changes is less than 0, call the cloud platform interface to disable and release the elastic host; the number of elastic hosts enabled or disabled is equal to the number of host resource changes. S46: The resource scheduler starts and stops elastic hosts based on the actual start and completion of tasks; S47: Based on the number of elastic hosts deployed or released in S45 and the number of elastic hosts started or stopped in S46, the resource scheduler obtains a list of host resources and reports it to the central scheduler in real time.

4. A financial and tax platform system that utilizes the system for efficient resource scheduling of massive tasks as described in claim 1.

Images