Task scheduling method and system for scheduling platform

Abstract

The invention provides a task scheduling method and system of a scheduling platform. The task scheduling method of the scheduling platform comprises the following steps: acquiring historical execution data of a plurality of task execution nodes, and sorting the plurality of task execution nodes to obtain a node sorting list; obtaining a plurality of to-be-distributed tasks, and sorting the plurality of to-be-distributed tasks to obtain a task sorting list; and distributing the to-be-distributed tasks according to the node sorting list and the task sorting list to obtain a task distribution scheme, and performing task scheduling by the scheduling platform according to the task distribution scheme. The stability of the nodes is evaluated by analyzing the historical execution data of the nodes, the complexity of the tasks is evaluated by analyzing the task parameters of the tasks to be allocated, the stability of the nodes and the complexity of the tasks are comprehensively considered, the task allocation scheme is generated, the task processing efficiency of a server is improved, and the use efficiency of computing resources is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a task scheduling method and system for a scheduling platform. Background Art

[0002] In some business scenarios, the task scheduling platform often receives some scheduled tasks, which require periodic repetition of certain operations at specific times, such as sending reminders and pushing content at fixed times every day. The role of the task scheduling platform is to reasonably distribute the various tasks received to various task execution nodes, so that the server can efficiently process various tasks.

[0003] The stability of the task execution nodes in the cluster is difficult to unify, resulting in different probabilities of failures during task execution. In this case, the task scheduling platform receives multiple tasks, and the scheduled execution time of different tasks may be different, and the time required to complete the tasks may also be different. In the face of complex and diverse tasks to be assigned, if the stability of the task execution nodes and the complexity of the tasks to be assigned are not considered, it is easy to cause the server to handle various tasks inefficiently, wasting computing resources. Summary of the invention

[0004] In view of this, the present invention provides a task scheduling method and system for a scheduling platform, which generates a reasonable task allocation scheme for the task scheduling platform by analyzing the stability of each task execution node and the differences between different tasks to be assigned, thereby improving the efficiency of the server in processing various tasks and saving computing resources.

[0005] In a first aspect, an embodiment of the present invention provides a task scheduling method of a scheduling platform, comprising: Acquire historical execution data of multiple task execution nodes, determine stability parameters of the multiple task execution nodes according to the historical execution data, sort the multiple task execution nodes according to the stability parameters to obtain a node sorting list, wherein the node sorting list includes node numbers and the stability parameters; Acquire multiple tasks to be assigned, extract task parameters of each task to be assigned from the multiple tasks to be assigned, the task parameters include subtask execution start and end time points, subtask execution duration and task cycle period, calculate the execution reference factor of each task to be assigned according to the task parameters of the tasks to be assigned, sort the multiple tasks to be assigned according to the execution reference factor, and obtain a task sorting list, the task sorting list includes the task number, the task parameters and the execution reference factor; A plurality of tasks to be assigned are assigned according to the node sorting list and the task sorting list to obtain a task assignment plan, and the scheduling platform performs task scheduling according to the task assignment plan.

[0006] Further, the determining of stability parameters of a plurality of task execution nodes according to the historical execution data, sorting the plurality of task execution nodes according to the stability parameters, and obtaining a node sorting list includes: The total number of tasks executed by each task execution node and the number of faults that occurred during the task execution are determined based on the historical execution data, and the stability parameter is calculated using the following formula: ;in, is a stable parameter, is the total number of tasks, is the number of failures; After calculating the stable parameters of multiple task execution nodes Then, according to the stable parameters The plurality of task execution nodes are sorted in ascending order to obtain the node sorting list.

[0007] Furthermore, the step of calculating the execution reference factor of each task to be assigned according to the task parameters of the task to be assigned, and sorting the multiple tasks to be assigned according to the execution reference factor to obtain a task sorting list includes: According to the task parameters of the task to be assigned, the execution reference factor is calculated according to the following formula: ;in, Represents the execution reference factor, Indicates the execution time of the subtask. Indicates the task cycle period, Subtask execution time The weight parameter, Task cycle The weight parameter of After calculating the execution reference factor of each task to be assigned Then, according to the execution reference factor The plurality of tasks to be assigned are sorted in order from large to small to obtain the task sorting list.

[0008] Furthermore, allocating a plurality of tasks to be allocated according to the node sorting list and the task sorting list to obtain a task allocation scheme includes: S11. For the task sorting list, the execution reference factor of the i-th item in the task sorting list is , let i=1; S12, traverse the node sorting list with the i-th item of the task sorting list, and judge whether the i-th item of the task sorting list can be assigned to the task execution node corresponding to the first item of the node sorting list according to the task parameter corresponding to the i-th item of the task sorting list, if yes, record the assignment relationship and go to S13, otherwise judge whether the i-th item of the task sorting list can be assigned to the task execution node corresponding to the next item of the node sorting list, and so on until the i-th item of the task sorting list matches the task execution node, and go to step S13; S13, determine whether "i<n" holds, where n is the total number of tasks in the task sorting list. If so, set i=i+1 and go to step S12, otherwise go to step S14; S14. Output all the allocation relationships recorded in step S12 to generate the task allocation plan.

[0009] Further, for step S12, judging whether the i-th item in the task sorting list can be allocated to the task execution node corresponding to the first item in the node sorting list according to the task parameter corresponding to the i-th item in the task sorting list includes: According to the subtask execution start and end time points and subtask execution duration of the i-th item in the task sorting list, determine whether the i-th item in the task sorting list overlaps with the subtask execution start and end time points and subtask execution duration of the task assigned to the task execution node corresponding to the first item in the node sorting list, and whether there is any overlap. If not, the i-th item in the task sorting list can be assigned to the task execution node corresponding to the first item in the node sorting list; otherwise, it means that the i-th item in the task sorting list cannot be assigned to the task execution node corresponding to the first item in the node sorting list.

[0010] Furthermore, it also includes: After any to-be-assigned task in the task sorting list is assigned to one of the task execution nodes in the node sorting list, the to-be-assigned task assigned to the task execution node is regarded as a task that has been assigned to the execution node.

[0011] In a second aspect, an embodiment of the present invention provides a task scheduling system of a scheduling platform, wherein the task scheduling system of the scheduling platform is applied to a task scheduling method of a scheduling platform described in any one of the above items, including: A data acquisition module is used to acquire historical execution data of multiple task execution nodes and multiple tasks to be assigned; A node analysis module, used to determine the stability parameters of multiple task execution nodes according to the historical execution data, and sort the multiple task execution nodes according to the stability parameters to obtain a node sorting list; A task analysis module is used to extract task parameters of each task to be assigned from multiple tasks to be assigned, calculate the execution reference factor of each task to be assigned according to the task parameters of the task to be assigned, and sort the multiple tasks to be assigned according to the execution reference factor to obtain a task sorting list; The task scheduling module is used to allocate multiple tasks to be allocated according to the node sorting list and the task sorting list, and obtain a task allocation plan used by the scheduling platform for task scheduling.

[0012] Furthermore, the node analysis module and the task analysis module further include: The node sorting list generated by the node analysis module includes the node number and the stability parameter, and the task sorting list generated by the task analysis module includes the task number, the task parameter and the execution reference factor.

[0013] The beneficial effects of the present invention are: The present invention analyzes the historical execution data of the nodes to evaluate the stability of the nodes, analyzes the task parameters of the tasks to be assigned to evaluate the complexity of the tasks, comprehensively considers the stability of the nodes and the complexity of the tasks, generates a task allocation plan, improves the efficiency of the server in processing tasks, and improves the efficiency of the use of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0015] Figure 1 The present invention is a flowchart of a task scheduling method of a scheduling platform provided in an embodiment of the present invention.

[0016] Figure 2 A schematic diagram of the structure of a task scheduling system of a scheduling platform provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantages of the present application clearer, some embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented.

[0018] See also Figure 1 , is a flow chart of a task scheduling method of a scheduling platform provided in an embodiment of the present invention. As one aspect of the present invention, a task scheduling method of a scheduling platform is provided, comprising: S1. Obtain historical execution data of multiple task execution nodes, determine stable parameters of the multiple task execution nodes according to the historical execution data, sort the multiple task execution nodes according to the stable parameters, and obtain a node sorting list; Specifically, the task execution node is the node used for data processing in the cluster, and the historical execution number is the relevant data of various tasks executed by the task cluster node, including but not limited to the start and end time, end time, failure events and other information of the execution task. By analyzing the information related to the failure events recorded in the historical execution data of the task execution node, the stability of the task execution node is evaluated and the stability parameters of the task execution node are calculated.

[0019] In this embodiment, the total number of tasks executed by each task execution node and the number of faults that occurred during the task execution are determined based on historical execution data, and the stability parameter is calculated using the following formula: ;in, is a stable parameter, is the total number of tasks, is the number of failures; The stable parameters of the task execution node are calculated by the above formula: , using the stable parameter Characterize the stability of the task execution node, specifically, the stability parameters The larger the value, the higher the stability of the task execution node. Then, according to the stable parameters Sort multiple task execution nodes in ascending order to obtain a node sorting list.

[0020] The node sorting list includes a node number and a stable parameter, and the node number is specifically a number used to represent the task execution identity information.

[0021] S2, obtaining a plurality of tasks to be assigned, extracting task parameters of each task to be assigned from the plurality of tasks to be assigned, calculating an execution reference factor of each task to be assigned according to the task parameters of the task to be assigned, and sorting the plurality of tasks to be assigned according to the execution reference factor to obtain a task sorting list; Specifically, the tasks to be assigned can be scheduled tasks received by the task scheduling platform, such as scheduled article push every day, scheduled analysis of performance data, scheduled sending of reminders for work and off-get off work, etc. The task parameters include but are not limited to the start and end time points of subtask execution, the execution duration of subtask and the task cycle period. For scheduled tasks, generally an operation is performed at a specific time point, and the task is repeated at a certain period. Taking the scheduled analysis of performance data as an example, the performance data of employees is analyzed after get off work at 6 pm every day, and the analysis time is 5 minutes. The start and end time points of the subtask execution are 6 pm, the subtask execution time is 5 minutes, and the task cycle period is 24 hours. The execution reference factor of each task to be assigned is calculated according to the task parameters of the task to be assigned.

[0022] In this embodiment, the execution reference factor is calculated according to the task parameters of the task to be assigned according to the following formula: ;in, Represents the execution reference factor, Indicates the execution time of the subtask. Indicates the task cycle period, Subtask execution time The weight parameter, Task cycle The weight parameter of The execution reference factor of each task to be assigned is calculated by the above formula , using the execution reference factor Characterizes the complexity of the task to be assigned, specifically, the execution reference factor The larger it is, the lower the complexity of the task to be assigned. It is worth noting that the shorter the subtask execution time and the shorter the task cycle, the lower the complexity of the task to be assigned.

[0023] After calculating the execution reference factor of each task to be assigned, Sort multiple tasks to be assigned from large to small to obtain a task sorting list, wherein the task sorting list at least includes a task number, a task parameter and an execution reference factor, and the task number is used to uniquely represent one of the tasks to be assigned.

[0024] S3. Allocate multiple tasks to be assigned according to the node sorting list and the task sorting list to obtain a task allocation plan, and the scheduling platform schedules tasks according to the task allocation plan.

[0025] Specifically, after obtaining the node sorting list and the task sorting list, multiple tasks to be assigned are assigned according to the node sorting list and the task sorting list, wherein the principle of assignment is to assign task execution nodes to multiple tasks to be assigned in sequence according to the order in the task sorting list, and for any task to be assigned, determine in turn whether the task execution nodes recorded in the node sorting list meet the assignment conditions, thereby completing the assignment of all tasks to be assigned.

[0026] In this embodiment, the following allocation steps are specifically included: S11. For the task sorting list, the execution reference factor of the i-th item in the task sorting list is , let i=1; S12, use the i-th item in the task sorting list to traverse the node sorting list, and determine whether the i-th item in the task sorting list can be assigned to the task execution node corresponding to the first item in the node sorting list according to the task parameter corresponding to the i-th item in the task sorting list, if yes, record the assignment relationship and go to S13, otherwise determine whether the i-th item in the task sorting list can be assigned to the task execution node corresponding to the next item in the node sorting list, and so on until the i-th item in the task sorting list matches the task execution node, and go to step S13; Specifically, in this step, based on the subtask execution start and end time points and subtask execution duration of the i-th item in the task sorting list, it is determined whether the i-th item in the task sorting list overlaps with the subtask execution start and end time points and subtask execution duration of the task assigned to the task execution node corresponding to the first item in the node sorting list, and whether there is any overlap. If yes, the i-th item in the task sorting list can be assigned to the task execution node corresponding to the first item in the node sorting list; otherwise, it means that the i-th item in the task sorting list cannot be assigned to the task execution node corresponding to the first item in the node sorting list.

[0027] It is worth noting that each task execution node may already have an assigned task. When a new task is to be assigned to any task execution node, it is determined whether the new task conflicts with the original task. Specifically, whether there is a conflict can be analyzed based on the start and end time points of the subtask execution, the execution duration of the subtask, etc. In an optional implementation process, for any task execution node, the free time of the task execution node can be determined by analyzing the relevant information of the task that has been assigned to the task execution node. For the task to be assigned, if the task to be assigned will occupy time other than the free time of the task execution node, it means that the task to be assigned cannot be assigned to the task execution node.

[0028] S13, determine whether "i<n" holds, where n is the total number of tasks in the task sorting list. If so, set i=i+1 and go to step S12, otherwise go to step S14; S14. Output all the allocation relationships recorded in step S12 to generate a task allocation plan.

[0029] Through the above steps, a task allocation scheme recording the allocation relationship of multiple tasks to be allocated can be obtained, and the task scheduling platform can allocate the multiple tasks to be allocated according to the task allocation scheme.

[0030] In an optional implementation scheme, for step S12, in the process of assigning suitable task execution nodes to multiple tasks to be assigned, in order to improve the allocation efficiency, after any task to be assigned in the task sorting list is assigned to one of the task execution nodes in the node sorting list, the task to be assigned to the task execution node is regarded as a task that has been assigned to the execution node, and when judging whether the next task to be assigned can be assigned to the task execution node, the relevant information of the task to be assigned that has been prepared to be assigned to the task execution node is also considered to avoid conflicts in the allocation relationship.

[0031] In the process of processing tasks through different nodes, due to the differences in the stability of different nodes, if the stability of the node is not taken into consideration, and the tasks are simply scheduled based on whether there is a conflict between the tasks assigned to the node and the tasks to be assigned, then for some tasks with long subtask execution time and short task cycle, if they are finally assigned to a node with relatively poor stability, if the node fails in the middle of the task execution, such as when it is about to end, it will cause a large loss. However, if it is assigned to a node with relatively good stability and some tasks with short subtask execution time and long task cycle are assigned to nodes with relatively poor stability, then even if the node fails, the wasted computing resources will be relatively small.

[0032] A task scheduling method for a scheduling platform provided in an embodiment of the present invention analyzes the historical execution data of nodes to evaluate the stability of nodes, analyzes the task parameters of tasks to be assigned to evaluate the complexity of tasks, comprehensively considers the stability of nodes and the complexity of tasks, generates a task allocation plan, improves the efficiency of server processing tasks, and improves the efficiency of computing resource utilization.

[0033] See also Figure 2 , is a structural diagram of a task scheduling system of a scheduling platform provided in an embodiment of the present invention. As another aspect of the present invention, based on the task scheduling method of a scheduling platform provided above, a task scheduling system of a scheduling platform is also provided, including: A data acquisition module is used to acquire historical execution data of multiple task execution nodes and multiple tasks to be assigned; A node analysis module, used to determine the stability parameters of multiple task execution nodes according to the historical execution data, sort the multiple task execution nodes according to the stability parameters, and obtain a node sorting list, wherein the node sorting list includes at least a node number and a stability parameter; The task analysis module is used to extract the task parameters of each task to be assigned from the multiple tasks to be assigned, calculate the execution reference factor of each task to be assigned according to the task parameters of the task to be assigned, sort the multiple tasks to be assigned according to the execution reference factor, and obtain a task sorting list, wherein the task sorting list at least includes the task number, the task parameter and the execution reference factor; The task scheduling module is used to allocate multiple tasks to be allocated according to the node sorting list and the task sorting list, and obtain a task allocation plan used by the scheduling platform for task scheduling.

[0034] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention. Parts not described in detail in this specification belong to the prior art known to those skilled in the art.

Claims

1. A task scheduling method for a scheduling platform, characterized in that: include: Acquire historical execution data of multiple task execution nodes, determine stability parameters of the multiple task execution nodes according to the historical execution data, sort the multiple task execution nodes according to the stability parameters to obtain a node sorting list, wherein the node sorting list includes node numbers and the stability parameters; Acquire multiple tasks to be assigned, extract task parameters of each task to be assigned from the multiple tasks to be assigned, the task parameters include subtask execution start and end time points, subtask execution duration and task cycle period, calculate the execution reference factor of each task to be assigned according to the task parameters of the tasks to be assigned, sort the multiple tasks to be assigned according to the execution reference factor, and obtain a task sorting list, the task sorting list includes the task number, the task parameters and the execution reference factor; A plurality of tasks to be assigned are assigned according to the node sorting list and the task sorting list to obtain a task assignment plan, and the scheduling platform performs task scheduling according to the task assignment plan.

2. The method according to claim 1, characterized in that The step of determining stable parameters of a plurality of task execution nodes according to the historical execution data, and sorting the plurality of task execution nodes according to the stable parameters to obtain a node sorting list includes: The total number of tasks executed by each task execution node and the number of faults that occurred during the task execution are determined based on the historical execution data, and the stability parameter is calculated using the following formula: ;in, is a stable parameter, is the total number of tasks, is the number of failures; After calculating the stable parameters of multiple task execution nodes Then, according to the stable parameters The plurality of task execution nodes are sorted in ascending order to obtain the node sorting list.

3. The method according to claim 1, characterized in that The step of calculating the execution reference factor of each task to be assigned according to the task parameters of the task to be assigned, and sorting the multiple tasks to be assigned according to the execution reference factor to obtain a task sorting list includes: According to the task parameters of the task to be assigned, the execution reference factor is calculated according to the following formula: ;in, Represents the execution reference factor, Indicates the execution time of the subtask. Indicates the task cycle period, Subtask execution time The weight parameter, Task cycle The weight parameter of After calculating the execution reference factor of each task to be assigned Then, according to the execution reference factor The plurality of tasks to be assigned are sorted in order from large to small to obtain the task sorting list.

4. The method according to claim 3, characterized in that The step of allocating a plurality of tasks to be allocated according to the node sorting list and the task sorting list to obtain a task allocation scheme includes: S11. For the task sorting list, the execution reference factor of the i-th item in the task sorting list is , let i=1; S12, traverse the node sorting list with the i-th item of the task sorting list, and judge whether the i-th item of the task sorting list can be assigned to the task execution node corresponding to the first item of the node sorting list according to the task parameter corresponding to the i-th item of the task sorting list, if yes, record the assignment relationship and go to S13, otherwise judge whether the i-th item of the task sorting list can be assigned to the task execution node corresponding to the next item of the node sorting list, and so on until the i-th item of the task sorting list matches the task execution node, and go to step S13; S13, determine whether "i<n" holds, where n is the total number of tasks in the task sorting list. If so, set i=i+1 and go to step S12, otherwise go to step S14; S14. Output all the allocation relationships recorded in step S12 to generate the task allocation plan.

5. The method according to claim 4, characterized in that For step S12, judging whether the i-th item in the task sorting list can be allocated to the task execution node corresponding to the first item in the node sorting list according to the task parameter corresponding to the i-th item in the task sorting list includes: According to the subtask execution start and end time points and subtask execution duration of the i-th item in the task sorting list, determine whether the i-th item in the task sorting list overlaps with the subtask execution start and end time points and subtask execution duration of the task assigned to the task execution node corresponding to the first item in the node sorting list, and whether there is any overlap. If not, the i-th item in the task sorting list can be assigned to the task execution node corresponding to the first item in the node sorting list; otherwise, it means that the i-th item in the task sorting list cannot be assigned to the task execution node corresponding to the first item in the node sorting list.

6. The method according to claim 5, characterized in that Also includes: After any to-be-assigned task in the task sorting list is assigned to one of the task execution nodes in the node sorting list, the to-be-assigned task assigned to the task execution node is regarded as a task that has been assigned to the execution node.

7. A task scheduling system of a scheduling platform, wherein the task scheduling system of the scheduling platform is applied to the method according to any one of claims 1 to 6, characterized in that: include: A data acquisition module is used to acquire historical execution data of multiple task execution nodes and multiple tasks to be assigned; A node analysis module, used to determine the stability parameters of multiple task execution nodes according to the historical execution data, and sort the multiple task execution nodes according to the stability parameters to obtain a node sorting list; A task analysis module is used to extract task parameters of each task to be assigned from multiple tasks to be assigned, calculate the execution reference factor of each task to be assigned according to the task parameters of the task to be assigned, and sort the multiple tasks to be assigned according to the execution reference factor to obtain a task sorting list; The task scheduling module is used to allocate multiple tasks to be allocated according to the node sorting list and the task sorting list, and obtain a task allocation plan used by the scheduling platform for task scheduling.

8. The system according to claim 7, characterized in that The node analysis module and the task analysis module further include: The node sorting list generated by the node analysis module includes the node number and the stability parameter, and the task sorting list generated by the task analysis module includes the task number, the task parameter and the execution reference factor.

Images