Task arrangement method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a task arrangement method and device, electronic equipment and a storage medium, and relates to the technical field of task arrangement. The method comprises the following steps: acquiring historical scheduling operation data of each task in the same arrangement period; and according to the historical scheduling operation data, arranging each task based on resource constraint limitation, and obtaining planned starting time of each task. According to the method, all the tasks are arranged on the basis of resource constraint and limitation in combination with historical scheduling operation data, the planned starting time of each task is obtained, resource waste and conflicts are avoided, cluster resources can be fully utilized, and the task execution efficiency can also be improved.

Description

Technical Field

[0001] The present invention relates to the field of task scheduling technology, and in particular to a task scheduling method, device, electronic equipment, storage medium and program product. Background Art

[0002] At present, the batch scheduling system of the enterprise big data platform has significant shortcomings in terms of intelligence and automation. Especially during business peak periods, scheduling tasks often face serious queuing and failure problems, resulting in low data processing efficiency. During business idle periods, big data platform resources are relatively idle, resulting in a serious imbalance between task execution and resource usage. This imbalance not only affects the operational efficiency of the enterprise, but also wastes precious computing resources. Therefore, there is an urgent need for a task scheduling method that can achieve effective resource utilization. Summary of the invention

[0003] The embodiments of the present invention provide a task scheduling method, device, electronic device, storage medium and program product to solve the technical problem of imbalance between task execution and resource usage.

[0004] In a first aspect, an embodiment of the present invention provides a task scheduling method, comprising: obtaining historical scheduling and operation data of each task in the same scheduling cycle; scheduling each task based on resource constraints according to the historical scheduling and operation data to obtain a planned start time for each task.

[0005] In some embodiments, the tasks are scheduled based on resource constraints according to the historical scheduling operation data to obtain the planned start time of each task, including: determining the execution duration, priority, resource requirement information of each task, and the dependency relationship between the tasks according to the historical scheduling operation data; determining the task scheduling order according to the priority of each task and the dependency relationship between the tasks; determining the planned start time of each task in turn according to the task scheduling order according to the execution duration and resource requirement information of each task.

[0006] In some embodiments, the method of determining the planned start time of each task in sequence according to the task scheduling order based on the execution duration and resource requirement information of each task includes: determining the task completion time of the i-th predecessor dependent task corresponding to the i-th task; wherein i is an integer greater than 1; the i-th task is the task ranked as the i-th task in the task scheduling order; determining the first planned start time of the i-th task based on the task completion time of the i-th predecessor dependent task; determining the i-th running task that is in a running state at the first planned start time of the i-th task from the i-1 tasks based on the planned start time and execution duration of the first i-1 tasks; when i=2, the planned start time of the i-1 task is a set value; determining the i-th idle resource information based on the resource requirement information of the i-th running task; determining the planned start time of the i-th task based on the i-th idle resource information and the resource requirement information of the i-th task; setting i to i+1, and returning to the step of determining the first planned start time of the i-th task until all the tasks are scheduled.

[0007] In some embodiments, determining the planned start time of the i-th task based on the i-th idle resource information and the resource requirement information of the i-th task includes: determining that the i-th idle resource information meets the resource requirement for execution of the i-th task based on the i-th idle resource information and the resource requirement information of the i-th task, and determining the first planned start time of the i-th task as the planned start time of the i-th task; or, determining that the i-th idle resource information does not meet the resource requirement for execution of the i-th task based on the i-th idle resource information and the resource requirement information of the i-th task, determining the earliest task completion time of the i-th running task, updating the earliest task completion time as the first planned start time of the i-th task, and continuing to determine the planned start time of the i-task based on the updated first planned start time of the i-task.

[0008] In some embodiments, determining the first planned start time of the i-th task as the planned start time of the i-th task includes: determining that the first planned start time of the i-th task is less than or equal to the latest planned start time of the i-th task, and determining the first planned start time of the i-th task as the start execution time of the i-th task; or, determining that the first planned start time of the i-th task is greater than the latest planned start time of the i-th task, and determining the latest planned start time of the i-th task as the planned start time of the i-th task.

[0009] In some embodiments, determining the task scheduling order based on the priority of each task and the dependency relationship between the tasks includes: constructing a directed acyclic graph based on the dependency relationship between the tasks; and topologically sorting the directed acyclic graph based on the Kahn algorithm combined with the priority of each task to obtain the task scheduling order.

[0010] In a second aspect, an embodiment of the present invention provides a task scheduling device, including: The acquisition module is used to obtain the historical scheduling and operation data of each task in the same orchestration cycle; The scheduling module is used to schedule the tasks according to the historical scheduling operation data and based on resource constraints to obtain the planned start time of each task.

[0011] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory storing a computer program, wherein the processor implements the task scheduling method described in the first aspect when executing the program.

[0012] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the task scheduling method described in the first aspect.

[0013] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the task scheduling method described in the first aspect when executed by a processor.

[0014] The task scheduling method, device, electronic device, storage medium and program product provided by the embodiment of the present invention obtain the historical scheduling operation data of each task in the same scheduling cycle; according to the historical scheduling operation data, each task is scheduled based on resource constraints to obtain the planned start time of each task. The present invention combines the historical scheduling operation data and schedules each task based on resource constraints to obtain the planned start time of each task, so as to avoid waste and conflict of resources, which can not only make full use of cluster resources, but also improve the efficiency of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 One of the flowcharts of the task scheduling method provided by an embodiment of the present invention; Figure 2 The second flowchart of the task scheduling method provided by the embodiment of the present invention; Figure 3 The third flowchart of the task scheduling method provided by the embodiment of the present invention; Figure 4 A fourth flowchart of the task scheduling method provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a task scheduling device provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] At present, the batch scheduling system of the enterprise big data platform mainly relies on the operation and maintenance personnel to manually set the execution time and sequence, and configure it based on the dependency and priority of the task, which limits the flexibility and efficiency of task scheduling and causes a serious imbalance between task execution and resource utilization. This imbalance not only affects the operational efficiency of the enterprise, but also wastes valuable computing resources. Specific disadvantages include: 1. Fixed time scheduling: The existing task scheduling is usually executed based on a fixed schedule, which is not flexible and efficient for the use of system resources. If the task arrives at the scheduled time and the system resources happen to be under high load, the task may need to wait, resulting in resource waste and extended task response time. 2. Uneven resource allocation: Due to the fixed task execution time, resources may be unevenly distributed in time, with idle resources in some periods and tight resources in other periods, making it impossible to achieve optimal resource allocation and maximum utilization. 3. High task failure rate: During periods of resource shortage, tasks may fail to execute due to insufficient resources, affecting the stability of the task and the reliability of the system. 4. Inefficiency: Since only human intervention can be used to adjust the task execution plan, the overall task execution efficiency is not high, affecting the throughput and response speed of the system.

[0019] In order to solve the above problems, the present invention provides a task scheduling method, device, electronic device, storage medium and program product.

[0020] Figure 1This is one of the flow charts of the task scheduling method provided by the embodiment of the present invention. Figure 1 As shown, the method may include the following steps.

[0021] Step 101, obtaining historical scheduling and operation data of each task in the same orchestration cycle.

[0022] In some embodiments, the same orchestration cycle refers to the cycle type of the batch processing task of the big data platform, such as daily execution, monthly execution, hourly execution, etc. In other words, tasks of the same orchestration cycle can be uniformly orchestrated, and when there are multiple orchestration cycles, tasks of each orchestration cycle can be separately orchestrated.

[0023] In some embodiments, since the big data platform usually executes tasks in a tenant resource isolation manner, the tasks in the same orchestration cycle may be the tasks of the same tenant in the same orchestration cycle.

[0024] In some embodiments, the historical scheduling operation data of each task may include scheduling operation data of multiple historical operation cycles, and the specific number of operation cycles included may be preset based on actual needs, for example, scheduling operation data of 7 operation cycles may be included. The historical scheduling operation data of each task may include the start time and end time of each task in each operation cycle, the priority of each task, the dependency relationship between each task, and the resource occupancy of the big data platform at each time.

[0025] In some embodiments, the implementation process of obtaining historical scheduling and operation data of each task in the same orchestration cycle may include: first obtaining the historical operation data of the scheduling task on the big data platform, which historical operation data may be the historical operation data of the scheduling task for a continuous period of time, such as the historical operation data of the scheduling task in the past N days; performing data cleaning on the historical operation data of the scheduling task to filter out tasks of other tenants and retain only the task information of the target tenant; converting the data structure, which may include data type conversion (such as conversion of numerical type to time type, etc.), data splitting (split of data in different periods), and data dimension conversion (such as conversion of milliseconds to minutes, etc.); extracting the data after the structure conversion to obtain the historical scheduling and operation data of each task in the same orchestration cycle with a preset number of cycles.

[0026] In some embodiments, based on the acquired historical running data of the scheduled tasks, the historical scheduling running data of each task in each orchestration period may be acquired, and the tasks in each orchestration period may be orchestrated respectively.

[0027] Step 102: Schedule each task based on resource constraints according to historical scheduling operation data to obtain a planned start time for each task.

[0028] In some embodiments, resource constraint means that the content resource demand during multi-task parallel operation needs to be less than or equal to the total resource upper limit, and the resource utilization rate in the entire cycle is maximized to reduce the resource idle time.

[0029] It can be understood that since the historical scheduling operation data can include relevant information such as the operation data of each task in the historical operation cycle, resource occupancy data, priority and the relationship between each task, resource constraints can be imposed based on the historical scheduling operation data to optimize the historical scheduling operation data and obtain the planned start time of each task after optimization.

[0030] In some embodiments, based on historical scheduling and operation data, the average execution time of each task in the historical operation cycle and the average resource requirement of each task can be calculated, and the priority of each task and the dependency relationship between tasks can also be obtained; based on the priority of each task and the dependency relationship between tasks, the scheduling order of each task is determined; according to the average execution time and the average resource requirement of each task, the initial start time of each task is determined in turn according to the scheduling order; based on the initial start time, the resource occupancy at each moment is calculated; with the resource occupancy being less than or equal to the resource upper limit as the first goal and the resource utilization being maximized within the scheduling cycle as the second goal, the initial start time of each task is optimized and iterated respectively, and finally the planned start time of each task is obtained.

[0031] In other embodiments, step 102 can be implemented based on a pre-trained orchestration model. The training process of the orchestration model includes: obtaining historical scheduling operation data samples of each task in the same orchestration cycle, the historical scheduling operation data samples are the same as the number of historical operation cycles contained in the historical scheduling operation data in step 101; inputting the historical scheduling operation data samples into the initial orchestration model to extract the features of each task, and outputting the planned start time prediction value of each task based on the extracted features; determining the resource utilization rate of the entire cycle and the resource occupancy at each moment through simulation operation based on the planned time prediction value of each task; based on the resource utilization rate of the entire cycle and the resource occupancy at each moment, continuously adjusting the model parameters for repeated iterations until the resource occupancy at each moment and the resource utilization rate of the entire cycle meet the preset conditions, and obtaining the trained orchestration model; inputting the historical scheduling operation data into the trained orchestration model to orchestrate each task and obtain the planned start time of each task.

[0032] In some embodiments, the scheduled task schedule for the next operating cycle can be determined based on the planned start time of each task. For example, in the next operating cycle, the planned start time of each task can be used as the scheduled task start time, that is, the planned start time of each task can be used as the start time of the corresponding task in the next operating cycle.

[0033] In some embodiments, the planned start time of each task of the next operation cycle can be determined based on step 101 and step 102, that is, it can be executed once every other operation cycle. In other embodiments, the planned start time of each task of the subsequent multiple operation cycles can also be determined based on step 101 and step 102, that is, it can be executed once every multiple operation cycles.

[0034] According to the task scheduling method implemented by the present invention, the historical scheduling operation data of each task in the same scheduling cycle is obtained; according to the historical scheduling operation data, each task is scheduled based on resource constraints to obtain the planned start time of each task. The present invention combines the historical scheduling operation data and schedules each task based on resource constraints to obtain the planned start time of each task, so as to avoid waste and conflict of resources, which can not only make full use of cluster resources, but also improve task execution efficiency.

[0035] Figure 2 The second flowchart of the task scheduling method provided by the embodiment of the present invention is as follows. Figure 2 As shown, based on the above embodiments, Figure 1 The implementation process of step 102 in may include the following steps.

[0036] Step 201, determining the execution time, priority, resource requirement information of each task, and the dependency relationship between the tasks based on historical scheduling operation data.

[0037] In some embodiments, the historical scheduling operation data includes the start time and end time of each task in each historical operation cycle. Based on the start time and end time of each task, the execution duration of each task in each historical operation cycle can be determined. The execution duration of each historical operation cycle is averaged to obtain the execution duration of each task.

[0038] In some embodiments, the historical scheduling operation data includes priority configuration information of each task and successor task configuration information of each task. Based on the above information, the priority of each task and the dependency relationship between tasks can be determined.

[0039] Step 202: Determine the task scheduling order according to the priority of each task and the dependencies between the tasks.

[0040] It can be understood that the predecessor tasks that the subsequent tasks depend on need to be scheduled first, and the tasks with high priority should be scheduled first to ensure the smooth execution of each task, and also to ensure that the tasks with high priority are executed first.

[0041] In some embodiments, according to the priority of each task and the dependency relationship between each task, the implementation process of determining the task scheduling order may include: determining the hierarchical relationship between the tasks based on the dependency relationship between each task, there is a dependency relationship between the tasks at each layer, the first layer tasks are tasks that can be executed independently, the second layer tasks are tasks that depend on the execution results of the first layer tasks, the third layer tasks are tasks that depend on the execution results of the second layer tasks, and so on; based on the above hierarchical relationship, determining the preliminary task scheduling order, that is, if there are N layers of tasks in total, then the preliminary task scheduling order is first layer tasks, second layer tasks, third layer tasks, ..., Nth layer tasks; based on the priority of each task, determining the scheduling order of each task in each layer of tasks, so as to refine the first preliminary task scheduling order and obtain the task scheduling order.

[0042] In other embodiments, according to the priority of each task and the dependency relationship between the tasks, the implementation process of determining the task scheduling order may include: constructing a directed acyclic graph according to the dependency relationship between the tasks; and topologically sorting the directed acyclic graph based on the Kahn algorithm combined with the priority of each task to obtain the task scheduling order. The nodes in the directed acyclic graph are the tasks, and the edges between the nodes are the dependency relationships between the tasks.

[0043] Since there are a large number of nodes with in-degree 0 when topologically sorting based on the Kahn algorithm, there are multiple possibilities for sorting. In order to ensure that tasks with high priority are arranged first, the priority of each task can be considered during topological sorting. In other words, when topologically sorting the directed acyclic graph based on the Kahn algorithm, tasks with high priority are arranged first to improve the efficiency of task execution.

[0044] It should be noted that the task scheduling sequence may also include marks of dependency relationships between tasks. For example, the task scheduling sequence may be Task A (1), Task B (1), Task C (2), Task D (2), Task E (2), Task F (3), and Task G (3). That is, the relationship between tasks is marked by numbers. There is no relationship between Task A and Task B and they can be executed at the same time. There is a relationship between Task B and Task C and Task C needs to be executed after Task B is completed. There is no relationship between Task C, Task D, and Task E and they can be executed at the same time. There is a dependency relationship between Task E and Task F and Task F needs to be executed after Task E is completed. There is no dependency relationship between Task F and Task G and they can be executed at the same time.

[0045] Step 203, according to the execution time and resource requirement information of each task, the planned start time of each task is determined in sequence according to the task scheduling sequence.

[0046] In some embodiments, each task can be stored in a task list according to the order in which the tasks are scheduled; the i-th task in the task list is taken out, and the i-th dependent task that has a dependency relationship with the i-th task in the scheduled tasks is determined, and the latest completion time among the task completion times of the i-th dependent task is determined based on the execution duration and the planned start time of the i-th dependent task; the i-th running task that is in a running state at the latest completion time is determined, and based on the resource requirement information of the i-th running task and the resource requirement information of the i-th task, it is determined whether the i-th task can be run in parallel with the i-th running task; if the i-th task and the i-th running task can be run in parallel, the latest completion time is determined as the planned start time of the i-th task; if the i-th task cannot be run in parallel with the i-th running task, the first latest completion time of the i-th running task is determined The resource occupancy after the earliest completed task is determined, and whether the resource occupancy meets the resource demand information of the i-th task; if so, the task completion time of the first earliest completed task of the i-th running task is determined as the planned start time of the i-th task, the i-th task is put into the task execution list, and i+1 tasks are continued to be taken out from the task list; if not, the resource occupancy after the second earliest completed task of the i-th running task is continued to be determined until the resource demand information of the i-th task is met, the task completion time of the corresponding task is determined as the planned start time of the i-th task, the i-th task is put into the task execution list, and i+1 tasks are continued to be taken out from the task list.

[0047] According to the task scheduling method of the embodiment of the present invention, the execution time, priority, resource requirement information of each task, and the dependency relationship between each task are determined based on historical scheduling operation data; the task scheduling order is determined based on the priority of each task and the dependency relationship between each task; and the planned start time of each task is determined in turn according to the task scheduling order based on the execution time and resource requirement information of each task. On the one hand, this solution determines the task scheduling order by combining the dependency relationship and priority between tasks to ensure the smooth progress of tasks, and can also ensure that tasks with high priority are executed first. On the other hand, scheduling tasks based on the execution time and resource requirement information of tasks can reduce the waiting time of scheduling tasks, improve task execution efficiency, and also improve resource utilization.

[0048] Figure 3 The third flowchart of the task scheduling method provided by the embodiment of the present invention. Figure 3 As shown, based on the above embodiments, Figure 2 The implementation process of step 203 in may include the following steps.

[0049] Step 301, determine the latest task completion time of the i-th predecessor dependent task corresponding to the i-th task; wherein i is an integer greater than 1; the i-th task is the i-th task in the task scheduling sequence.

[0050] Among them, the i-th predecessor dependent task corresponding to the i-th task refers to the upper-level dependent task of the i-th task. For example, the start of the i-th task depends on the completion of task A, and the start of task A depends on the completion of task B. Then the i-th predecessor dependent task corresponding to the i-th task is task A.

[0051] In some embodiments, there may be one or more i-th predecessor dependent tasks corresponding to the i-th task, and when there are more than one i-th predecessor dependent tasks, determining the completion time of the i-th predecessor dependent task is to determine the completion time of each i-th predecessor dependent task. The task completion time of the i-th predecessor dependent task can be determined based on the planned start time of the i-th predecessor dependent task and the execution time of the i-th predecessor dependent task.

[0052] Step 302: Determine a first planned start time of the i-th task based on the completion time of the i-th predecessor dependent task.

[0053] In some embodiments, the first planned start time of the i-th task refers to the start time of the i-th task determined by taking into account dependencies between tasks but not taking into account resource usage.

[0054] In some embodiments, if the number of the i-th predecessor dependent task is 1, the task completion time of the i-th predecessor dependent task can be determined as the first planned start time of the i-th task. If the number of the i-th predecessor dependent task is multiple, the latest task completion time of the i-th predecessor dependent task can be determined as the first planned start time of the i-th task.

[0055] Step 303, based on the planned start time and execution duration of the first i-1 tasks, determine the i-th running task that is in running state at the first planned start time of the i-th task from the first i-1 tasks; when i=2, the planned start time of the i-1th task is the set value.

[0056] That is, based on the planned start time and execution duration of the first i-1 tasks, the i-th running task that is still in the running state when the time reaches the first planned start time of the i-th task is determined from the first i-1 tasks.

[0057] In some embodiments, the task completion time of each of the first i-1 tasks can be determined based on the planned start time and execution duration of the first i-1 tasks, and the task completion time of each of the first i-1 tasks can be compared with the first planned start time of the i-th task, and the task whose task completion time is later than the first planned start time of the i-th task can be determined as the i-th running task.

[0058] It should be noted that the number of tasks in the i-th running process may be 0, 1, or more.

[0059] Step 304: Determine the i-th idle resource information based on the resource demand information of the i-th running task.

[0060] The i-th idle resource information refers to the idle resource information of the data platform at the first planned start time of the i-th task.

[0061] In some embodiments, by summing up the resource requirement information of each task in the i-th running task, the total resource requirement information of the i-th running task can be obtained, and based on the resource upper limit information and the total resource requirement information of the i-th running task, the i-th idle resource information can be obtained.

[0062] Step 305 : determining a planned start time of the i th task based on the i th idle resource information and the resource demand information of the i th task.

[0063] It can be understood that based on the i-th idle resource information and the resource requirement information of the i-th task, it can be determined whether the idle resources at the first planned start time of the i-th task can meet the resource requirements of the i-th task, so as to determine whether it is feasible to execute the i-th task according to the corresponding first planned start time, and determine the planned start time of the i-th task if resource conditions permit.

[0064] In some embodiments, based on the i-th idle resource information and the resource requirement information of the i-th task, the implementation process of determining the planned start time of the i-th task may include the following steps.

[0065] Step 305-1, based on the i-th idle resource information and the resource requirement information of the i-th task, determine that the i-th idle resource information meets the resource requirement for executing the i-th task, and determine the first planned start time of the i-th task as the planned start time of the i-th task.

[0066] Step 305-2, based on the i-th idle resource information and the resource requirement information of the i-th task, determine that the i-th idle resource information does not meet the resource requirement for executing the i-th task, determine the earliest task completion time of the i-th running task, update the earliest task completion time to the first planned start time of the i-th task, and continue to determine the planned start time of the i-th task based on the updated first planned start time of the i-th task.

[0067] In some embodiments, if the i-th idle resource information is greater than or equal to the i-th task resource requirement information, it is determined that the i-th idle resource information meets the resource requirement for executing the i-th task; if the i-th idle resource information is less than the i-th task resource requirement information, it is determined that the i-th idle resource information does not meet the resource requirement for executing the i-th task.

[0068] In some embodiments, the implementation process of determining the earliest task completion time of the i-th running task includes: determining the task completion time of each task in the i-th running task based on the execution duration and planned start time of each task in the i-th running task; and determining the minimum value of the task completion time of each task in the i-th running task as the earliest task completion time of the i-th running task.

[0069] Among them, continuing to determine the planned start time of the i-th task based on the updated first planned start time of the i-th task is equivalent to returning to continue executing steps 303 to 305 based on the updated first planned start time of the i-th task until the planned start time of the i-th task is obtained.

[0070] Step 306, i is set to i+1, and the process returns to the step of determining the first planned start time of the i-th task until all tasks are scheduled.

[0071] According to the task scheduling method of the embodiment of the present invention, the first planned start time of each task is determined in turn according to the task scheduling order, and the idle resource information of each task when it is planned to be executed is determined based on the resource requirement information and execution duration of each task, and the planned start time of each task is determined based on the idle resource information and the resource requirement information of each task. The present invention optimizes the allocation of resources by analyzing with time as the main line and combining the direct dependency relationship and priority of tasks, which can not only improve resource utilization, but also improve the execution efficiency of tasks and improve the success rate of task execution.

[0072] In order to shorten the total execution time of batch running tasks and ensure that each task is completed according to the time required by the actual scenario, the embodiment of the present invention also introduces the latest planned start time of each task, as follows.

[0073] In some embodiments, Figure 3In step 305-1, the first planned start time of the i-th task is determined as the planned start time of the i-th task, including: determining that the first planned start time of the i-th task is less than or equal to the latest planned start time of the i-th task, and determining the first planned start time of the i-th task as the start execution time of the i-th task; or, determining that the first planned start time of the i-th task is greater than the latest planned start time of the i-th task, and determining the latest planned start time of the i-th task as the planned start time of the i-th task.

[0074] The latest planned start time of the i-th task refers to the latest start time according to the actual demand scenario, which can be set based on the actual demand. As an implementation method, the latest planned start time of the last task in the task scheduling sequence can be determined based on the actual demand scenario, and the latest planned start time of each task can be calculated forward according to the task scheduling sequence and the execution time of each task.

[0075] That is to say, when determining the planned start time of the i-th task, first compare the first planned start time of the i-th task with the latest start time of the i-th task. If the first planned start time of the i-th task is earlier than the latest start time of the i-th task, the planned start time of the i-th task is the first planned start time. If the first planned start time of the i-th task is later than the latest start time of the i-th task, the planned start time of the i-th task is forcibly determined as the latest start time to meet the needs of the actual scenario.

[0076] In order to facilitate an overall understanding of the task scheduling method of the above embodiment, the implementation process of the task scheduling method of the above embodiment will be introduced in the form of examples.

[0077] Figure 4 FIG4 is a flow chart of a task scheduling method provided by an embodiment of the present invention. Figure 4 As shown, the task scheduling method of the embodiment of the present invention includes the following steps.

[0078] (1) Data collection and processing. Collect the historical operation details of the platform scheduling tasks and perform preprocessing to obtain the historical scheduling operation data of each task in the unified scheduling cycle.

[0079] (2) Construct a directed acyclic graph. Construct a directed acyclic graph based on the dependencies between tasks.

[0080] (3) Combine the priorities of each task, perform topological sorting based on the Kahn algorithm, and generate a TaskList based on the sorting results. The TaskList contains tasks arranged in the order of task scheduling.

[0081] (4) Task scheduling based on resource constraints; (4-1) Initialize the earliest and latest start time of each task in the TaskList list; wherein the earliest start time is the first planned start time in the above embodiment, and the latest start time is the latest planned start time in the above embodiment; (4-2) Take tasks from the TaskList list and arrange them in sequence; (4-3) Calculate the resource usage at the beginning of the task plan; (4-4) Determine whether the idle resources at the beginning of the task plan meet the resource requirements for the task execution. If the idle resources at the beginning of the task plan meet the resource requirements for the task execution, continue to execute (4-5); otherwise, execute (4-7); (4-5) Determine whether the first planned start time of the task is less than the latest start time. If the first planned start time of the task is less than the latest start time, the first planned start time of the task is determined as the planned start time of the task, and the task is moved to the RunList list; if the first planned start time of the task is greater than or equal to the latest start time, the first planned start time of the task is determined as the latest start time of the task, and the task is moved to the RunList list; (4-6) If the TaskList list is not empty, return to continue executing (4-2). If the TaskList list is empty, move the tasks in the RunList list to the EndList list; (4-7) Determine the task with the earliest completion time in the RunList list, and move the task with the earliest completion time to the EndList list; (4-8) Update the first scheduled start time of the remaining tasks in the TaskList list based on the earliest task completion time, and return to continue execution (4-2); (5) Output the task scheduling plan. The planned start time of each task in the obtained EndList list is determined as the scheduling plan for the next cycle of scheduling tasks on the big data platform.

[0082] In order to implement the above embodiment, the present invention also provides a task scheduling device.

[0083] Figure 5 The structure diagram of the task scheduling device provided by the embodiment of the present invention is as follows. Figure 5 As shown, the device includes an acquisition module 510 and an orchestration module 520. The acquisition module is used to acquire the historical scheduling operation data of each task in the same orchestration cycle; the orchestration module is used to orchestrate each task based on the historical scheduling operation data and resource constraints to obtain the planned start time of each task.

[0084] In some embodiments, the orchestration module 520 is specifically used to: determine the execution duration, priority, resource requirement information of each task, and the dependencies between tasks based on historical scheduling operation data; determine the task orchestration order based on the priority of each task and the dependencies between tasks; determine the planned start time of each task in turn according to the task orchestration order based on the execution duration and resource requirement information of each task.

[0085] In some embodiments, the orchestration module 520 is also used to: determine the task completion time of the i-th predecessor dependent task corresponding to the i-th task; wherein i is an integer greater than 1; the i-th task is the task ranked as the i-th task in the task orchestration order; based on the task completion time of the i-th predecessor dependent task, determine the first planned start time of the i-th task; based on the planned start time and execution duration of the first i-1 tasks, determine the i-th running task that is in a running state at the first planned start time of the i-th task from the first i-1 tasks; when i=2, the planned start time of the i-1 task is a set value; based on the resource requirement information of the i-th running task, determine the i-th idle resource information; based on the i-th idle resource information and the resource requirement information of the i-th task, determine the planned start time of the i-th task; set i to i+1, and return to the step of determining the first planned start time of the i-th task until all tasks are orchestrated.

[0086] In some embodiments, the orchestration module 520 is further used to: determine, based on the i-th idle resource information and the resource requirement information of the i-th task, whether the i-th idle resource information meets the resource requirement for executing the i-th task, and determine the first planned start time of the i-th task as the planned start time of the i-th task; or, determine, based on the i-th idle resource information and the resource requirement information of the i-th task, whether the i-th idle resource information does not meet the resource requirement for executing the i-th task, determine the earliest task completion time of the i-th running task, update the earliest task completion time as the first planned start time of the i-th task, and continue to determine the planned start time of the i-th task based on the updated first planned start time of the i-th task.

[0087] In some embodiments, the orchestration module 520 is further used to: determine that the first planned start time of the i-th task is less than or equal to the latest planned start time of the i-th task, and determine the first planned start time of the i-th task as the start execution time of the i-th task; or, determine that the first planned start time of the i-th task is greater than the latest planned start time of the i-th task, and determine the latest planned start time of the i-th task as the planned start time of the i-th task.

[0088] In some embodiments, the orchestration module 520 is further used to: construct a directed acyclic graph according to the dependency relationship between the tasks; and topologically sort the directed acyclic graph based on the Kahn algorithm combined with the priority of each task to obtain the task orchestration order.

[0089] It should be noted that the above explanations and descriptions in the embodiment of the task scheduling method are also applicable to the task scheduling device in the embodiment of the present invention, and will not be repeated here.

[0090] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call a computer program in the memory 630 to execute the steps of the task scheduling method.

[0091] For example, the method includes: obtaining historical scheduling and operation data of each task in the same scheduling cycle; scheduling each task based on resource constraints according to the historical scheduling and operation data, and obtaining a planned start time for each task.

[0092] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps of the task scheduling method provided in the above embodiments.

[0094] The method comprises: obtaining historical scheduling and running data of each task in the same scheduling cycle; scheduling each task based on resource constraints according to the historical scheduling and running data, and obtaining the planned start time of each task.

[0095] On the other hand, an embodiment of the present invention further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable a processor to execute the steps of the task scheduling method provided in the above embodiments.

[0096] For example, the method includes: obtaining historical scheduling and operation data of each task in the same scheduling cycle; scheduling each task based on resource constraints according to the historical scheduling and operation data to obtain a planned start time for each task.

[0097] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0098] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A task scheduling method, characterized in that: include: Obtain the historical scheduling and operation data of each task in the same orchestration cycle; According to the historical scheduling operation data, the tasks are scheduled based on resource constraints to obtain a planned start time for each task.

2. The method according to claim 1, characterized in that The step of scheduling the tasks based on the resource constraints according to the historical scheduling operation data to obtain the planned start time of each task includes: Determine the execution time, priority, resource requirement information of each task, and the dependency relationship between the tasks according to the historical scheduling operation data; Determine the task scheduling order according to the priority of each task and the dependency relationship between the tasks; According to the execution time and resource requirement information of each task, the planned start time of each task is determined in sequence according to the task scheduling sequence.

3. The method according to claim 2, characterized in that The step of determining the planned start time of each task in sequence according to the execution time and resource requirement information of each task and in accordance with the task scheduling sequence includes: Determine the task completion time of the i-th predecessor dependent task corresponding to the i-th task; wherein i is an integer greater than 1; the i-th task is the task ranked as the i-th task in the task scheduling sequence; Determine a first planned start time of the i-th task based on the task completion time of the i-th predecessor dependent task; Based on the planned start time and execution duration of the first i-1 tasks, determine the i-th running task that is in the running state at the first planned start time of the i-th task from the first i-1 tasks; when i=2, the planned start time of the i-1th task is the set value; Determining i-th idle resource information based on the resource requirement information of the i-th running task; Determining a planned start time of the i-th task based on the i-th idle resource information and the resource requirement information of the i-th task; Set i to i+1, and return to the step of determining the first planned start time of the i-th task until all the tasks are scheduled to be completed.

4. The method according to claim 3, characterized in that The determining the planned start time of the i-th task based on the i-th idle resource information and the resource requirement information of the i-th task includes: Based on the i-th idle resource information and the resource requirement information of the i-th task, determining that the i-th idle resource information meets the resource requirement for executing the i-th task, and determining the first planned start time of the i-th task as the planned start time of the i-th task; or Based on the i-th idle resource information and the resource requirement information of the i-th task, it is determined that the i-th idle resource information does not meet the resource requirement for executing the i-th task, the earliest task completion time of the i-th running task is determined, the earliest task completion time is updated to the first planned start time of the i-th task, and the planned start time of the i-task is continued to be determined based on the updated first planned start time of the i-task.

5. The method according to claim 4, characterized in that The step of determining the first planned start time of the i-th task as the planned start time of the i-th task includes: Determine that the first planned start time of the i-th task is less than or equal to the latest planned start time of the i-th task, and determine the first planned start time of the i-th task as the start execution time of the i-th task; or, It is determined that the first planned start time of the i-th task is greater than the latest planned start time of the i-th task, and the latest planned start time of the i-th task is determined as the planned start time of the i-th task.

6. The method according to any one of claims 2 to 5, characterized in that Determining the task arrangement order according to the priority of each task and the dependency relationship between the tasks includes: Constructing a directed acyclic graph according to the dependency relationship between the tasks; Based on the Kahn algorithm combined with the priority of each task, the directed acyclic graph is topologically sorted to obtain the task scheduling order.

7. A task scheduling device, characterized in that: include: The acquisition module is used to obtain the historical scheduling and running data of each task in the same orchestration cycle; The scheduling module is used to schedule the tasks according to the historical scheduling operation data and based on resource constraints to obtain the planned start time of each task.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the task scheduling method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task scheduling method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the task scheduling method according to any one of claims 1 to 6 is implemented.

Images