Task scheduling method, electronic equipment, computer storage medium and computer program product

Abstract

The embodiment of the invention provides a task scheduling method, electronic equipment, a computer storage medium and a computer program product. The task scheduling method comprises the steps of obtaining a plurality of tasks to be scheduled; the plurality of tasks comprise an upstream task composed of a plurality of upstream task fragments and a downstream task composed of a plurality of downstream task fragments; determining a task fragment dependency relationship, wherein the task fragment dependency relationship represents a dependency relationship between each upstream task fragment and each downstream task fragment; in response to completion of execution of the current upstream task fragment, allocating a task executor to a target downstream task fragment depending on the completed upstream task fragment, so that the task executor executes the target downstream task; the target downstream task fragment is determined through the task fragment dependency relationship. According to the embodiment of the invention, the resource utilization rate of the task executor can be improved, the task processing efficiency is improved, the task execution time is shortened, and the task processing delay is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a task scheduling method, electronic equipment, computer storage medium, and computer program product. Background Art

[0002] Batch processing of tasks refers to a one-time processing of the data set involved in the task, usually after a certain amount of data has accumulated. Batch processing is delayed in nature and is suitable for large-scale data analysis tasks. It can be run regularly through task scheduling, such as processing specific data sets every day or every hour. For example: in the field of index-based web search technology, the full index construction process can be divided into the following three tasks in order of execution: light feature merging task, data selection task, and heavy feature merging task. Among them, each of the above tasks can be executed in batch mode.

[0003] Traditionally, for multi-task scenarios with dependencies between tasks, that is, scenarios where the output of an upstream task is used as the input of a downstream task, upstream tasks and downstream tasks are often executed in sequence. In the above index building process, the downstream task of data selection is usually started after the upstream task of merging all light features is completed.

[0004] However, when one or some subtasks (i.e., task shards) of the upstream light feature merging task take a long time to execute, it may significantly slow down the completion of the entire light feature merging task, thereby affecting the start timing of the downstream data selection task. Therefore, the above-mentioned sequential batch processing method not only reduces resource utilization, but may also cause large delays in task processing. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a task scheduling solution to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present application, a task scheduling method is provided, including: Acquire multiple tasks to be scheduled; the multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; Determine a task slice dependency relationship, wherein the task slice dependency relationship represents a dependency relationship between each upstream task slice and each downstream task slice; In response to the completion of the execution of the current upstream task slice, a task executor is allocated to a target downstream task slice that depends on the completed upstream task slice, so that the target downstream task is executed by the task executor; the target downstream task slice is determined by the task slice dependency.

[0007] According to a second aspect of an embodiment of the present application, another task scheduling method is provided, including: In a scheduling process for multiple tasks, determining a task slice dependency relationship; the multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the task slice dependency relationship represents a dependency relationship between each upstream task slice and each downstream task slice; Receive a task assignment request sent by a target task executor; Allocate the unexecuted target downstream task slice to the target task executor so as to execute the target downstream task slice through the target task executor; the target downstream task slice is a downstream task slice determined according to the task slice dependency and depends on the completed upstream task slice.

[0008] According to a third aspect of an embodiment of the present application, another task scheduling method is provided, including: Sending a task allocation request to a task scheduler so that the task scheduler allocates unexecuted target downstream task slices to the target task executor during the scheduling process of multiple tasks; Obtain the target downstream task slice assigned by the task scheduler, and execute the target downstream task slice; The multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the target downstream task slice is a downstream task slice that depends on the completed upstream task slice.

[0009] According to the fourth aspect of the embodiments of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in any one of the first to third aspects.

[0010] According to a fifth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method as described in any one of the first to third aspects is implemented.

[0011] According to the task scheduling scheme provided in the embodiment of the present application, multiple tasks to be scheduled are obtained; the multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the task slice dependency is determined, and the task slice dependency characterizes the dependency between each upstream task slice and each downstream task slice; in response to the completion of the execution of the current upstream task slice, a task executor is allocated to the target downstream task slice that depends on the completed upstream task slice, so that the task executor executes the target downstream task; the target downstream task slice is determined by the above-mentioned task slice dependency.

[0012] In the above-mentioned embodiment of the present application, after obtaining a plurality of batch-processed upstream and downstream tasks with dependencies, the dependency of the downstream task on the upstream task at the overall level is decomposed and converted into the dependency of the downstream task slice on the individual upstream task slice through the dependency relationship between each task slice. According to the dependency relationship between the task slices, during the scheduling and execution of the task slices, there is no need to wait for the upstream task to be executed as a whole. Instead, after the individual upstream task slices in the upstream task are executed, the execution program of the downstream task slice with the dependency relationship with the above-mentioned individual upstream task slice can be started. Therefore, through the scheme of the embodiment of the present application, after receiving the batch-processed upstream and downstream tasks with dependencies, the upstream and downstream tasks originally processed in batch processing can be converted into task slices processed in stream processing mode, and the stream processing at the task slice level is realized while retaining the batch processing operation on the internal data of each task slice.

[0013] Through the above method, the task submitter can still set and submit the task logic in the manner of batch processing tasks without making any changes to the task logic. In the process of task scheduling and execution, the solution provided by the embodiment of the present application can automatically convert batch processing tasks into stream processing task slices. Therefore, pseudo-streaming processing of batch tasks is achieved, so that the batch task execution process has the advantages of stream processing: data continues to flow, improving the resource utilization of the task executor, improving task processing efficiency, shortening task execution time, and reducing task processing delays.

[0014] In addition, in the embodiment of the present application, once an upstream task slice is executed, the corresponding downstream task slice execution program can be started without waiting for other upstream task slices to be executed. Therefore, for the first slice execution result corresponding to the first upstream task slice that is executed, the use of the embodiment of the present application can effectively shorten the output time of the above-mentioned first slice execution result. This can effectively improve the troubleshooting efficiency for the scenario of checking the correctness of task logic based on task execution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 A schematic diagram of multiple batch tasks with association relationships in the index building scenario; Figure 2 A schematic diagram of a batch task scheduling process in the related art; Figure 3 is a flowchart of a task scheduling method according to an embodiment of the present application; Figure 4 for Figure 1 The diagram of task slicing dependency in the batch processing task scenario shown in the figure; Figure 5 A schematic diagram of a comparison between a batch task scheduling process according to an embodiment of the present application and a batch task scheduling process according to a related art; Figure 6 Another comparison diagram between the batch task scheduling process according to the embodiment of the present application and the batch task scheduling process according to the related art; Figure 7 A schematic diagram of an exemplary system for a task scheduling method applicable to an embodiment of the present application; Figure 8 is a flowchart of another task scheduling method according to an embodiment of the present application; Fig. 9 For Figure 8 A schematic diagram of the interactive flow of the task scheduling process corresponding to the illustrated embodiment; Fig.10 For execution Figure 8 A schematic diagram of the functional architecture of a task scheduler of the illustrated embodiment; Fig.11 is a flowchart of steps of another task scheduling method according to an embodiment of the present application; Fig.12 This is a diagram of the interactive process of task scheduling during the execution of long-tail task slicing; Fig.13 is a structural block diagram of a task scheduling device according to an embodiment of the present application; Fig.14 is a structural block diagram of another task scheduling device according to an embodiment of the present application; Fig.15 is a structural block diagram of another task scheduling device according to an embodiment of the present application; Fig.16It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0018] Background of this application See also Figure 1 , Figure 1 The schematic diagram of multiple batch processing tasks with associations in the index construction scenario. The index construction process includes the following three tasks that are executed in order: light feature merging, data selection, and heavy feature merging. Among them, the light feature merging task specifically refers to: for the initial document table containing different original dimension feature values ​​of each page, for each page in the table, multiple preset light features (new features with less impact on index construction) are newly added to obtain a light feature table, which contains multiple groups of data, and one group of data corresponds to one page; the data selection task refers to: according to the preset selection rules, data is selected from the multiple groups of data contained in the light feature table to obtain multiple groups of selected data; the heavy feature merging task refers to: adding features to the above multiple groups of data again, that is, for each group of data in the above multiple groups of data, multiple preset heavy features (new features with greater impact on index construction) are newly added to obtain a heavy feature table, so as to perform index construction based on the obtained heavy feature table.

[0019] From the above content, we can see that the above three tasks have the following dependency relationship: the execution of the data selection task depends on the result of the light feature merging task, and the execution of the heavy feature merging task depends on the result of the data selection task. In other words, for the first two tasks, the light feature merging task is the upstream task, and the data selection task is the downstream task; and for the last two tasks, the data selection task is the upstream task, and the heavy feature merging task is the downstream task.

[0020] To improve the efficiency of task execution, during the task execution process, the complete task can usually be divided into multiple task slices (also called subtasks or task fragments, etc.), so that each task slice can be executed in parallel. Figure 1In the embodiment, the light feature merging task is divided into 65536 task slices, which are respectively recorded as 0-0 task slice, 0-1 task slice, etc.; the data selection task is divided into 65536 task slices, which are respectively recorded as 1-0 task slice, 1-1 task slice, etc.; the heavy feature merging task is divided into 131072 task slices, which are respectively recorded as 2-0 task slice, 2-1 task slice, 2-2 task slice, 2-3 task slice, etc. (the number of task slices obtained by the specific division can be customized according to the actual situation and is not limited in the embodiments of the present application).

[0021] In the related task scheduling scheme, for multi-task batch processing scenarios with dependencies between each other, upstream tasks and downstream tasks are often executed in sequence. Figure 2 , Figure 2 Schematic diagram of batch task scheduling process in related technology. Figure 2 For Figure 1 The schematic diagram of multi-task scheduling and execution process in the index construction scenario is shown in Figure 2. Figure 1 Differently, assuming Figure 2 The three tasks of light feature merging, data selection and heavy feature merging are divided into three task slices. The task slices of light feature merging task are 0-0, 0-1 and 0-2 respectively; the task slices of data selection task are 1-0, 1-1 and 1-2 respectively; the task slices of heavy feature merging task are 2-0, 2-1 and 2-2 respectively. The specific scheduling and execution process is as follows: Start three task executors in parallel, and assign the 0-0 task slice, 0-1 task slice, and 0-2 task slice in the light feature merging task to the three task executors respectively. During the execution of the light feature merging task, after the 0-0 task slice and the 0-1 task slice are executed, since the 0-2 task slice has not been executed yet, the execution process of the downstream data selection task will not be started. Instead, the 0-2 task slice will be executed until it is completed, and then the 1-0 task slice, 1-1 task slice, and 1-2 task slice in the data selection task will be assigned to the three task executors respectively. Similarly, during the execution of the data selection task, when the 1-0 task slice and the 1-1 task slice are executed, since the 1-2 task slice has not been executed yet, the execution process of the downstream heavy feature merging task will not be started. Instead, the task slice 2-0, 2-1 and 2-2 in the heavy feature merging will be allocated to the above three task executors respectively after the 1-2 task slice is executed.

[0022] from Figure 2It can be seen that no matter which of the three tasks, light feature merging, data selection, and heavy feature merging, there is an idle waiting period before task slice 0 and task slice 1 are executed and task slice 2 is executed. During this idle waiting period, although there are unexecuted downstream task slices, the resources of the task executor are idle. Therefore, the above-mentioned batch processing method of sequential execution not only reduces resource utilization, but also causes delays in task processing.

[0023] Detailed implementation process of the embodiment of this application Reference Figure 3 , Figure 3 The following is a flowchart of a task scheduling method according to an embodiment of the present application. The task scheduling method provided by the embodiment of the present application may include the following steps: Step 302, obtaining multiple tasks to be scheduled; the multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices.

[0024] Specifically, the multiple tasks in the embodiment of the present application may be multiple tasks with upstream and downstream dependencies between each other, and the specific number of tasks is not limited. The specific number may be 2, or any natural number greater than 2, such as 3, 4, etc. When the number of tasks to be scheduled is greater than 2, then in the order of execution, any two adjacent tasks have an upstream and downstream dependency relationship. Figure 1 In the index building scenario shown, there are three tasks to be scheduled. In terms of the execution order, for the first two tasks, the light feature merging task is the upstream task, and the data selection task is the downstream task; for the last two tasks, the data selection task is the upstream task, and the heavy feature merging task is the downstream task.

[0025] In order to improve the efficiency of task execution and facilitate parallel execution, the complete task can usually be divided into multiple task slices. As for upstream tasks, they can be divided into multiple upstream task slices. Similarly, as for downstream tasks, they can be divided into multiple downstream task slices. In the actual scheduling process, the number of task slices contained in each task can be pre-set by the task submitter, or it can be set or calculated by the task scheduler before task scheduling in combination with the total amount of data to be processed contained in the task and the computing power of the task scheduler. The embodiment of the present application does not limit the specific setting method of the task slice data contained in each task.

[0026] After determining the number of task slices contained in the task, multiple pieces of data to be processed can be classified into one task slice according to the amount of data contained in the task. Furthermore, when classifying the data to be processed, the amount of data processing contained in each task slice can be made as balanced as possible from the perspective of data amount.

[0027] Step 304: determine the task slice dependency relationship, where the task slice dependency relationship represents the dependency relationship between each upstream task slice and each downstream task slice.

[0028] Specifically, if there is a dependency relationship between an upstream task slice and a downstream task slice, it means that the execution of the downstream task slice depends on the execution result corresponding to the upstream task slice, and the downstream task slice can only start execution after the upstream task slice is completed, and the execution result of the upstream task slice corresponding to the upstream task slice will serve as the input condition for the execution of the downstream task slice.

[0029] The dependency between task slices can be determined based on the relationship between the number of upstream task slices and the number of downstream task slices. When the number of upstream task slices is equal to the number of downstream task slices, there is a one-to-one dependency between the downstream task slices and the upstream task slices, that is, the execution result of the upstream task slice corresponding to one upstream task slice will be used as the execution input condition of one downstream task slice; when the number of downstream task slices is N (N is a natural number greater than 1) times the number of upstream task slices, there is an N-to-one dependency between the downstream task slices and the upstream task slices, that is, the execution result of the upstream task slice corresponding to one upstream task slice will be used as the execution input condition of N downstream task slices.

[0030] like Figure 4 As shown, Figure 4 for Figure 1 The diagram of task slice dependency in the batch task scenario is shown. For light feature merging tasks and data selection tasks, since the number of task slices contained in each task is equal, both are 65536, there is a one-to-one dependency relationship between the downstream data selection task slice and the upstream light feature merging task slice, that is: the execution result of the upstream task slice corresponding to one upstream task slice will be used as the execution input condition of one downstream task slice; and for data selection tasks and heavy feature merging tasks, since the number of downstream task slices contained in the downstream heavy feature merging task is twice the number of upstream task slices, there is a 2-to-1 dependency relationship between the downstream task slice and the upstream task slice, that is: the execution result of the upstream task slice corresponding to one upstream data selection task slice will be used as the execution input condition of two downstream heavy feature merging task slices.

[0031] Step 306, in response to the completion of the execution of the current upstream task slice, a task executor is allocated to the target downstream task slice that depends on the completed upstream task slice, so that the task executor executes the target downstream task; the target downstream task slice is determined by the task slice dependency.

[0032] Specifically, the current upstream task slice is the upstream task slice that is currently being executed. In the embodiment of the present application, when it is determined that the current upstream task slice has been executed, it means that the execution result corresponding to the upstream task slice has been obtained at this time, and therefore, a task executor can be assigned to the target downstream task slice that depends on the completed upstream task slice, so as to use the above execution result as an input condition to execute the target downstream task slice.

[0033] See also Figure 5 , Figure 5 A schematic diagram comparing a batch task scheduling process according to an embodiment of the present application and a batch task scheduling process according to related technologies. Figure 5 Figure (a) shows the batch task scheduling process according to the related technology, and Figure 5 Figure (b) in the figure shows the skin dispatching process according to an embodiment of the present application. Specifically: according to Figure 5 As shown in Figure (b), the specific scheduling and execution process is as follows: Similar to Figure (a), three task executors are started in parallel, and the 0-0 task slice, 0-1 task slice, and 0-2 task slice in the light feature merging task are respectively assigned to the above three task executors. During the execution of the light feature merging task, when the 0-0 task slice and the 0-1 task slice are executed, there is no need to wait for the 0-2 task slice to be executed, and the 1-0 task slice that depends on the 0-0 task slice and the 1-1 task slice that depends on the 0-1 task slice in the downstream data selection task are started respectively; after that, when the 1-0 task slice is executed, there is no need to wait for the remaining task slices in the data selection task to be executed, and the 2-1 task slice that depends on the 1-0 task slice in the downstream heavy feature merging task can be started. For task slice 0, when task slice 1-1 is executed, there is no need to wait for the completion of other task slices in the data selection task, and the 2-1 task slice that depends on task slice 1-1 in the downstream heavy feature merging task can be started; similarly, when task slice 0-2 is executed, the 1-2 task slice that depends on task slice 0-2 in the downstream data selection task can be started, and when task slice 1-2 is executed, the 2-2 task slice that depends on task slice 1-2 in the downstream heavy feature merging task can be started.

[0034] contrast Figure 5It can be seen from Figures (a) and (b) in the figure that: through the task scheduling solution provided in the embodiment of the present application, during the task scheduling and execution process, the idle waiting period originally existing in Figure (a) can be eliminated, the resource utilization of the task executor can be improved, and the total time consumed by the task execution process can be reduced.

[0035] In summary, the task scheduling method provided in the embodiment of the present application, after obtaining multiple batch-processed upstream and downstream tasks with dependencies, decomposes the dependency of the downstream task on the upstream task at the overall level into the dependency of the downstream task slice on the individual upstream task slice through the dependency relationship between each task slice. According to the dependency relationship between the task slices, during the scheduling and execution of the task slices, there is no need to wait for the upstream task to be executed as a whole. Instead, after the individual upstream task slices in the upstream task are executed, the downstream task slice execution program with the dependency relationship with the above-mentioned individual upstream task slice can be started. Therefore, through the scheme of the embodiment of the present application, after receiving the batch-processed upstream and downstream tasks with dependencies, the upstream and downstream tasks originally processed in batch processing can be converted into task slices processed in stream processing mode, and the stream processing at the task slice level is realized while retaining the batch processing operation on the internal data of each task slice.

[0036] Through the above method, the task submitter can still set and submit the task logic in the manner of batch processing tasks without making any changes to the task logic. In the process of task scheduling and execution, the solution provided by the embodiment of the present application can automatically convert batch processing tasks into stream processing task slices. Therefore, pseudo-streaming processing of batch tasks is achieved, so that the batch task execution process has the advantages of stream processing: data continues to flow, improving the resource utilization of the task executor, improving task processing efficiency, shortening task execution time, and reducing task processing delays.

[0037] In addition, in the embodiment of the present application, once an upstream task slice is executed, the corresponding downstream task slice execution program can be started without waiting for other upstream task slices to be executed. Therefore, for the first slice execution result corresponding to the first upstream task slice that is executed, the use of the embodiment of the present application can effectively shorten the output time of the above-mentioned first slice execution result. This can effectively improve the troubleshooting efficiency for the scenario of checking the correctness of task logic based on task execution results.

[0038] The task scheduling method provided in the embodiment of the present application can be executed by any appropriate device with data processing capabilities, including but not limited to: PCs and servers.

[0039] Optionally, in some of the embodiments, in response to the completion of the execution of the current upstream task slice, the process of allocating a task executor to a target downstream task slice that depends on the completed upstream task slice may include: In response to the current upstream task slice being executed, determining whether there is an unexecuted upstream task slice; If there is no unexecuted upstream task slice, assign a task executor to the target downstream task slice that depends on the completed upstream task slice; If there are unexecuted upstream task slices, assign a task executor to the unexecuted upstream task slices, so that the unexecuted upstream task slices are used as the updated current upstream task slices, and the unexecuted upstream task slices are executed by the task executor; return in response to the completion of the execution of the current upstream task slice, and determine whether there are any unexecuted upstream task slices until all task slices are allocated.

[0040] Specifically, when the number of task processors is limited, and the number of task slices obtained by dividing the task is greater than the number of task processors that can perform task slice processing, all task slices cannot be executed in parallel. Instead, a part of the task slices are first executed in parallel by multiple task processors, and the remaining task slices are executed after waiting for the above-mentioned part of the task slices to be completed. Therefore, in view of the above situation, in an embodiment of the present application, after the current upstream task is executed, it can be determined whether there are unexecuted upstream task slices (that is, upstream task slices to be scheduled). If so, the unexecuted upstream task slices are executed first. After all upstream task slices are scheduled and executed, the execution process of the downstream task slices is started until all task slices are executed.

[0041] See also Figure 6 , Figure 6 FIG. 2 is another comparison diagram between the batch task scheduling process according to the embodiment of the present application and the batch task scheduling process according to the related art. Figure 6 The above embodiments of the present application are explained as follows: Figure 6 Also for Figure 1 The schematic diagram of multi-task scheduling and execution process in the index construction scenario is shown in Figure 2. Figure 1 Differently, assuming Figure 6The three tasks of light feature merging, data selection and heavy feature merging are divided into four task slices. The task slices of light feature merging task are 0-0, 0-1, 0-2 and 0-3 respectively; the task slices of data selection task are 1-0, 1-1, 1-2 and 1-3 respectively; the task slices of heavy feature merging task are 2-0, 2-1, 2-2 and 2-3 respectively. In addition, it is assumed that due to the limitation of computing power resources, the number of task executors that can run in parallel at the same time is 3.

[0042] Figure 6 Figure (a) shows the batch task scheduling process according to the related technology, and Figure 6 Figure (b) shows the task scheduling process according to an embodiment of the present application.

[0043] Specifically, according to Figure 6 As shown in Figure (a), the specific task scheduling and execution process of the related technology is as follows: start three task executors in parallel, and assign the 0-0, 0-1 and 0-2 task slices to the above three task executors respectively; when the 0-0 and 0-1 task slices are executed at the same time, any task executor that executes the above task slices can be assigned to execute the 0-3 task slice again ( Figure 6 In the example, the task executor that executes the 0-1 task slice is used to execute 0-3. At this time, the 0-2 task slice is also executed synchronously. When the 0-2 task is executed, since the 0-3 task slice has not been executed yet, the task executor that originally executed 0-2 and 0-0 is in an idle waiting period until the 0-3 task slice is executed, and then the 1-0, 1-1 and 1-2 task slices are started in parallel. After that, the 1-1 task slice is executed first, and at this time, the task executor that is assigned to execute the 1-1 task slice executes the 1-3 task slice again. At this time, the 1-0 and 1-2 task slices are also executed synchronously. When 1-0 or 1-2 is executed, since the 1-3 task slice has not been executed yet, the task executor that originally executed 1-0 and 1-2 is in an idle waiting period until the 1-3 task slice is executed, and then the 2-0, 2-1 and 2-2 task slices are started in parallel. Afterwards, task slice 2-0 is executed first, and then the task executor that executes task slice 2-0 is assigned to execute task slice 2-3 until task slice 2-3 is executed.

[0044] according to Figure 6 As shown in Figure (b), the specific scheduling and execution process is as follows: Similar to Figure (a), three task executors are started in parallel, and task slices 0-0, 0-1, and 0-2 are respectively assigned to the above three task executors; when task slices 0-0 and 0-1 are executed at the same time, first determine whether there are unexecuted (to be scheduled) upstream tasks. Since there is an upstream task slice 0-3, the task executor that executes 0-1 can be assigned to execute task slice 0-3 again; at this time, there are no unexecuted (to be scheduled) upstream tasks, so the downstream task slices that depend on the completed 0-0 and 0-1 can be determined. Specifically: the downstream task slice that depends on 0-0 is 1-0, and the downstream task slice that depends on the completed 0-1 is 1-1. Therefore, the task executor that has completed 0-0 can be assigned to start executing 1-0. When 0-2 is executed, there are no unexecuted upstream task slices. Therefore, the task executor that has completed 0-2 can be assigned to start executing the downstream task slice 1-2 that depends on 0-2. And so on, we can get Figure 6 The task processing flow chart shown in Figure (b) is shown in Figure 1.

[0045] contrast Figure 6 It can be seen from Figures (a) and (b) in the figure that: through the task scheduling solution provided in the embodiment of the present application, during the task scheduling and execution process, the idle waiting period originally existing in Figure (a) can be eliminated, the resource utilization of the task executor can be improved, and the total time consumed by the task execution process can be reduced.

[0046] Furthermore, in the above-mentioned embodiment of the present application, when an upstream task slice is executed to completion, it is determined whether there are any unexecuted upstream task slices. If so, the upstream task slices are executed first. Afterwards, when there are no unexecuted upstream task slices, the target downstream task slices that depend on the completed upstream task slices are assigned for execution. Through the above-mentioned process, the corresponding execution results can be obtained as early as possible for the remaining upstream task slices that failed to be executed in the previous parallel execution process, and then the execution process of the downstream task slices that depend on the above-mentioned remaining upstream task slices can be started as early as possible, which helps to improve the efficiency of obtaining the final slice execution results corresponding to the task slice.

[0047] Optionally, in some of the embodiments, the task scheduling method may further include: During the execution of the current task slice, the task execution time of the first task executor on the current task slice is recorded; the first task executor is a task executor configured to execute the current task slice; the current task slice represents the upstream task slice currently being executed, or the downstream task slice being executed; If the execution time exceeds the preset time, the second task executor is allocated to the current task slice, so that the current task slice is executed in parallel by the first task executor and the second task executor; In response to obtaining the task execution result of the current task slice, the execution process for the current task slice is terminated.

[0048] Specifically, during the scheduling and execution of task slices, multiple task slices may be scheduled and executed at the same time. However, since different task slices may involve different amounts of data and complexities, and the computing resources or operating states of different task executors may also differ, the execution process of some task slices may take too long (for ease of description, in the embodiment of the present application, task slices whose execution time exceeds the preset time are referred to as long-tail task slices). In this case, the completion time of the overall task is affected by the above-mentioned task slices with too long execution time, which may cause an increase in waiting time. This situation is especially obvious when processing large-scale data, which in turn affects the overall system performance and resource utilization.

[0049] In order to solve the above-mentioned problem, in the above-mentioned embodiment of the present application, a solution of parallel operation of long-tail task slices is adopted: when a certain task executor (for the convenience of description, referred to as the first task executor in the embodiment of the present application) executes a certain task slice for too long (such as exceeding the preset time), while retaining the first task executor to execute the task slice, the task slice is executed in parallel by the second task executor; when either the first task executor or the second task executor obtains the task execution result of the above-mentioned long-tail task slice, the execution process of the long-tail task slice can be ended. For example: if the second task executor obtains the task execution result of the long-tail task slice first, the execution process of the long-tail task slice by the first task executor can be ended; conversely, if the first task executor obtains the task execution result of the long-tail task slice first, the execution process of the long-tail task slice by the second task executor can be ended.

[0050] When the computing resources of the first task executor are insufficient, or the first task executor runs abnormally, resulting in a long-tail problem in the execution time of the task slices, the parallel operation solution provided in the embodiment of the present application can regain the execution results of the long-tail task slices in a shorter time, thereby improving the overall efficiency and stability of task scheduling and execution.

[0051] In addition, in the embodiments of the present application, there is no limitation on the specific value of the preset duration and the specific determination method, and the setting can be customized according to the actual situation. For example: the specific value of the preset duration can be set based on experience, or the data volume and replication degree of the historical task slices can be combined, and the execution time of the historical task slices can be analyzed and counted, so as to infer the average execution time of each task slice, and then the above-mentioned preset duration can be set based on the average execution time. Exemplarily, if the execution time of the task slice exceeds 1.5 times of the above-mentioned average execution time, it can be determined that the above-mentioned long tail problem occurs.

[0052] See also Figure 7 , Figure 7 This is a schematic diagram of an exemplary system for the task scheduling method of the embodiment of the present application. For ease of understanding, first, Figure 7 The application scenario of the task scheduling method provided in the embodiment of the present application is explained.

[0053] like Figure 7 As shown, the system 700 may include a task scheduler 702, a communication network 704 and / or one or more task executors 706. Figure 7 The example in the example shows multiple task executors.

[0054] The task scheduler 702 may be any suitable device for performing task scheduling, including but not limited to a server cluster, a computing cloud server cluster, etc. In some embodiments, the task scheduler 702 may perform any suitable function. For example, in some embodiments, the task scheduler 702 may receive a task allocation request sent by the task executor 706, and allocate a suitable task slice to the task executor 706 for execution by the task executor 706.

[0055] In some embodiments, the communication network 704 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 704 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN) and / or any other suitable communication network. The task executor 706 can be connected to the communication network 704 via one or more communication links (e.g., communication link 712), and the communication network 704 can be linked to the task scheduler 702 via one or more communication links (e.g., communication link 714). The communication link can be any communication link suitable for transmitting data between the task executor 706 and the task scheduler 702, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link or any suitable combination of such links.

[0056] Task executor 706 may include any one or more devices suitable for performing task processing. In some embodiments, task executor 706 may include any suitable type of device. For example, in some embodiments, task executor 706 may include a laptop computer, a desktop computer and / or any other suitable type of user equipment.

[0057] Reference Figure 8 , Figure 8 FIG. 1 is a flowchart of another task scheduling method according to an embodiment of the present application. The task scheduling method provided in the embodiment of the present application can be Figure 7 The task scheduler 702 in the system shown in the figure executes. Specifically, the task scheduling method provided in this embodiment includes the following steps: Step 802, in the scheduling process for multiple tasks, determine the task slice dependency; the multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the task slice dependency represents the dependency between each upstream task slice and each downstream task slice.

[0058] Step 804: Receive a task allocation request sent by the target task executor.

[0059] Specifically, the target task executor in the embodiment of the present application can be Figure 7Any task executor in the system shown. When the target task executor is in an idle state, or, although it is in a task slice execution state, its own computing resources can support the target task executor to start multiple task slice execution processes in parallel, the task processor can send a task allocation request to the task scheduler.

[0060] Step 806, allocating the unexecuted target downstream task slice to the target task executor so as to execute the target downstream task slice through the target task executor; the target downstream task slice is a downstream task slice determined according to the task slice dependency and dependent on the completed upstream task slice.

[0061] In the above-mentioned embodiment of the present application, after obtaining a plurality of batch-processed upstream and downstream tasks with dependencies, the dependency of the downstream task on the upstream task at the overall level is decomposed and converted into the dependency of the downstream task slice on the individual upstream task slice through the dependency relationship between each task slice. According to the dependency relationship between the task slices, during the scheduling and execution of the task slices, there is no need to wait for the upstream task to be executed as a whole. Instead, after the individual upstream task slices in the upstream task are executed, the execution program of the downstream task slice with the dependency relationship with the above-mentioned individual upstream task slice can be started. Therefore, through the scheme of the embodiment of the present application, after receiving the batch-processed upstream and downstream tasks with dependencies, the upstream and downstream tasks originally processed in batch processing can be converted into task slices processed in stream processing mode, and the stream processing at the task slice level is realized while retaining the batch processing operation on the internal data of each task slice.

[0062] Through the above method, the task submitter can still set and submit the task logic in the manner of batch processing tasks without making any changes to the task logic. In the process of task scheduling and execution, the solution provided by the embodiment of the present application can automatically convert batch processing tasks into stream processing task slices. Therefore, pseudo-streaming processing of batch tasks is achieved, so that the batch task execution process has the advantages of stream processing: data continues to flow, improving the resource utilization of the task executor, improving task processing efficiency, shortening task execution time, and reducing task processing delays.

[0063] In addition, in the embodiment of the present application, once an upstream task slice is executed, the corresponding downstream task slice execution program can be started without waiting for other upstream task slices to be executed. Therefore, for the first slice execution result corresponding to the first upstream task slice that is executed, the use of the embodiment of the present application can effectively shorten the output time of the above-mentioned first slice execution result. This can effectively improve the troubleshooting efficiency for the scenario of checking the correctness of task logic based on task execution results.

[0064] Optionally, in some embodiments, allocating the unexecuted target downstream task slice to the target task executor so as to execute the target downstream task slice through the target task executor includes: Determine whether there are any unexecuted upstream task slices; if there are no unexecuted upstream task slices, assign the unexecuted target downstream task slices to the target task executor to execute the target downstream task slices through the target task executor; if there are unexecuted upstream task slices, assign the unexecuted upstream task slices to the target task executor to execute the unexecuted upstream task slices through the target task executor.

[0065] Specifically, in the above embodiment of the present application, when an upstream task slice is executed, it is determined whether there are any unexecuted upstream task slices. If so, the upstream task slices are executed first. After that, when there are no unexecuted upstream task slices, the target downstream task slices that depend on the completed upstream task slices are assigned for execution. Through the above process, the corresponding execution results can be obtained as early as possible for the remaining upstream task slices that failed to be executed in the previous parallel execution process, and then the execution process of the downstream task slices that depend on the above-mentioned remaining upstream task slices can be started as early as possible, which helps to improve the efficiency of obtaining the final slice execution results corresponding to the task slice.

[0066] Optionally, in some embodiments, allocating the unexecuted target downstream task slice to the target task executor so as to execute the target downstream task slice through the target task executor includes: Determine whether there is a long-tail task slice whose execution time exceeds the preset time; if there is no long-tail task slice, assign the unexecuted target downstream task slice to the target task executor to execute the target downstream task slice through the target task executor; if there is a long-tail task slice, assign the long-tail task slice to the target task executor to execute the long-tail task slice through the target task executor; after obtaining the task execution result of the long-tail task slice, end the execution process for the long-tail task slice.

[0067] Specifically, in response to the possible existence of long-tail task slices with excessively long execution time, the above-mentioned embodiments of the present application adopt a solution of parallel operation of the long-tail task slices: when the task scheduler determines that there is a long-tail task slice, while retaining the execution of the task slice, the long-tail task slice will be executed in parallel by a new task executor (the above-mentioned target task executor) that sends a task allocation request; when any task executor obtains the task execution result of the above-mentioned long-tail task slice, the execution process of the long-tail task slice can be ended.

[0068] When the long-tail problem of the task slice execution time occurs due to insufficient computing resources of the task executor or abnormal operating status, the parallel operation solution provided in the embodiment of the present application can regain the execution results of the long-tail task slices in a shorter time, thereby improving the overall efficiency and stability of task scheduling and execution.

[0069] In addition, in the embodiments of the present application, there is no limitation on the specific value of the preset duration and the specific determination method, and the setting can be customized according to the actual situation. For example: the specific value of the preset duration can be set based on experience, or the data volume and replication degree of the historical task slices can be combined, and the execution time of the historical task slices can be analyzed and counted, so as to infer the average execution time of each task slice, and then the above-mentioned preset duration can be set based on the average execution time. Exemplarily, if the execution time of the task slice exceeds 1.5 times of the above-mentioned average execution time, it can be determined that the above-mentioned long tail problem occurs.

[0070] Optionally, in some of the embodiments, receiving a task allocation request sent by a target task executor includes: Receive the heartbeat signal sent periodically by the target task executor, the heartbeat signal includes: current task slice identification information and task request identification information; the task request identification information indicates whether the target task executor requests a new task slice; if it is determined according to the task request identification information that the target task executor requests a new task slice, it is determined that the task allocation request sent by the target task executor is received.

[0071] Correspondingly, allocating the long-tail task slice to the target task executor includes: judging whether the task slice being executed by the target task executor is a long-tail task slice according to the current task slice identification information; if not, allocating the long-tail task slice to the target task executor.

[0072] Specifically, Figure 7As described above, the task scheduling system usually includes a large number of task schedulers, and during the task scheduling and execution process, the specific execution status of each task scheduler is not the same. In order to facilitate the task scheduler to timely understand the task execution status of each task executor, as well as the progress of the entire task scheduling work, in the above embodiment of the present application, the task executor regularly sends a heartbeat signal containing the current task slice identification information and the task request identification information to the task scheduler, so as to report the task slice related information (such as the identification information of the currently executing task slice, the task slice execution details information, etc.) to the task scheduler through the signal, and inform the task scheduler whether it can receive and execute a new task slice at the current moment. After receiving the above heartbeat signal, the task scheduler will update the execution status of the task slice according to the current task slice identification information contained therein. In addition, when the task request identification information contained in the heartbeat signal indicates that the target task executor requests a new task slice, a new task slice to be executed is allocated to the target task executor according to the execution status of the above task slice.

[0073] Moreover, after the task scheduler assigns a task slice to a task executor, if the task scheduler fails to receive the heartbeat signal sent by the task executor on time, it indicates that the operation of the task executor is abnormal. At this time, the task scheduler can reallocate the task slice originally assigned to the task processor to other task executors. In this way, the task scheduler can promptly discover the abnormal situation during the execution of the task slice, and then promptly adopt the corresponding remedial measures to avoid the situation where the task slice fails to be executed smoothly.

[0074] In addition, for task executors that execute long-tail task shards in parallel, when the task scheduler receives the shard execution result fed back by one of the task executors through a heartbeat signal, it can return an execution stop notification when the other task executors that execute the long-tail task shard in parallel send a heartbeat signal to stop the continued execution of the long-tail task shard.

[0075] In summary, in the above embodiments of the present application, the task scheduler can timely understand the task execution status of each task executor and the progress of the entire task scheduling work, so as to better schedule the subsequent task slices. In addition, when the task executor that is executing the task slice runs abnormally, the task scheduler can promptly discover and adopt corresponding remedial measures to avoid the situation where the task slice fails to be executed smoothly.

[0076] See also Fig. 9 , Fig. 9 For Figure 8 The following is a schematic diagram of the task scheduling process interaction process corresponding to the embodiment shown. Fig. 9Explain the above task scheduling process: During the task execution process, the task executor periodically sends a heartbeat signal to the task scheduler. Fig. 9 As shown, for one heartbeat cycle, the heartbeat signal sent may include the current task slice identification information and the task request identification information. Assuming that the task executor is currently executing a task slice, it can report the identification of the currently executing task slice and other related information to the task scheduler. At the same time, if computing resources allow, it can also report to the task scheduler the task request identification information representing the expectation to apply for a new task slice.

[0077] After receiving the above heartbeat signal, the task scheduler enters the judgment phase: whether there is a task slice to be executed, and if there is a task slice to be executed, whether the task slice to be executed is the same as the task slice being executed by the task executor; if there is a task slice to be executed, and whether the task slice to be executed is different from the task slice being executed by the task executor, the task slice to be executed is assigned to the task executor; otherwise, if there is no task slice to be executed, a notification that there is no new task slice to be executed is returned. After that, the task scheduler can update the heartbeat signal record according to this interaction process.

[0078] In addition, when a certain task executor (referred to as the first task executor here for ease of description) is assigned to execute long-tail tasks in parallel, the task scheduler can also return a notification of whether to continue execution to the first task executor based on whether it has received a notification of completion of long-tail task execution reported by other task executors. Specifically: when a notification of completion of long-tail task execution reported by other task executors is received, a notification of stopping execution can be returned to the first task executor. Conversely, if a notification of completion of long-tail task execution reported by other task executors is not received, a notification of continuing execution can be returned to the first task executor.

[0079] See also Fig.10 , Fig.10 For execution Figure 8 The functional architecture diagram of the task scheduler of the embodiment shown is shown. Fig.10 The task scheduler of the embodiment of the present application can manage multiple batch tasks with dependencies, and has the capabilities of "pseudo-streaming of batch tasks", "long-tail parallel operation" and "failure retry". Specifically, in the task scheduling and management process, the upstream and downstream tasks originally processed by batch processing can be converted into task slices processed by stream processing. While retaining the batch processing operation on the internal data of each task slice, the streaming processing at the task slice level is realized, that is: Fig.10The "batch task pseudo-streaming" capability shown in the figure; for long-tail task slices whose execution time exceeds the preset time, a solution of parallel operation of long-tail task slices is adopted, that is: Fig.10 The "long-tail parallel operation" capability shown in the figure; in addition, after the task scheduler assigns a task slice to a task executor, if the task scheduler fails to receive the heartbeat signal sent by the task executor on time, it indicates that the task executor is running abnormally. At this time, the task scheduler can reallocate the task slice originally assigned to the task processor to other task executors, that is: Fig.10 The "failure retry" capability is shown.

[0080] Reference Fig.11 , Fig.11 FIG. 1 is a flowchart of another task scheduling method according to an embodiment of the present application. The task scheduling method provided in the embodiment of the present application can be Figure 7 The task executor 706 in the system shown in the figure executes. Specifically, the task scheduling method provided in this embodiment includes the following steps: Step 1102 : Send a task allocation request to the task scheduler, so that the task scheduler allocates the unexecuted target downstream task slices to the target task executor during the scheduling process of multiple tasks.

[0081] Step 1104, obtain the target downstream task slice assigned by the task scheduler, and execute the target downstream task slice.

[0082] The multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the target downstream task slice is a downstream task slice that depends on the completed upstream task slice.

[0083] In the above-mentioned embodiment of the present application, after obtaining a plurality of batch-processed upstream and downstream tasks with dependencies, the dependency of the downstream task on the upstream task at the overall level is decomposed and converted into the dependency of the downstream task slice on the individual upstream task slice through the dependency relationship between each task slice. According to the dependency relationship between the task slices, during the scheduling and execution of the task slices, there is no need to wait for the upstream task to be executed as a whole. Instead, after the individual upstream task slices in the upstream task are executed, the execution program of the downstream task slice with the dependency relationship with the above-mentioned individual upstream task slice can be started. Therefore, through the scheme of the embodiment of the present application, after receiving the batch-processed upstream and downstream tasks with dependencies, the upstream and downstream tasks originally processed in batch processing can be converted into task slices processed in stream processing mode, and the stream processing at the task slice level is realized while retaining the batch processing operation on the internal data of each task slice.

[0084] Through the above method, the task submitter can still set and submit the task logic in the manner of batch processing tasks without making any changes to the task logic. In the process of task scheduling and execution, the solution provided by the embodiment of the present application can automatically convert batch processing tasks into stream processing task slices. Therefore, pseudo-streaming processing of batch tasks is achieved, so that the batch task execution process has the advantages of stream processing: data continues to flow, improving the resource utilization of the task executor, improving task processing efficiency, shortening task execution time, and reducing task processing delays.

[0085] In addition, in the embodiment of the present application, once an upstream task slice is executed, the corresponding downstream task slice execution program can be started without waiting for other upstream task slices to be executed. Therefore, for the first slice execution result corresponding to the first upstream task slice that is executed, the use of the embodiment of the present application can effectively shorten the output time of the above-mentioned first slice execution result. This can effectively improve the troubleshooting efficiency for the scenario of checking the correctness of task logic based on task execution results.

[0086] Optionally, in some of the embodiments, the target downstream task slice is a long-tail task slice, and the method further includes: After the target downstream task shard is executed, a write request to the target directory is sent to the task scheduler; In response to receiving the write permission notification returned by the task scheduler, the task execution result of the target downstream task slice is written into the target directory.

[0087] Specifically, in an embodiment of the present application, for different task executors that execute long-tail task slices in parallel, the data generated during the execution process can be stored in their respective corresponding private domain directories, rather than in the target directory that the long-tail task slices ultimately correspond to. Afterwards, when a task executor that executes the long-tail task slices in parallel completes execution, it requests the task scheduler to write the data in the private domain directory to the above-mentioned target directory. The specific request process is: a target directory write request is sent to the task scheduler. If the task scheduler determines that there are currently no other task executors that have completed the above-mentioned long-tail task slices, a write permission notification is returned. At this time, the task executor can write the data in the private domain directory to the target directory.

[0088] Through the above-mentioned application mechanism (also called application lock mechanism), it can be ensured that among multiple task executors that execute long-tail task slices in parallel, the first task executor that completes the execution task can smoothly store the execution result to the target directory, that is, it can avoid parallel write conflicts while improving the execution efficiency of long-tail task slices.

[0089] Optionally, in some of the embodiments, after sending the target directory write request to the task scheduler, the method further includes: In response to receiving the write permission occupied notification returned by the task scheduler, a target directory write request is periodically sent to the task scheduler at a preset time interval.

[0090] Specifically, when the task scheduler determines that there are other task executors that have completed the above-mentioned long-tail task slices, the write permission has been occupied by the above-mentioned other task executors. Therefore, the task scheduler can return a write permission occupation notification to the current task executor. At this time, the current task executor cannot write data to the target directory.

[0091] Furthermore, considering that write errors may also occur when performing data writing operations to the target directory. Therefore, in the above-mentioned embodiment of the present application, before it is determined that the above-mentioned other task executors have written successfully, the current task executor can continue to send write requests to the task scheduler on a regular basis. After the task scheduler determines that the above-mentioned other task executors have written successfully, when the task scheduler receives the heartbeat signal sent by the current task executor again, as feedback, the task scheduler returns a notification to stop task execution to the current task executor, and the current task executor will stop sending write requests to the task scheduler for the target directory.

[0092] Through the above process, the reliability of obtaining the execution results of the long-tail task shards can be effectively improved, avoiding the problem of being unable to obtain the final execution results of the long-tail task shards due to data writing errors.

[0093] Optionally, in some of the embodiments, sending a task allocation request to a task scheduler includes: A heartbeat signal is periodically sent to the task scheduler, wherein the heartbeat signal includes: task request identification information; the task request identification information indicates whether the task executor requests a new task slice; if it is determined based on the task request identification information that the task executor requests a new task slice, then a task allocation request is sent to the task scheduler; the method also includes: if the target downstream task slice fails to execute, then the failure reason is recorded and the heartbeat signal is stopped from being sent to the task scheduler.

[0094] Specifically, through the heartbeat mechanism in the above embodiment of the present application, the task scheduler can timely understand the task execution status of each task executor, as well as the progress of the entire task scheduling work, so as to better schedule the subsequent task slices. In addition, when the task executor that is executing the task slice runs abnormally, the task scheduler can promptly discover and adopt corresponding remedial measures to avoid the situation where the task slice fails to be executed smoothly.

[0095] See also Fig.12 , Fig.12The following is a diagram of the interactive process of task scheduling during the execution of long-tail task slicing. Fig.12 The following is an explanation of the above long-tail task slicing execution process: For the task executor that executes the long-tail task slice, during the task execution process, it first enters the first judgment stage: judge whether the task is executed successfully. If the execution fails, the failure reason is recorded and the heartbeat signal is stopped from being sent to the task scheduler; if the task is executed successfully, a target directory preemption lock request (that is, a target directory write request) is sent to the task scheduler, and the second judgment stage is entered: if it is judged that there is no other task executor that has completed the long-tail task slice at this time, the preemption is successful, and the task scheduler will return a preemption success notification (that is, a write permission notification). At this time, the task executor can write the data in the private domain directory to the target directory. At this point, the execution process of the long-tail task slice ends, and the task executor returns a task completion notification to the task scheduler, and the task scheduler records the relevant status data of the long-tail task slice, counts the execution time, deletes the process data, etc.; if it is judged that there are other task executors that have completed the long-tail task slice at this time, the preemption fails, and the task scheduler will return a preemption failure notification (that is, a write permission occupation notification). After that, the task executor periodically sends the target directory preemption lock request to the task scheduler again at a preset time interval.

[0096] Fig.13 1 is a structural block diagram of a task scheduling device according to an embodiment of the present application. The task scheduling device provided by the embodiment of the present application includes: The task acquisition module 1302 is used to acquire multiple tasks to be scheduled; the multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; A dependency determination module 1304 is used to determine the task slice dependency, where the task slice dependency represents the dependency between each upstream task slice and each downstream task slice; The scheduling module 1306 is used to allocate a task executor to a target downstream task slice that depends on the completed upstream task slice in response to the completion of the execution of the current upstream task slice, so that the task executor executes the target downstream task; the target downstream task slice is determined by the task slice dependency.

[0097] Optionally, in some embodiments, the scheduling module 1306, in response to the completion of the execution of the current upstream task slice, allocates a task executor to the target downstream task slice that depends on the completed upstream task slice, is specifically configured to: In response to the current upstream task slice being executed, determining whether there is an unexecuted upstream task slice; If there is no unexecuted upstream task slice, assign a task executor to the target downstream task slice that depends on the completed upstream task slice; If there are unexecuted upstream task slices, assign a task executor to the unexecuted upstream task slices, so that the unexecuted upstream task slices are used as the updated current upstream task slices, and the unexecuted upstream task slices are executed by the task executor; return in response to the completion of the execution of the current upstream task slice, and determine whether there are any unexecuted upstream task slices until all task slices are allocated.

[0098] Optionally, in some of the embodiments, the task scheduling device further includes: An execution duration recording module is used to record the task execution duration of the first task executor on the current task slice during the execution of the current task slice; the first task executor is a task executor configured to execute the current task slice; the current task slice represents the upstream task slice currently being executed, or the downstream task slice being executed; A parallel allocation module, used for allocating a second task executor to the current task slice if the execution time exceeds a preset time, so as to execute the current task slice in parallel through the first task executor and the second task executor; The execution stop module is used to terminate the execution process for the current task slice in response to obtaining the task execution result of the current task slice.

[0099] The task scheduling device of this embodiment is used to implement the corresponding task scheduling method in the aforementioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. In addition, the functional implementation of each module in the task scheduling device of this embodiment can refer to the description of the corresponding part in the aforementioned method embodiment, which will not be repeated here.

[0100] Fig.14 1 is a structural block diagram of another task scheduling device according to an embodiment of the present application. The task scheduling device provided in the embodiment of the present application is located in the task executor, and the task scheduling module includes: The relationship determination module 1402 is used to determine the task slice dependency relationship in the scheduling process for multiple tasks; the multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the task slice dependency relationship represents the dependency relationship between each upstream task slice and each downstream task slice; The request receiving module 1404 is used to receive the task allocation request sent by the target task executor; The task allocation module 1406 is used to allocate the unexecuted target downstream task slices to the target task executor so as to execute the target downstream task slices through the target task executor; the target downstream task slices are downstream task slices determined according to the task slice dependency and dependent on the completed upstream task slices.

[0101] Optionally, in some embodiments, the task allocation module 1406 is specifically configured to: Determine whether there are any unexecuted upstream task slices; if there are no unexecuted upstream task slices, assign the unexecuted target downstream task slices to the target task executor to execute the target downstream task slices through the target task executor; if there are unexecuted upstream task slices, assign the unexecuted upstream task slices to the target task executor to execute the unexecuted upstream task slices through the target task executor.

[0102] Optionally, in some embodiments, the task allocation module 1406 is specifically configured to: Determine whether there are long-tail task slices whose execution time exceeds the preset time; If there is no long-tail task slice, the unexecuted target downstream task slice is assigned to the target task executor, so that the target downstream task slice is executed by the target task executor; If there are long-tail task slices, the long-tail task slices are assigned to the target task executor so that the long-tail task slices are executed by the target task executor; After obtaining the task execution result of the long-tail task slice, the execution process for the long-tail task slice is ended.

[0103] Optionally, in some embodiments, the request receiving module 1404 is specifically used to: receive a heartbeat signal periodically sent by the target task executor, the heartbeat signal including: current task slice identification information and task request identification information; the task request identification information indicates whether the target task executor requests a new task slice; if it is determined according to the task request identification information that the target task executor requests a new task slice, it is determined that a task allocation request sent by the target task executor is received; Correspondingly, when executing the step of allocating long-tail task slices to the target task executor, the task allocation module 1406 is specifically used to: determine whether the task slice being executed by the target task executor is a long-tail task slice based on the current task slice identification information; if not, allocate the long-tail task slice to the target task executor.

[0104] The task scheduling device of this embodiment is used to implement the corresponding task scheduling method in the aforementioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. In addition, the functional implementation of each module in the task scheduling device of this embodiment can refer to the description of the corresponding part in the aforementioned method embodiment, which will not be repeated here.

[0105] Fig.15 The task scheduling device provided in the embodiment of the present application is located in the task executor, and the task scheduling module includes: The request sending module 1502 is used to send a task allocation request to the task scheduler, so that the task scheduler allocates the unexecuted target downstream task slices to the target task executor during the scheduling process of multiple tasks; The task slice acquisition module 1504 is used to acquire the target downstream task slice assigned by the task scheduler and execute the target downstream task slice; The multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the target downstream task slice is a downstream task slice that depends on the completed upstream task slice.

[0106] Optionally, in some embodiments, the target downstream task slice is a long-tail task slice, and the task scheduling device further includes: The result writing module is used to send a target directory write request to the task scheduler after the target downstream task slice is executed; in response to receiving a write permission notification returned by the task scheduler, write the task execution result of the target downstream task slice into the target directory.

[0107] Optionally, in some embodiments, the result writing module is further used to: after sending a target directory write request to the task scheduler, in response to receiving a write permission occupied notification returned by the task scheduler, periodically send a target directory write request to the task scheduler at preset time intervals.

[0108] Optionally, in some embodiments, the request sending module 1502, when executing the step of sending the task allocation request to the task scheduler, is specifically used to: regularly send a heartbeat signal to the task scheduler, the heartbeat signal including: task request identification information; the task request identification information indicates whether the task executor requests a new task slice; if it is determined according to the task request identification information that the task executor requests a new task slice, then determine to send the task allocation request to the task scheduler; Optionally, the task scheduling device further includes: The failure recording module is used to record the failure reason and trigger the request sending module 1502 to stop sending heartbeat signals to the task scheduler if the target downstream task slice fails to execute.

[0109] The task scheduling device of this embodiment is used to implement the corresponding task scheduling method in the aforementioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. In addition, the functional implementation of each module in the task scheduling device of this embodiment can refer to the description of the corresponding part in the aforementioned method embodiment, which will not be repeated here.

[0110] Reference Fig.16 , shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0111] like Fig.16 As shown, the electronic device may include: a processor (processor) 1602 , a communication interface (Communications Interface) 1604 , a memory (memory) 1606 , and a communication bus 1608 .

[0112] in: The processor 1602 , the communication interface 1604 , and the memory 1606 communicate with each other via a communication bus 1608 .

[0113] The communication interface 1604 is used to communicate with other electronic devices or servers.

[0114] The processor 1602 is used to execute the program 1610, and specifically can execute the relevant steps in the above method embodiment.

[0115] Specifically, the program 1610 may include program codes, which include computer operation instructions.

[0116] The processor 1602 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0117] The memory 1606 is used to store the program 1610. The memory 1606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0118] The program 1610 may include multiple computer instructions. Specifically, the program 1610 may enable the processor 1602 to execute operations corresponding to the method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0119] The specific implementation of each step in program 1610 can refer to the corresponding description of the corresponding steps and units in the above method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0120] The present application also provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any of the above-mentioned multiple method embodiments is implemented. The computer storage medium includes, but is not limited to: a compact disc read-only memory (CD-ROM), a random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk, etc.

[0121] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0122] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0123] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0124] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., a random access memory (RAM), a read-only memory (ROM), a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0125] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0126] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A task scheduling method, comprising: Get multiple tasks to be scheduled; The multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; Determine a task slice dependency relationship, wherein the task slice dependency relationship represents a dependency relationship between each upstream task slice and each downstream task slice; In response to the completion of the execution of the current upstream task slice, a task executor is allocated to a target downstream task slice that depends on the completed upstream task slice, so that the target downstream task is executed by the task executor; the target downstream task slice is determined by the task slice dependency.

2. The method according to claim 1, wherein: In response to the current upstream task slice being executed and completed, allocating a task executor to a target downstream task slice that depends on the completed upstream task slice includes: In response to the current upstream task slice being executed, determining whether there is an unexecuted upstream task slice; If there are no unexecuted upstream task slices, assign a task executor to the target downstream task slice that depends on the completed upstream task slice.

3. The method according to claim 2, wherein: After determining whether there are unexecuted upstream task slices, the method further includes: If there are any unexecuted upstream task slices, a task executor is assigned to the unexecuted upstream task slice, so that the unexecuted upstream task slice is used as the updated current upstream task slice, and the unexecuted upstream task slice is executed by the task executor; and the step of judging whether there are any unexecuted upstream task slices in response to the completion of the execution of the current upstream task slice is returned until all task slices are allocated.

4. The method according to any one of claims 1 to 3, wherein: The method further comprises: During the execution of the current task slice, the task execution time of the first task executor on the current task slice is recorded; the first task executor is a task executor configured to execute the current task slice; the current task slice represents the upstream task slice currently being executed, or the downstream task slice being executed; If the execution time exceeds the preset time, a second task executor is allocated to the current task slice, so that the current task slice is executed in parallel by the first task executor and the second task executor; In response to obtaining the task execution result of the current task slice, the execution process for the current task slice is terminated.

5. A task scheduling method, comprising: In the process of scheduling multiple tasks, determine the task slice dependencies; The multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the task slice dependency relationship represents the dependency relationship between each upstream task slice and each downstream task slice; Receive a task assignment request sent by a target task executor; Allocate the unexecuted target downstream task slice to the target task executor so as to execute the target downstream task slice through the target task executor; the target downstream task slice is a downstream task slice determined according to the task slice dependency and depends on the completed upstream task slice.

6. The method according to claim 5, wherein: The allocating the unexecuted target downstream task slices to the target task executor so as to execute the target downstream task slices through the target task executor includes: Determine whether there are unexecuted upstream task shards; If there is no unexecuted upstream task slice, the unexecuted target downstream task slice is allocated to the target task executor, so that the target downstream task slice is executed by the target task executor; If there are unexecuted upstream task slices, the unexecuted upstream task slices are allocated to the target task executor so as to execute the unexecuted upstream task slices through the target task executor.

7. The method according to claim 5, wherein: The allocating the unexecuted target downstream task slices to the target task executor so as to execute the target downstream task slices through the target task executor includes: Determine whether there are long-tail task slices whose execution time exceeds the preset time; If there is no long-tail task slice, the unexecuted target downstream task slice is allocated to the target task executor, so that the target downstream task slice is executed by the target task executor; If there is a long-tail task slice, the long-tail task slice is allocated to the target task executor so as to execute the long-tail task slice through the target task executor; After obtaining the task execution result of the long-tail task slice, the execution process for the long-tail task slice is ended.

8. The method according to claim 7, wherein: The receiving the task allocation request sent by the target task executor includes: Receive a heartbeat signal periodically sent by a target task executor, wherein the heartbeat signal includes: current task slice identification information and task request identification information; the task request identification information indicates whether the target task executor requests a new task slice; If it is determined according to the task request identification information that the target task executor requests a new task slice, it is determined that a task allocation request sent by the target task executor is received; The allocating the long-tail task slices to the target task executor includes: It is determined whether the task slice being executed by the target task executor is the long-tail task slice according to the current task slice identification information; if not, the long-tail task slice is allocated to the target task executor.

9. A task scheduling method, comprising: Sending a task allocation request to a task scheduler so that the task scheduler allocates unexecuted target downstream task slices to target task executors during the scheduling process of multiple tasks; Obtain the target downstream task slice assigned by the task scheduler, and execute the target downstream task slice; The multiple tasks include: an upstream task composed of multiple upstream task slices, and a downstream task composed of multiple downstream task slices; the target downstream task slice is a downstream task slice that depends on the completed upstream task slice.

10. The method according to claim 9, wherein: The target downstream task slice is a long-tail task slice, and the method further includes: After the target downstream task slice is executed, a target directory write request is sent to the task scheduler; In response to receiving a write permission notification returned by the task scheduler, the task execution result of the target downstream task slice is written into the target directory.

11. The method according to claim 10, wherein: After sending the target directory write request to the task scheduler, the method further includes: In response to receiving the write permission occupied notification returned by the task scheduler, the target directory write request is periodically sent to the task scheduler at a preset time interval.

12. The method according to any one of claims 9 to 11, wherein: The sending of the task allocation request to the task scheduler comprises: Periodically sending a heartbeat signal to the task scheduler, wherein the heartbeat signal includes: task request identification information; the task request identification information indicates whether the task executor requests a new task slice; If it is determined according to the task request identification information that the task executor requests a new task slice, then it is determined to send a task allocation request to the task scheduler; The method further comprises: If the target downstream task slice fails to execute, the failure reason is recorded and the heartbeat signal is stopped from being sent to the task scheduler.

13. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1 to 12.

14. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the method according to any one of claims 1 to 12.

Images