Method, apparatus and device for task scheduling

Abstract

The application discloses a task scheduling method, device and equipment, which can be applied to the technical field of big data analysis. The task scheduling method comprises the following steps: obtaining a first task, determining N second tasks when the first task is executed completely. The second task is a downstream task of the first task. When an upstream task of an i-th second task in the N second tasks is executed completely, the i-th second task is executed. In the application, the downstream task can trigger execution when the upstream task is determined to be executed completely, without waiting for a fixed execution time, so that the delay between the upstream task and the downstream task is effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data analysis, and in particular to a task scheduling method, device and equipment. BACKGROUND

[0002] At present, with the rapid development of the Internet, there are more and more types of tasks that need to be run by means of the network. In the technical field of big data analysis, there are complex dependency relationships between multiple tasks, which requires task scheduling to be used to manage the tasks.

[0003] The traditional task scheduling mainly adopts a periodic timing execution and a scheduling mode taking a task as an upstream dependency. Since a user cannot give a most suitable timing execution time when configuring a task, the traditional task scheduling method has the problem of high delay. SUMMARY

[0004] The present application provides a task scheduling method, device and equipment to reduce the delay between upstream and downstream tasks.

[0005] In a first aspect, the present application provides a task scheduling method, comprising: obtaining a first task; determining N second tasks when the first task is executed, the second tasks being downstream tasks of the first task, and N being a positive integer; and executing an i-th second task in the N second tasks when an upstream task of the i-th second task is executed.

[0006] In some possible implementation manners, after the first task is obtained, the method further comprises: executing the first task according to a first dependency data table of the first task and a first execution parameter; and the first dependency data table indicates a data table output by an upstream task on which the first task depends when the first task is executed.

[0007] In some possible implementation manners, when the first task is executed, the N second tasks are determined, comprising: obtaining a first output data table when the first task is executed, the first output data table being a data table output by the first task when the first task is executed; and determining N downstream tasks of the first task that depend on the first output data table as the N second tasks.

[0008] In some possible implementation manners, the first execution parameter comprises at least one of the following: an execution granularity, a dependency granularity, a dependency time offset and an output time offset; wherein the execution granularity is used to represent an execution period of executing the first task; the dependency granularity is used to represent a period of the first dependency data table on which the first task depends when the first task is executed; the dependency time offset is used to represent an offset value between an execution time of executing the first task and an output time of the first dependency data table; and the output time offset is used to represent an offset value between the execution time of executing the first task and an output time of the first task output data table.

[0009] In some possible implementation manners,

[0010] The execution cycle comprises at least one of the following: one month, one week, one day, and one hour.

[0011] In some possible implementation manners, before the i-th second task is executed when the upstream task of the i-th second task in the N second tasks is executed, the method further comprises: polling the N second tasks; determining the upstream task of each second task according to the second dependency data table of each second task, wherein the second dependency data table is used to indicate the data table output by the upstream task on which each second task depends when the second task is executed.

[0012] In some possible implementation manners, after the upstream task of each second task is determined according to the second dependency data table of each second task, the method further comprises: detecting whether the upstream task of each second task outputs a data table; and when it is detected that the upstream task of the i-th second task has output a data table, it is determined that the upstream task of the i-th second task is executed.

[0013] In some possible implementation manners, after the N second tasks are determined when the first task is executed, the method further comprises: registering an execution trigger for each second task according to the second execution parameter of each second task in the N second tasks, the execution trigger being used to trigger the execution of the second task when the execution time of the corresponding second task arrives; and when the upstream task of the i-th second task in the N second tasks is executed, the i-th second task is executed, comprising: when the upstream task of the i-th second task is executed, triggering the execution trigger corresponding to the i-th second task.

[0014] In a second aspect, the present application provides a task scheduling apparatus, which can be a chip or a system on chip in an electronic device, and can also be a functional module for implementing the method of the first aspect and any possible implementation manner thereof in an electronic device. The task scheduling apparatus can implement the functions of the electronic device of the first aspect and any possible implementation manner thereof, and the functions can be implemented by hardware executing corresponding software. The hardware or software comprises one or more modules corresponding to the above functions. The task scheduling apparatus comprises: an acquisition module configured to acquire a first task; a determination module configured to determine N second tasks when the first task is executed, the second tasks being downstream tasks of the first task, and N being a positive integer; and an execution module configured to execute an i-th second task in the N second tasks when an upstream task of the i-th second task is executed.

[0015] In some possible implementation, the obtaining module is further configured to execute the first task according to the first dependency data table and the first execution parameter of the first task after obtaining the first task; and the first dependency data table indicates a data table output by an upstream task on which the first task depends when being executed.

[0016] In some possible implementation, the determining module is further configured to obtain a first output data table when the first task is executed completely, the first output data table being a data table output by the first task when being executed completely; and determine N second tasks from downstream tasks of the first task as the N second tasks, the N second tasks depending on the first output data table.

[0017] In some possible implementation, the first execution parameter comprises at least one of an execution granularity, a dependency granularity, a dependency time offset and an output time offset; the execution granularity is used to indicate an execution period of executing the first task; the dependency granularity is used to indicate a period of the first task depending on the first dependency data table when being executed; the dependency time offset is used to indicate an offset value between an execution time of executing the first task and an output time of the first dependency data table; and the output time offset is used to indicate an offset value between the execution time of executing the first task and an output time of the first output data table.

[0018] In some possible implementation, the execution period comprises at least one of a month, a week, a day and an hour.

[0019] In some possible implementation, the executing module is further configured to poll the N second tasks before executing an i-th second task in the N second tasks when upstream tasks of the i-th second task are executed completely; and determine upstream tasks of each second task according to a second dependency data table of each second task, the second dependency data table being used to indicate a data table output by an upstream task on which each second task depends when being executed.

[0020] In some possible implementation, the executing module is further configured to detect whether the upstream tasks of each second task output data tables after determining the upstream tasks of each second task according to the second dependency data table of each second task; and determine that the upstream tasks of the i-th second task are executed completely when it is detected that the upstream tasks of the i-th second task have output data tables.

[0021] In some possible implementation manners, the execution module is further configured to, after the N second tasks are determined when the first task is executed, register an execution trigger for each of the N second tasks according to a second execution parameter of the second task, the execution trigger being configured to trigger execution of the second task when an execution time of the corresponding second task arrives; and execute an i th second task of the N second tasks when an upstream task of the i th second task is executed, including triggering the execution trigger corresponding to the i th second task when the upstream task of the i th second task is executed.

[0022] In a third aspect, the present application provides an electronic device, comprising: a memory for storing processor-executable instructions; a processor; wherein the processor is configured to execute the executable instructions to implement the method according to the first aspect and any possible implementation manner thereof.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions can implement the method according to the first aspect and any possible implementation manner thereof when executed by a processor.

[0024] The technical solution provided by the present application has the following beneficial effects compared with the prior art:

[0025] In the present application, the first task is acquired, and N second tasks are determined when the first task is executed, wherein the second tasks are downstream tasks of the first task. An i th second task of the N second tasks is executed when an upstream task of the i th second task is executed. It can be seen that the downstream task in the present application can trigger execution when the upstream task is determined to be executed, without waiting for a fixed execution time, thereby effectively reducing the delay between the upstream and downstream tasks.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the protection scope of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 FIG. 1 is an implementation flow diagram of the task scheduling method in the embodiments of the present application;

[0028] Figure 2 FIG. 2 is another implementation flow diagram of the task scheduling method in the embodiments of the present application;

[0029] Figure 3 FIG. 3 is a schematic diagram of the task configuration structure in the embodiments of the present application;

[0030] Figure 4 FIG. 4 is a schematic diagram of the structure of the system table in the embodiments of the present application;

[0031] Figure 5 Another schematic diagram of the structure of a table for a system in an embodiment of the present application;

[0032] Figure 6 Yet another implementation flow schematic diagram of a method for task scheduling in an embodiment of the present application;

[0033] Figure 7 A structure schematic diagram of a device for task scheduling in an embodiment of the present application;

[0034] Figure 8 A structure schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0036] In order to illustrate the technical solutions described in the present application, the following will be described through specific embodiments.

[0037] At present, with the rapid development of the Internet, the types of tasks that need to be run with the aid of the network are increasing. Especially in the field of big data, more and more enterprises begin to focus on data. Enterprises usually use the powerful computing power of server clusters to obtain various data reports, so as to intuitively understand and understand the related business through these reports.

[0038] With the increase of network data, the demand for data analysis is also increasing. For example, when a certain business requires daily big data analysis of various dimensions, a large number of data analysis tasks will be generated. Because the execution time of each data processing task is limited by factors such as changing computing cluster resources and processing data volume, it is impossible to estimate how much time a task actually needs. When the user configures the task, it is impossible to give a most appropriate timing execution time. Further, there is a high delay between upstream and downstream tasks when the upstream task is completed, and the downstream task has to wait because it has not reached the timing execution time.

[0039] For example, suppose task A is executed once a day at a set time, producing data 'a'. However, because task A actually executes Structured Query Language (SQL) to produce data for T-2 (T-2 means the current period calculates data from the previous two periods), when task A executes on January 3rd, it actually produces data for January 1st. When users are unaware of the specific execution content of task A, they will assume that task A, when executed daily, should produce data for that day (T-0). Therefore, when configuring downstream task B, the dependency time range associated with task A's execution time will be configured, causing task B to run data without actually producing it each day, because the data 'a' from T-0 will always be produced two days later.

[0040] Therefore, existing task scheduling methods suffer from high task latency due to user configuration.

[0041] To address the aforementioned problems, this application provides a task scheduling method applied in the field of big data analytics. The execution entity for each step of this method can be an electronic device with computing and processing capabilities. In one embodiment, the electronic device can be a terminal, such as a mobile phone, tablet computer, or smart wearable device; in another embodiment, the electronic device can be a server, which can be a single server, a server cluster consisting of multiple servers, or a cloud server. This application does not limit the specific implementation of this method.

[0042] Figure 1 This is a schematic diagram illustrating an implementation flow of the task scheduling method in this application. See also: Figure 1 As shown, the task scheduling method may include S101 to S103.

[0043] S101, the electronic device acquires the first task.

[0044] It should be understood that the first task can be any single configuration task; or, the first task can be multiple configuration tasks. The configuration process can be completed by the user inputting information on a dedicated configuration page; or, the configuration process can be completed by the electronic device itself. The configuration process can also be completed in other ways, which are not specifically limited in this application.

[0045] It should be noted that the process of configuring the task can include configuring the dependency relationship of the task and configuring the execution parameter of the task. The dependency relationship is used to indicate the execution dependency between tasks, for example, if the execution of task A needs to use the execution result of task B, then the dependency relationship of task A and task B is that the upstream task of task A is task B. The execution parameter is used to indicate the rule followed by the current task when executing, for example, task A follows the rule of executing once a week when executing, so that executing once a week is the execution parameter of task A.

[0046] It should be understood that after the task configuration is completed, the electronic device will store the task and the configuration information (dependency relationship and execution parameter) of the task, so as to be subsequently called.

[0047] In some possible implementation manners, S101 can further include S201. S201 can be executed after S101 is executed and before S102 is executed. Figure 2 An implementation flow diagram of the method for task scheduling in the embodiments of the present application.

[0048] S201, the electronic device executes the first task according to the first dependency data table and the first execution parameter of the first task.

[0049] The first dependency data table indicates the data table output by the upstream task on which the first task depends when executing.

[0050] It should be understood that in the big data scenario, according to the task configuration information, the upstream data required when the task is executed and the output data of the executed task can be obtained. The upstream data refers to the data table to be queried when the current task is executed, and the output data refers to the data table to which the data is finally written after the current task is executed. The upstream data can be multiple, and the output data can be only one. Each task can output data to obtain the output data table (which can be understood as the output data table). The output data table can be used as the upstream data (which can be understood as the dependency data table) of the downstream task.

[0051] As described above, the electronic device obtains the first task through S101, and then obtains the configuration information of the first task. S201 is executed to obtain the first dependency data table and the first execution parameter of the first task through the configuration information of the first task. The first task is executed according to the first dependency data table and the first execution parameter.

[0052] In some possible implementation manners, the first execution parameter can include at least one of the following: an execution granularity, a dependency granularity, a dependency time offset, and an output time offset. The execution granularity is used to represent an execution period of executing the first task; the dependency granularity is used to represent a period of the first task depending on the first dependency data table when the first task is executed; the dependency time offset is used to represent an offset value between an execution time of executing the first task and an output time of the first dependency data table; and the output time offset is used to represent an offset value between the execution time of executing the first task and an output time of the first output data table.

[0053] It should be understood that in the big data scenario, the data table always needs to store data in partitions. For example, the partitions can include month partitions, week partitions, and the like. In the month partition, the data is output once a month to store the calculation result in the partition belonging to the month. Therefore, in the embodiment of the present application, the execution period of the task and the dependency period of the task can be set, and the execution granularity and the dependency granularity can be used to uniformly manage the periods. Preferably, the execution period can include a month, a week, a day, and an hour.

[0054] Further, the execution time of the task represents the period running time of the task. For example, the execution time of the task running in the hour period can be 2022 / 03 / 01 00:00:00 for the first time and 2022 / 03 / 01 01:00:00 for the second time. The output time of the task represents the specific time of outputting the data each time the task is executed. For example, the output time of the task running in the day period can be 2022 / 03 / 01 00:00:00 for the first time and 2022 / 03 / 02 00:00:00 for the second time.

[0055] Further, the time offset is introduced according to the execution time and the output time. The time offset represents the offset value between the execution time and the output time. That is, the execution time + the offset value = the output time. In the embodiment of the present application, since the task is set to different granularities, it is not easy to represent the characteristics of the tasks of different granularities by using a single offset value. Therefore, the offset value is divided into two indexes of granularity and offset value to represent the characteristics of the tasks of different granularities. For example, the dependency time offset represents the offset value between the execution time of the task and the output time of the upstream task, which is expressed as the output time of the upstream task - the dependency time offset * the dependency granularity = the execution time of the downstream task. The output time offset represents the offset value between the execution time of the task and the output time of the task, which is expressed as the execution time of the task + the output time offset * the execution granularity = the output time of the data.

[0056] In an embodiment, Figure 3 A schematic diagram of the structure configured for the task in the embodiment of the present application is shown in FIG. 1. Figure 3As shown, there are six tasks. Among them, three tasks (task A, task B, task C) do not configure the dependency data table, so they are upstream tasks; three tasks (task D, task E, task F) are configured with the dependency data table, so they are downstream tasks.

[0057] The upstream task A outputs the data table a, the granularity of the table a is the hour level, and the data of T-1 is output. Wherein, T-1 represents that the output time of task A is one hour earlier than the execution time. The upstream task B outputs the data table b, the granularity of the table b is the day level, and the data of T-1 is output. Wherein, T-1 represents that the output time of task B is one day earlier than the execution time. The upstream task C outputs the data table c, the granularity of the table c is the day level, and the data of T-3 is output. Wherein, T-3 represents that the output time of task C is three days earlier than the execution time. The downstream task D outputs the data table d, the granularity of the table d is the day level. The data of T-1 of the table a is used in the day granularity. Wherein, T-1 represents that the output time of the upstream task A is one day earlier than the execution time of the downstream task D. The downstream task E outputs the data table e, the granularity of the table e is the day level. The data of T-2 of the table a and the data of T-1 of the table b are used in the hour granularity and the day granularity respectively. Wherein, T-2 represents that the output time of the upstream task A is two hours earlier than the execution time of the downstream task E; T-1 represents that the output time of the upstream task B is one day earlier than the execution time of the downstream task E. The downstream task F outputs the data table f, the granularity of the table f is the month level. The data of T-1 of the table c is used in the month granularity. Wherein, T-1 represents that the output time of the upstream task C is two months earlier than the execution time of the downstream task F.

[0058] For example, the task configuration of Figure 3 The electronic device determines whether the output data table a (abbreviated as table a) of task A has success information at the data output time (2022 / 02 / 28 23:00:00) corresponding to the current system time (2022 / 03 / 01 00:05:36), and if not, task A is screened out, triggering the execution of the task at the time (2022 / 03 / 01 00:00:00) corresponding to the current system time.

[0059] Specifically, the conversion logic of the system time and the data output time in the above example is: the number of units in the system time that is lower than the granularity of the output data of the current task is removed and set to 0, and the obtained is the task execution date, and then the data output date is obtained by the task execution date and the time period offset of the output data.

[0060] S102, when the first task is completed, the electronic device determines N second tasks, which are downstream tasks of the first task.

[0061] It should be understood that when the first task is completed, the electronic device can determine the downstream tasks of the first task based on the task's configuration information. Due to the complex dependencies between tasks, the first task may have one or more downstream tasks.

[0062] In some possible implementations, when it is necessary to determine the downstream task of the first task, the electronic device can retrieve configuration information stored in a database and determine the downstream task of the first task based on the dependency information in the configuration information. It should be understood that the configuration information can be stored in the form of a data table.

[0063] In one embodiment, Figure 4 This is a schematic diagram of the structure of the system table in an embodiment of this application. See also... Figure 4 As shown, the task's configuration information is written as follows: Figure 4 The relationship table (RELATION) is used for storage. The RELATION table records information including the task name, output data table, dependency data table, output granularity, and time offset. Electronic devices can retrieve task configuration information by querying the RELATION table. After the first task is completed, the electronic device can also store information indicating successful task execution. See also... Figure 4 As shown, the message indicating successful task execution is written as follows: Figure 4 The task status is stored in the DATASET_STATUE table shown. The DATASET_STATUE table records information including the task name, data output date, and status (determining success). Electronic devices can retrieve information about whether a task was executed successfully by querying the DATASET_STATUE table.

[0064] For example, with Figure 3 Taking the task configuration as an example, the output data granularity of task A is at the hourly level. Therefore, the electronic device will trigger task A to execute once every hour. Assuming that task A, which was executed at 00:00:00 on 2022 / 03 / 01, is successfully executed, since the execution parameters of task A are configured to produce T-1 data, the final status recorded in the DATASET_STATUE table at the time when the output data table a is entered (23:00:00 on 2022 / 02 / 28) will be "successful".

[0065] In some possible implementation manners, S102 further includes obtaining a first output data table when the first task is executed to completion, the first output data table being a data table output by the first task when the first task is executed to completion. Among the downstream tasks of the first task, N downstream tasks that depend on the first output data table are determined as N second tasks.

[0066] It should be understood that after the first task is executed to completion, the electronic device writes the output data of the first task into a table (which can be understood as an output data table), and obtains the first output data table of the first task. The electronic device can obtain N tasks that depend on the first output data table (that is, the dependent data table of the N tasks is the first output data table) by querying the task configuration information stored after the task configuration is completed, and determine the N tasks as second tasks.

[0067] For example, as shown in the task configuration of Figure 3 , the output data table of task A is a, and task D and task E are both configured with the dependent data table a. Therefore, task D and task E are determined as downstream data of task A.

[0068] S103, when the upstream tasks of the i-th second task in the N second tasks are executed to completion, the electronic device executes the i-th second task.

[0069] It should be understood that after the task is executed to completion, the electronic device stores the information that the task is executed to completion. The electronic device queries whether all upstream tasks of the i-th second task are executed to completion according to the stored information, and triggers execution of the i-th second task when the upstream tasks are executed to completion. Since the dependency relationship between tasks is complex, the i-th second task can have multiple upstream tasks.

[0070] Among them, whether all upstream tasks of the i-th second task are executed to completion is determined according to whether the upstream tasks output the dependent data table required by the second task, so the dependent blood relationship between the second task and the upstream task is generated in the form of data table, and is irrelevant to the task itself.

[0071] For example, as shown in the task configuration of Figure 3 , task E depends on task A. It is assumed that task A outputs data with a date of 2022 / 02 / 28 23:00:00 and writes the data into the data output table a. Since task E uses the T-2 data of table a at the hour granularity. According to the upstream task output time-dependent time offset * dependent granularity = downstream task execution time (that is, 2022 / 02 / 28 23:00:00-(-2)*day = 2022 / 03 / 02 00:00:00), it can be determined that the execution time of task E is 2022 / 03 / 02 00:00:00.

[0072] When the electronic device acquires that task E needs to be executed at 2022 / 03 / 02 00:00:00, it checks whether table a exists in a successful state every hour between 2022 / 02 / 28 00:00:00-2022 / 02 / 28 23:00:00 and whether table b exists in a successful state at date 2022 / 03 / 01 00:00:00. If the conditions are met, task E is executed immediately.

[0073] In some possible implementation manners, as shown in FIG. 2, S103 can include S202. S202 can be executed after S102 is executed and before S103 is executed. Figure 2

[0074] S202, the electronic device polls N second tasks; and determines an upstream task of each second task according to a second dependency data table of each second task. The second dependency data table is used to indicate a data table output by an upstream task on which each second task depends when executed.

[0075] It should be understood that when there are multiple second tasks, each second task needs to be polled. According to the configuration information stored after task configuration, the second dependency data table of the current second task is determined, and the task outputting the second dependency data table of the second task is determined as the upstream task of the current second task. How to determine the upstream task according to the task configuration information can be referred to S102.

[0076] In some possible implementation manners, the electronic device detects whether the upstream task of each second task outputs a data table; and when it is detected that the upstream task of the i th second task has output a data table, it is determined that the upstream task of the i th second task is executed.

[0077] It should be understood that after the task is executed, the electronic device stores the completion information. When the electronic device needs to query the information of the upstream task required by the i th second task, the stored information can be called, and when it is determined that the dependency data table at the current system time has been output by the upstream task, it is determined that the upstream task is executed.

[0078] For example, Figure 4 Table DATASET_STATUE in FIG. 2 is used to store the execution success information. The electronic device can query the information of whether the task is executed successfully by querying table DATASET_STATUE.

[0079] In some possible implementation manners, as shown in FIG. 2, S103 can include S202. S202 can be executed after S102 is executed and before S103 is executed. Figure 2 ​The dashed box and dashed arrow indicate that S203 can be included after S102, and S203 can be executed after S102 and before S103. Alternatively, S203 can be included after S102, and S202 can be executed after S203.

[0080] S203, based on the second execution parameters of each of the N second tasks, the electronic device registers an execution trigger for each second task. The execution trigger is used to trigger the execution of the second task when the execution time of the corresponding second task arrives.

[0081] It should be understood that the second execution parameter can be used to determine the execution time of the second task. After determining the execution time, the electronic device registers an execution trigger with the execution time for each second task. When the execution time is reached, the electronic device triggers the execution of the second task.

[0082] In some possible implementations, the execution time of the second task is determined by the time it takes for the upstream task to produce the dependency data table required by the second task. Therefore, the second execution parameters may include dependency granularity and dependency time offset.

[0083] For example, with Figure 3 Taking task configuration as an example, refer to the calculation process in S103. Assume that task A is executed on 2022 / 03 / 01 00:00:00. When task A is successfully executed, since downstream task E depends on data 'a' of T-2 with a daily granularity, a trigger for task E with an execution date of 2022 / 03 / 02 00:00:00 will be registered.

[0084] In one embodiment, Figure 5 This is another schematic diagram of the structure of the system table in the embodiments of this application. See also Figure 5 As shown, information about each second task execution trigger is recorded in the trigger table (TRIGGER). The TRIGGER table records information including the task name, task execution date, and status (determining success). Electronic devices can retrieve information about the second task corresponding to the current trigger by querying the TRIGGER table.

[0085] In some possible embodiments, such as Figure 2 As shown in the dashed box and dashed arrow, S204 is included after S203. S204 can be executed after S203 and before S103. Alternatively, S202 can be included before S204, S202 can be executed after S203, and S204 can be executed after S202.

[0086] S204, when the upstream task of the i-th second task is executed, the electronic device triggers the execution trigger corresponding to the i-th second task.

[0087] It should be understood that the electronic device obtains whether the upstream task of the second task in the trigger has a successful state by calling the stored data. If the upstream task of the second task has been successfully executed, the current task is triggered for execution.

[0088] Wherein, the upstream task of the second task has been successfully executed means that the dependent data table of the second task has been output at the current time.

[0089] For example, the electronic device obtains all triggers in the triggered state at an interval of 5s, and then checks the corresponding task required to be executed in the trigger. The trigger obtains the upstream task of the corresponding task, queries the state of the upstream task of the corresponding task, and if the upstream task has output the dependent data table, the corresponding task in the trigger is triggered for execution.

[0090] In this embodiment, by S101 to S103, the first task is obtained, and when the first task is executed, N second tasks are determined, wherein the second task is a downstream task of the first task. When the upstream task of the i-th second task in the N second tasks is executed, the i-th second task is executed. It can be seen that the downstream task in the present application can be executed when the upstream task is determined to be executed, without waiting for a fixed execution time, effectively reducing the delay between the upstream and downstream tasks.

[0091] In this embodiment, by S101 to S103 and S201 to 204, it can be seen that the first task is obtained, and when the first task is executed, N second tasks (downstream tasks) are determined. The execution trigger is registered for the N second tasks, and when it is checked that the upstream task of the i-th second task is executed, the execution trigger is triggered to execute the i-th second task. The electronic device obtains the task in the trigger at a very low time interval and executes. In the whole process, the delay of scheduling the upstream and downstream tasks is maintained at the level of seconds, which can effectively reduce the delay between the upstream and downstream tasks. At the same time, the output data table is directly used as the upstream, and the force and the dependent time offset are configured according to the current task. The problem of difficult configuration and configuration error caused by the inconsistency between the execution time and the output time when the task is used as the upstream is avoided.

[0092] The process of task scheduling in the embodiment of the present application will be described below.

[0093] Figure 6 Another implementation flow diagram of the task scheduling method in the embodiment of the present application is shown. As shown in Figure 6 , it includes:

[0094] S601, the electronic device parses the task, configures the dependency information and execution parameters of the task, and enters S602;

[0095] Firstly, the electronic device performs syntax analysis on the SQL task of the user by a syntax analysis tool, obtains the dependent data table and the output data table required by the task from the analysis result, and supplements the configuration information of the task; secondly, the electronic device configures the execution parameters of the task according to the use of the user SQL.

[0096] S602, the electronic device stores the dependency information and execution parameters of the task, and enters S603;

[0097] Among them, the electronic device can enter the dependency information and execution parameters of the task into the table RELATION.

[0098] S603, the electronic device obtains all upstream tasks and judges the task execution, and enters S604;

[0099] Among them, the electronic device can obtain all tasks without upstream data at an interval of 5s, and judge whether the output data table of the task has success information at the data output date corresponding to the current system time.

[0100] S604, the electronic device obtains all unexecuted upstream tasks, and triggers the execution of the task, and enters S605;

[0101] Among them, the electronic device filters out the data table without success information, obtains the corresponding task through the table RELATION, and triggers the execution of the task corresponding to the current system time.

[0102] S605, the electronic device registers the execution trigger for the downstream task after the upstream task is executed, and enters S606;

[0103] Among them, after the task is executed, the electronic device enters the success information of the corresponding output data into the table DATASET_STATUE. The electronic device finds the downstream task of the task through the table RELATION, registers the execution trigger for the downstream task, and stores the execution trigger information in the table TRIGGER.

[0104] S606, the electronic device queries the execution trigger to determine that the corresponding upstream task is all successful, and enters S607;

[0105] Among them, the electronic device obtains all execution triggers in the state of waiting for triggering in the table TRIGGER at an interval of 5s, and checks whether the upstream task of the corresponding task in the execution trigger has a success state in the table DATASET_STATUE.

[0106] S607, execute the downstream task.

[0107] Wherein, the electronic device queries the table DATASET_STATUE to show that the upstream dependent table has been successfully, immediately executes the current task, and modifies the table TRIGGER to be successfully. The electronic device executes all tasks by repeating S604 to S607, and realizes task scheduling.

[0108] In the embodiments of the present application, as can be known from S601 to S607, the electronic device obtains downstream tasks through upstream tasks, and registers execution triggers for each downstream task. The execution triggers are constantly polled to query whether the upstream tasks of the execution trigger corresponding tasks are executed. If all the upstream tasks are executed, the downstream task is executed. As can be seen, after each task in the present application is executed, the downstream task can be quickly found and the execution trigger is registered for it through the data bloodline maintained in the database. The electronic device acquires the task in the execution trigger and executes it at a very low time interval. The delay of scheduling upstream and downstream tasks in the whole process is maintained at seconds, realizing low delay of task scheduling. Further, the output data table is directly used as the upstream, and the configuration intensity and dependent time offset are configured according to the current task. The problem of difficult configuration and configuration error caused by the inconsistency between the execution time and the output time when the task is used as the upstream is avoided.

[0109] Based on the same inventive concept, the embodiments of the present application also provide a task scheduling device. The task scheduling device can be a chip or a system on chip in an electronic device, and can also be a functional module for realizing the method described in each of the above embodiments. The task scheduling device can realize the functions of the electronic device in each of the above embodiments, and these functions can be realized by executing corresponding software by hardware. These hardware or software include one or more modules corresponding to the above functions.

[0110] Figure 7 For the structure of a task scheduling device in an embodiment of the present application, refer to Figure 7 As shown in the figure, the task scheduling device 700 can include: an acquisition module 701, configured to acquire a first task; a determination module 702, configured to determine N second tasks when the first task is executed, the second task being a downstream task of the first task, and N being a positive integer; and an execution module 703, configured to execute an i th second task in the N second tasks when an upstream task of the i th second task is executed.

[0111] In some possible implementation manners, the acquisition module 701 is further configured to execute the first task according to a first dependent data table and a first execution parameter of the first task after acquiring the first task; and the first dependent data table indicates a data table output by an upstream task on which the first task depends when the first task is executed.

[0112] In some possible implementation, the determining module 702 is further configured to: obtain a first output data table when the first task is executed completely, the first output data table being a data table output by the first task when the first task is executed completely; and determine N second tasks in the downstream tasks of the first task as the N second tasks, the N second tasks depending on the first output data table.

[0113] In some possible implementation, the first execution parameter comprises at least one of: an execution granularity, a dependency granularity, a dependency time offset, and an output time offset; the execution granularity is used to represent an execution period of executing the first task; the dependency granularity is used to represent a period of the first task depending on the first dependency data table when the first task is executed; the dependency time offset is used to represent an offset value between an execution time of executing the first task and an output time of the first dependency data table; and the output time offset is used to represent an offset value between the execution time of executing the first task and an output time of the first output data table.

[0114] In some possible implementation, the execution period comprises at least one of: one month, one week, one day, and one hour.

[0115] In some possible implementation, the execution module 703 is further configured to: poll the N second tasks before executing an i-th second task in the N second tasks when upstream tasks of the i-th second task are executed completely; and determine upstream tasks of each second task according to a second dependency data table of each second task, the second dependency data table being used to indicate a data table output by an upstream task on which each second task depends when the second task is executed.

[0116] In some possible implementation, the execution module 703 is further configured to: after determining the upstream tasks of each second task according to the second dependency data table of each second task, detect whether the upstream tasks of each second task output data tables; and when it is detected that the upstream tasks of the i-th second task have output data tables, determine that the upstream tasks of the i-th second task are executed completely.

[0117] In some possible implementation, the execution module 703 is further configured to: after determining the N second tasks when the first task is executed completely, register an execution trigger for each second task according to a second execution parameter of each second task in the N second tasks, the execution trigger being used to trigger the second task to execute when an execution time of the corresponding second task arrives; and execute the i-th second task when the upstream tasks of the i-th second task are executed completely, comprising: triggering the execution trigger corresponding to the i-th second task when the upstream tasks of the i-th second task are executed completely.

[0118] It should be noted that the specific implementation process of the obtaining module 701, the determining module 702, and the execution module 703 can be referred to the description of the obtaining module 101, the determining module 102, and the execution module 103.Figures 1 to 6 The detailed description of the embodiments will not be repeated here for the sake of brevity of the description.

[0119] Based on the same inventive concept, the embodiments of the present application provide an electronic device, which can be the electronic device described in one or more of the above embodiments. Figure 8 FIG. 8 is a structural schematic diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown in FIG. 8, the electronic device 800 can employ general computer hardware, including a processor 801 and a memory 802.

[0120] In some possible implementations, the at least one processor can constitute any physical device having a circuit that performs logical operations on one or more inputs. For example, the at least one processor can include one or more integrated circuits (ICs), including application-specific integrated circuits (ASICs), microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), or other circuitry suitable for executing instructions or performing logical operations. The instructions executed by the at least one processor can be, for example, preloaded into a memory integrated with or embedded in the controller, or can be stored in a separate memory. The memory can include random access memory (RAM), read-only memory (ROM), a hard disk, an optical disk, a magnetic medium, a flash memory, other permanent, fixed, or volatile memory, or any other mechanism capable of storing instructions. In some embodiments, the at least one processor can include more than one processor. Each processor can have a similar structure, or the processors can have different structures that are electrically connected or disconnected from each other. For example, the processors can be separate circuits or integrated in a single circuit. When more than one processor is used, the processors can be configured to operate independently or cooperatively. The processors can be coupled electrically, magnetically, optically, acoustically, mechanically, or by other means that allow them to interact.

[0121] According to an embodiment of the present application, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which are executed by a processor to perform the steps of the above method for scheduling tasks. The memory 802 can include computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) and / or random access memory (RAM). The memory 802 can store operating systems, application programs, other program modules, executable code, program data, user data, and the like.

[0122] In addition, the above memory 802 stores computer execution instructions for implementing the functions of the acquisition module 701, the determination module 702, and the execution module 703 in the above method. Figure 7 Figure 7 ​The functions / implementation processes of the acquisition module 701, the determination module 702 and the execution module 703 can be implemented by calling the computer execution instructions stored in the memory 802 by the processor 801 in the computer device 800, and the specific implementation processes and functions are referred to the related embodiments above. Figure 8 The functions / implementation processes of the acquisition module 701, the determination module 702 and the execution module 703 can be implemented by calling the computer execution instructions stored in the memory 802 by the processor 801 in the computer device 800, and the specific implementation processes and functions are referred to the related embodiments above.

[0123] Those skilled in the art can understand that the sequence of the steps in the above embodiments does not mean the execution order, the execution order of the processes should be determined according to the functions and the inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0124] The above described embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that it can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A task scheduling method, characterized in that, The method includes: Obtain the first task; When the first task is completed, a first output data table is obtained; the first output data table includes the output data of the first task; Based on the first output data table and the stored configuration information, N second tasks are determined, wherein the second tasks are downstream tasks of the first tasks, and N is a positive integer; the configuration information indicates the offset between the execution time of the first task and the output time of the first output data table, and the N second tasks depend on the first output data table. Based on the configuration information, a second dependency data table is determined for each second task; wherein, the configuration information further indicates the offset between the execution time of the second task and the output time of the second dependency data table; the second dependency data table is used to indicate the data table produced by the upstream task that each second task depends on during execution; When the upstream task of the i-th second task among the N second tasks produces the data table indicated by the second dependent data table, it is determined that the upstream task of the i-th second task has been completed. The i-th second task is executed when the upstream task of the i-th second task is completed.

2. The method according to claim 1, characterized in that, After obtaining the first task, the method further includes: The first task is executed according to the first dependency data table and the first execution parameters of the first task; the first dependency data table indicates the data table produced by the upstream task that the first task depends on during execution.

3. The method according to claim 2, characterized in that, The step of determining N second tasks based on the first output data table and the stored configuration information includes: Based on the first output data table and the stored configuration information, N downstream tasks that depend on the first output data table are identified as the N second tasks.

4. The method according to claim 2, characterized in that, The first execution parameter includes at least one of the following: execution granularity, dependency granularity, dependency time offset, and output time offset; Wherein, the execution granularity is used to represent the execution cycle of the first task; The dependency granularity is used to represent the period of the first dependency data table that the first task depends on during execution; The dependency time offset is used to represent the offset value between the execution time of the first task and the output time of the first dependency data table; The output time offset is used to represent the offset value between the execution time of the first task and the output time of the first task output data table.

5. The method according to claim 4, characterized in that, The execution cycle includes at least one of the following: one month, one week, one day, and one hour.

6. The method according to claim 2, characterized in that, Before executing the i-th second task, after the upstream task of the i-th second task has completed execution, the method further includes: Poll the N second tasks mentioned above; Based on the second dependency data table for each second task, determine the upstream task for each second task, wherein the second dependency data table is used to indicate the data table produced by the upstream task that each second task depends on during execution.

7. The method according to claim 6, characterized in that, The method further includes: Check whether the upstream task of each second task has produced a data table.

8. The method according to claim 1, characterized in that, After determining the N second tasks, the method further includes: Based on the second execution parameters of each of the N second tasks, an execution trigger is registered for each second task. The execution trigger is used to trigger the execution of the second task when the execution time of the corresponding second task arrives. The step of executing the i-th second task when the upstream task of the i-th second task is completed includes: When the upstream task of the i-th second task is completed, the execution trigger corresponding to the i-th second task is triggered.

9. A task scheduling apparatus, characterized in that, include: The acquisition module is used to acquire the first task; A determining module is configured to: obtain a first output data table upon completion of the first task, the first output data table including the output data of the first task; determine N second tasks based on the first output data table and stored configuration information, wherein the second tasks are downstream tasks of the first tasks, and N is a positive integer; the configuration information indicates the offset between the execution time of the first task and the output time of the first output data table, and the N second tasks depend on the first output data table; determine a second dependency data table for each second task based on the configuration information; wherein the configuration information further indicates the offset between the execution time of the second task and the output time of the second dependency data table; the second dependency data table is used to indicate the data table produced by the upstream task that each second task depends on during execution; The execution module is configured to determine that the upstream task of the i-th second task has been completed when the upstream task of the i-th second task among the N second tasks produces the data table indicated by the second dependent data table; and to execute the i-th second task when the upstream task of the i-th second task has been completed.

10. The apparatus according to claim 9, characterized in that, The acquisition module is further configured to, after acquiring the first task, execute the first task according to the first dependency data table and the first execution parameters of the first task; the first dependency data table indicates the data table produced by the upstream task that the first task depends on during execution.

11. The apparatus according to claim 10, characterized in that, The determining module is further configured to determine, based on the first output data table and the stored configuration information, N downstream tasks of the first task that depend on the first output data table as the N second tasks.

12. The apparatus according to claim 10, characterized in that, The first execution parameter includes at least one of the following: execution granularity, dependency granularity, dependency time offset, and output time offset; Wherein, the execution granularity is used to represent the execution cycle of the first task; the dependency granularity is used to represent the cycle of the first dependency data table that the first task depends on during execution; the dependency time offset is used to represent the offset value between the execution time of the first task and the output time of the first dependency data table; and the output time offset is used to represent the offset value between the execution time of the first task and the output time of the first task's output data table.

13. The apparatus according to claim 12, characterized in that, The execution cycle includes at least one of the following: one month, one week, one day, and one hour.

14. The apparatus according to claim 10, characterized in that, The execution module is further configured to poll the N second tasks before executing the i-th second task when the upstream task of the i-th second task is completed; and to determine the upstream task of each second task according to the second dependency data table of each second task, wherein the second dependency data table is used to indicate the data table produced by the upstream task that each second task depends on during execution.

15. The apparatus according to claim 14, characterized in that, The execution module is also used to detect whether the upstream task of each second task has produced a data table.

16. The apparatus according to claim 9, characterized in that, The execution module is further configured to, after determining the N second tasks, register an execution trigger for each of the N second tasks according to the second execution parameters of each of the second tasks, wherein the execution trigger is configured to trigger the execution of the second task when the execution time of the corresponding second task arrives; The step of executing the i-th second task when the upstream task of the i-th second task is completed includes: triggering the execution trigger corresponding to the i-th second task when the upstream task of the i-th second task is completed.

17. An electronic device, characterized in that, include: Memory used to store processor-executable instructions; A processor; wherein the processor is configured to, when executing the executable instructions, implement the method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The readable storage medium stores an executable program, wherein the executable program, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images