Task processing method, device, server and storage medium

Abstract

The present disclosure relates to a task processing method, device, server and storage medium, the method comprising: obtaining an input initial task identifier and a target time interval; obtaining a task corresponding to a task identifier identical to the initial task identifier from a task dependency structure as a current task; obtaining a downstream task corresponding to the current task from the task dependency structure, and using the current task and the downstream task corresponding to the current task as target tasks; determining task execution data of the target task according to a scheduling period and a target time interval of the target task; determining a data dependency relationship between task execution data according to the task dependency relationship of the target task; generating a task execution data structure including task execution data according to the data dependency relationship, and executing the task execution data according to the task execution data structure. The task execution data structure of this embodiment occupies less memory resources, thereby improving the utilization rate of memory resources.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a task processing method, device, server and storage medium. Background Art

[0002] The calculation, analysis and processing of big data are usually completed by many independent processes, each of which completes a specific data processing logic. These processes are called tasks. In the actual processing of a big data workflow, there is a sequential dependency relationship between tasks, which can also be called a blood relationship.

[0003] For example, to process task A, you must first complete the processing of task B, because the processing of task A depends on the processing result of task B, so the dependency relationship between task A and task B is: task A depends on task B. Due to the dependency relationship between tasks, in order to ensure the normal operation of the overall task processing in the workflow, the tasks need to be executed in an orderly and efficient manner according to the dependency relationship. Therefore, before each task is executed, it is necessary to build a task instance lineage graph based on its task metadata. The task instance lineage graph sorts out the dependency relationship between the task instances of each task, and executes the task instances in order according to the task instance lineage graph to complete the execution of the task.

[0004] However, since the full task instance lineage graph will be stored in the database after construction, in order to facilitate the recovery of the task instance lineage graph for fast task execution, the scale of tasks written each time is around one million, which requires a large amount of memory resources, resulting in low memory resource utilization. Summary of the invention

[0005] The present disclosure provides a task processing method, device, server and storage medium to at least solve the problem in the related art that maintaining a full amount of task instance lineage graph occupies a large amount of memory resources, resulting in low utilization of memory resources. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided a task processing method, including:

[0007] Get the input initial task ID and target time interval;

[0008] From the task dependency structure, obtaining a task corresponding to a task identifier identical to the initial task identifier as the current task; the task dependency structure includes task identifiers, scheduling cycles and task dependencies corresponding to a plurality of tasks;

[0009] Acquire the downstream task corresponding to the current task from the task dependency structure, and use the current task and the downstream task corresponding to the current task as target tasks;

[0010] Determining task execution data of the target task according to the scheduling period and the target time interval of the target task;

[0011] Determining the data dependency relationship between the task execution data according to the task dependency relationship of the target task;

[0012] A task execution data structure including the task execution data is generated according to the data dependency, and the task execution data is executed according to the task execution data structure.

[0013] In one embodiment, determining the task execution data of the target task according to the scheduling period and the target time interval of the target task includes:

[0014] Determining at least one task execution time according to the scheduling period of the target task and the target time interval;

[0015] The task execution time is used as the execution time of the target task to obtain at least one task execution data of the target task.

[0016] In one embodiment, before obtaining, from the task dependency structure, a task corresponding to a task identifier identical to the initial task identifier as the current task, the method further includes:

[0017] Acquire task metadata; the task metadata includes a task identifier, a scheduling period, and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task;

[0018] Generate a dependency relation formula of the task according to the upstream dependency descriptor and the scheduling period of the upstream task;

[0019] The task dependency structure is generated according to the task identifier corresponding to the task and the dependency formula.

[0020] In one embodiment, the upstream dependency descriptor includes an operator and a dependency time, and generating the dependency relationship of the task according to the upstream dependency descriptor and the scheduling period of the upstream task includes:

[0021] Obtaining a bitmap; the bitmap includes the scheduling period, the operator and the assignment position of the dependent time;

[0022] According to the conversion relationship for the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship for the operator, the operator is converted into a second result value; according to the conversion relationship for the dependent time, the dependent time is converted into a third result value;

[0023] According to the scheduling period, the operator and the assignment position of the dependent time, the first result value, the second result value and the third result value are respectively imported into the corresponding positions in the bitmap to obtain the task dependency relationship of the task.

[0024] In one embodiment, after generating the task dependency structure according to the task identifier corresponding to the task and the dependency formula, the method further includes:

[0025] Get incremental task metadata and corresponding operation behaviors;

[0026] The task dependency structure is updated according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure.

[0027] In one of the embodiments, the incremental task metadata includes the incremental task identifier, incremental scheduling period and incremental upstream dependency data corresponding to the incremental task; the incremental upstream dependency data includes the incremental upstream task identifier and incremental upstream dependency descriptor of the incremental upstream task corresponding to the incremental task.

[0028] In one embodiment, updating the task dependency structure according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure includes:

[0029] When the operation behavior is a newly added operation behavior, determining a dependency relationship formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task;

[0030] The incremental task identifier corresponding to the incremental task and the dependency relationship formula are added to the task dependency relationship structure to obtain an updated task dependency relationship structure.

[0031] In one embodiment, updating the task dependency structure according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure includes:

[0032] When the operation behavior is an update operation behavior, obtaining the incremental task identifier in the incremental task metadata;

[0033] Determine, in the task dependency structure, a task identifier that is identical to the incremental task identifier as an update task identifier;

[0034] Determining an incremental dependency relation formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task;

[0035] The incremental dependency relationship formula is used to replace the dependency relationship formula of the task corresponding to the updated task identifier to obtain an updated task dependency relationship structure.

[0036] In one embodiment, updating the task dependency structure according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure includes:

[0037] When the operation behavior is a deletion operation behavior, obtaining the incremental task identifier in the incremental task metadata;

[0038] Determine, in the task dependency structure, a task identifier that is the same as the incremental task identifier as a deletion task identifier;

[0039] The dependency relationship formula of the task corresponding to the deleted task identifier in the task dependency relationship structure is deleted to obtain an updated task dependency relationship structure.

[0040] According to a second aspect of an embodiment of the present disclosure, there is provided a task processing device, including:

[0041] An input data acquisition unit is configured to execute and acquire an input initial task identifier and a target time interval;

[0042] The current task acquisition unit is configured to execute, from a task dependency structure, acquiring a task corresponding to a task identifier identical to the initial task identifier as the current task; the task dependency structure includes task identifiers, scheduling cycles and task dependencies corresponding to a plurality of tasks;

[0043] a target task acquisition unit, configured to acquire a downstream task corresponding to the current task from the task dependency structure, and use the current task and the downstream task corresponding to the current task as target tasks;

[0044] a task execution data determination unit, configured to determine the task execution data of the target task according to the scheduling period and the target time interval of the target task;

[0045] a data dependency determining unit, configured to determine the data dependency between the task execution data according to the task dependency of the target task;

[0046] The data execution unit is configured to generate a task execution data structure including the task execution data according to the data dependency relationship, and execute the task execution data according to the task execution data structure.

[0047] In one of the embodiments, the task execution data determination unit is further configured to determine at least one task execution time based on the scheduling period of the target task and the target time interval; use the task execution time as the execution time of the target task, and obtain at least one task execution data of the target task.

[0048] In one of the embodiments, the device also includes a task dependency structure generation unit, which is configured to execute the acquisition of task metadata; the task metadata includes a task identifier, a scheduling period and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task; based on the upstream dependency descriptor and the scheduling period of the upstream task, a dependency formula of the task is generated; based on the task identifier corresponding to the task and the dependency formula, the task dependency structure is generated.

[0049] In one embodiment, the upstream dependency descriptor includes an operator and a dependency time, and the task dependency structure generation unit is further configured to execute a bitmap acquisition; the bitmap includes the scheduling period, the operator and the assignment position of the dependency time; according to the conversion relationship for the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship for the operator, the operator is converted into a second result value; according to the conversion relationship for the dependency time, the dependency time is converted into a third result value; according to the assignment position of the scheduling period, the operator and the dependency time, the first result value, the second result value and the third result value are respectively imported into the corresponding positions in the bitmap to obtain the task dependency of the task.

[0050] In one of the embodiments, the device also includes an incremental processing unit, which is configured to execute the acquisition of incremental task metadata and corresponding operation behaviors; based on the operation behaviors and the incremental task metadata, the task dependency structure is updated to obtain an updated task dependency structure.

[0051] In one of the embodiments, the incremental task metadata includes the incremental task identifier, incremental scheduling period and incremental upstream dependency data corresponding to the incremental task; the incremental upstream dependency data includes the incremental upstream task identifier and incremental upstream dependency descriptor of the incremental upstream task corresponding to the incremental task.

[0052] In one of the embodiments, the incremental processing unit is further configured to execute, when the operation behavior is a new operation behavior, determine the dependency formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; add the incremental task identifier corresponding to the incremental task and the dependency formula to the task dependency structure to obtain an updated task dependency structure.

[0053] In one of the embodiments, the incremental processing unit is further configured to execute, when the operation behavior is an update operation behavior, obtaining the incremental task identifier in the incremental task metadata; determining the task identifier in the task dependency structure that is the same as the incremental task identifier as the update task identifier; determining the incremental dependency formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; and replacing the dependency formula of the task corresponding to the update task identifier with the incremental dependency formula to obtain an updated task dependency structure.

[0054] In one of the embodiments, the incremental processing unit is further configured to execute, when the operation behavior is a deletion operation behavior, obtaining the incremental task identifier in the incremental task metadata; determining the task identifier in the task dependency structure that is the same as the incremental task identifier as the deletion task identifier; deleting the dependency formula of the task corresponding to the deletion task identifier in the task dependency structure to obtain an updated task dependency structure.

[0055] According to a third aspect of an embodiment of the present disclosure, a server is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a task processing method as in the implementation of the first aspect.

[0056] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, comprising: when instructions in the storage medium are executed by a processor of a server, the server is enabled to execute a task processing method as in the implementation of the first aspect.

[0057] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a device reading and executing the computer program from the readable storage medium, so that the device performs the task processing method described in any one of the embodiments of the first aspect.

[0058] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0059] Obtain the input initial task identifier and target time interval, and obtain the task corresponding to the task identifier that is the same as the initial task identifier from the task dependency structure as the current task; the task dependency structure includes task identifiers, scheduling cycles and task dependencies corresponding to multiple tasks; then obtain the downstream task corresponding to the current task from the task dependency structure, and use the current task and the downstream task corresponding to the current task as target tasks; determine the task execution data of the target task according to the scheduling cycle and target time interval of the target task; then determine the data dependency between the task execution data according to the task dependency of the target task; finally, generate a task execution data structure including the task execution data according to the data dependency, and execute the task execution data according to the task execution data structure. In this embodiment, a full task dependency structure is maintained, and a task execution data structure of any target time interval can be obtained through the task dependency structure. Since the task dependency structure abstractly describes the dependencies between tasks, it occupies fewer memory resources compared to the task execution data structure that describes the dependencies between task execution data, thereby improving the utilization rate of memory resources. Moreover, each time a task needs to be executed, the corresponding task execution data is executed according to the task execution data structure abstracted based on the task dependency structure, so that the task dependency structure can be reused, and the data utilization rate is high.

[0060] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0062] Figure 1 The present invention is an application environment diagram of a task processing method according to an exemplary embodiment.

[0063] Figure 2 The figure is a flowchart of a task processing method according to an exemplary embodiment.

[0064] Figure 3 The diagram is a schematic diagram showing a task dependency structure according to an exemplary embodiment.

[0065] Figure 4 The figure is a schematic diagram showing a task execution data structure according to an exemplary embodiment.

[0066] Figure 5 The present invention is a flowchart showing a method of constructing a task dependency structure according to an exemplary embodiment.

[0067] Figure 6 The figure is a schematic diagram of task metadata according to an exemplary embodiment.

[0068] Figure 7 It is a schematic diagram showing a dependency relationship according to an exemplary embodiment.

[0069] Figure 8 The present invention is an overall block diagram showing a method for implementing real-time construction of a task execution data structure according to an exemplary embodiment.

[0070] Fig. 9 The figure is an incremental schematic diagram of a task dependency structure according to an exemplary embodiment.

[0071] Fig.10 The invention is a flow chart showing a generation flow of a task execution data structure according to an exemplary embodiment.

[0072] Fig.11 The figure is a block diagram of a task processing device according to an exemplary embodiment.

[0073] Fig.12 The figure is a diagram showing the internal structure of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0074] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0075] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0076] The task processing method provided by the present disclosure can be applied to Figure 1In the application environment shown. Among them, the terminal 110 interacts with the server 120 through the network, and the server 120 obtains the initial task identifier and target time interval uploaded by the terminal 110; from the task dependency structure, the task corresponding to the task identifier that is the same as the initial task identifier is obtained as the current task; the task dependency structure includes task identifiers, scheduling cycles and task dependencies corresponding to multiple tasks; the downstream task corresponding to the current task is obtained from the task dependency structure, and the current task and the downstream task corresponding to the current task are used as the target task; according to the scheduling cycle and target time interval of the target task, the task execution data of the target task is determined; according to the task dependency of the target task, the data dependency between the task execution data is determined; according to the data dependency, a task execution data structure including the task execution data is generated, and the task execution data is executed according to the task execution data structure. Among them, the terminal 110 can be but not limited to various smart phones, tablet computers or laptops, etc., and the server 120 can be implemented by an independent server or a server cluster composed of multiple servers.

[0077] Figure 2 is a flowchart of a task processing method according to an exemplary embodiment. Figure 2 As shown, the task processing method is used in a server and includes the following steps.

[0078] In step S210, the input initial task identifier and target time interval are obtained.

[0079] The initial task identifier refers to the identification information of the first task to be executed, such as the starting task name, the starting task sequence number, etc. In actual scenarios, the initial task identifier may refer to the root node task name.

[0080] The target time interval refers to the time interval that the task needs to be executed; specifically, the target time interval includes a start time (startDate) and an end time (endDate). It should be noted that both the initial task identifier and the target time interval can be uploaded to the server by the user through the terminal.

[0081] Specifically, the server obtains the initial task identifier and target time interval uploaded by the terminal, or obtains the initial task identifier and target time interval from a local database; in this way, by obtaining the initial task identifier and target time interval, it is convenient to subsequently obtain the task corresponding to the task identifier that is the same as the initial task identifier from the task dependency structure as the current task.

[0082] For example, the terminal responds to the user's input operation on the task processing interface, obtains the initial task identifier and target time interval input by the user; generates a task processing request based on the initial task identifier and target time interval input by the user, and sends the task processing request to the corresponding server; the server parses the received task processing request to obtain the initial task identifier and target time interval.

[0083] For example, the user can specify to start executing the task corresponding to the third node in the task dependency structure, and the time interval for executing the task is within the previous three days. In this embodiment, the user can input parameters through the terminal according to actual needs, that is, input the root node task name and the target time interval, and the server will obtain the corresponding task execution data structure according to the root node task name and the target time interval and the task dependency structure.

[0084] In step S212, a task corresponding to a task identifier identical to the initial task identifier is obtained from the task dependency structure as the current task; the task dependency structure includes task identifiers, scheduling cycles and task dependencies corresponding to multiple tasks.

[0085] The task dependency structure includes tasks and the task dependencies between tasks. The task dependencies are used to characterize the relationship between upstream tasks and downstream tasks. Task identification refers to the identification information of tasks, such as task name, task sequence number, etc. The scheduling cycle refers to the execution cycle of tasks, such as the day of the week to execute.

[0086] In actual scenarios, the task dependency structure refers to the task instance lineage model. The task instance lineage model is a model that abstracts the task instance lineage graph and extracts the task instance dependency. This model is called the "task instance lineage model". The task instance lineage model can generate a task instance lineage graph according to any specified routine time (ExecutionDate) or time period. The task instance lineage graph refers to a graph composed of multiple task instance lineages (with dependencies). Task instance lineage refers to a graph that adds dependencies between task instances on the basis of task lineage to make the dependency more detailed and specific. Task lineage refers to the dependency between tasks. This dependency is relatively simple and can reflect a one-to-one relationship, but it cannot express a one-to-many or many-to-one relationship well. For example, if the downstream is a daily task instance and the upstream is a 24-hour task instance, the task lineage cannot be expressed well, so the task instance lineage needs to be used to express this relationship. Among them, the task is the smallest unit entity scheduled for execution in the task scheduling system of the workflow. Tasks support jobs of types such as Hive, MapReduce, and Spark.

[0087] In actual scenarios, the task instance lineage model describes the task dependency between tasks, and the task dependency reflects the dependency between tasks. The task instance lineage model includes the task names, scheduling cycles, and dependency formulas corresponding to multiple tasks. Among them, the task name (taskid) is a unique identifier used to identify the task, and the scheduling cycle refers to the routine scheduling of the task according to certain rules, such as execution at a certain time of day, execution on a certain day of the week, etc. The dependency formula is used to describe the task dependency between tasks, and the task dependency can reflect the one-to-one relationship between tasks. For example, if task A depends on task B, then task B is the upstream task of task A, and task A needs to depend on the execution result of task B.

[0088] For example, refer to Figure 3 The task instance lineage model shown in the figure, where nodes node1, node2, node3, node4, and node5 represent the task entities corresponding to tasks task1, task2, task3, task4, and task5. The task dependency relationship between nodes is reflected by depExp (dependency expression), that is, Figure 3 In depExp1, depExp2, depExp3, depExp4, and depExp5, for example, depExp1 can reflect the task dependency relationship between task1 and task2, depExp2 can reflect the task dependency relationship between task1 and task4, depExp3 can reflect the task dependency relationship between task1 and task3, depExp4 can reflect the task dependency relationship between task2 and task5, and depExp5 can reflect the task dependency relationship between task3 and task5.

[0089] Specifically, the server traverses the task dependency structure according to the initial task identifier, and selects the task corresponding to the initial task identifier from the task dependency structure as the current task. In this way, the current task corresponding to the initial task identifier is first obtained from the task dependency structure, which facilitates the subsequent acquisition of the downstream task corresponding to the current task from the task dependency structure, and the current task and the downstream task corresponding to the current task are used as the target task.

[0090] For example, the initial task identifier is the task identifier corresponding to the first task to be executed, which can be uploaded to the server by the user through the terminal. After the server obtains it, it can obtain the corresponding task from the task dependency structure according to the initial task identifier as the current task.

[0091] In step S214, the downstream task corresponding to the current task is obtained from the task dependency structure, and the current task and the downstream task corresponding to the current task are used as target tasks.

[0092] Specifically, the server obtains a preset downstream task query instruction, and according to the preset downstream task query instruction, queries all downstream tasks corresponding to the current task from the task dependency structure; and takes the current task and all downstream tasks corresponding to the current task as the target task. In this way, by determining the target task, it is helpful to determine the task execution data of the target task according to the scheduling cycle and target time interval of the target task.

[0093] For example, first find the task name that is consistent with the input root node task name from the task instance lineage model, and then use the task corresponding to the task name as the current task. According to the above, there is a task dependency relationship between tasks. After determining the first task, that is, the current task, the downstream task corresponding to the current task will be obtained and used together as the target task of this embodiment.

[0094] In step S216 , task execution data of the target task is determined according to the scheduling period and the target time interval of the target task.

[0095] Among them, task execution data refers to task instances, which are scheduling entities for executing tasks at specified routine times, and routine time (ExecutionDate) refers to a specific moment in the corresponding scheduling cycle. In actual scenarios, one or more task instances corresponding to the target task can be obtained according to the scheduling cycle and the target time interval.

[0096] In step S218, the data dependency relationship between the task execution data is determined according to the task dependency relationship of the target task.

[0097] Among them, task dependency is used to characterize the relationship between upstream tasks and downstream tasks, which can be expressed by a dependency formula; data dependency is used to characterize the relationship between task execution data, which is the relationship between upstream task execution data and downstream task execution data. In actual scenarios, data dependency refers to instance dependency, which is used to characterize the relationship between task instances, which is the relationship between upstream task instances and downstream task instances. After obtaining the task instance corresponding to the target task, the instance dependency between task instances can be derived based on the dependency expression.

[0098] Specifically, the server obtains preset data dependency building instructions, and generates data dependencies between task execution data based on the preset data dependency building instructions, based on the task dependencies between each target task and the task execution data corresponding to each target task; in this way, by determining the data dependencies between the task execution data, it is beneficial to subsequently generate a task execution data structure including the task execution data according to the data dependencies, and execute the task execution data according to the task execution data structure.

[0099] For example, suppose that task instance 1 is obtained according to target task 1, and task instance 2 and task instance 3 are obtained according to target task 2, where there is a task dependency relationship in which target task 1 depends on target task 2, then it can be obtained that task instance 1 depends on task instance 2 and task instance 3. It can be seen that the instance dependency relationship between task instances is generated on the basis of the task dependency relationship between tasks, and the instance dependency relationship is more detailed and specific, which can reflect the dependency relationship between one-to-many or many-to-one task instances.

[0100] In step S220, a task execution data structure including the task execution data is generated according to the data dependency, and the task execution data is executed according to the task execution data structure.

[0101] Among them, the task execution data structure refers to the data structure of the task execution data and the data dependency relationship of the task execution data, which can specifically refer to the task instance lineage graph, which is a DAG (Directed acyclic graph), a directed graph without a cycle. Since the data dependency relationship represents the dependency relationship between the task execution data, the task execution data can be combined according to the data dependency relationship to obtain a task execution data structure that describes the data dependency relationship between all the task execution data. When executing a task, the task execution data will be executed according to the task execution data structure to complete the execution of the corresponding task.

[0102] Specifically, the server obtains a preset task execution data structure construction instruction, and constructs a corresponding task execution data structure based on the preset task execution data structure construction instruction and the task execution data and the data dependency relationship between the task execution data. When executing a task, the generated task execution data structure is called to execute the corresponding task execution data to complete the task. In this way, each time a task needs to be executed, the corresponding task execution data is executed according to the task execution data structure abstracted based on the task dependency structure, so that the task dependency structure can be reused and the data utilization rate is high.

[0103] For example, see Figure 4Each node in the task instance lineage graph represents a combination of task instance and executionDate (routine time). For example, node1_1, node1_2, and node1_3 in the graph are task instances of task1 corresponding to different routine times, and node2 is a task instance of task1 corresponding to a routine time. Node2 depends on node1. Therefore, in the actual execution scheduling process, node2 will only be scheduled after the task instances corresponding to node1_1, node1_2, and node1_3 are successfully scheduled.

[0104] In the above-mentioned task processing method, the input initial task identifier and target time interval are obtained, and the task corresponding to the task identifier that is the same as the initial task identifier is obtained from the task dependency structure as the current task; the task dependency structure includes task identifiers, scheduling cycles and task dependencies corresponding to multiple tasks; then the downstream task corresponding to the current task is obtained from the task dependency structure, and the current task and the downstream task corresponding to the current task are used as target tasks; according to the scheduling cycle and target time interval of the target task, the task execution data of the target task is determined; then according to the task dependency of the target task, the data dependency between the task execution data is determined; finally, according to the data dependency, a task execution data structure including the task execution data is generated, and the task execution data is executed according to the task execution data structure. In this embodiment, a full task dependency structure is maintained, and a task execution data structure of any target time interval can be obtained through the task dependency structure. Since the task dependency structure abstractly describes the dependencies between tasks, it occupies less memory resources than the task execution data structure that describes the dependencies between task execution data, thereby saving server resources and further improving server resource utilization. Each time a task needs to be executed, the corresponding task execution data is executed according to the task execution data structure abstracted based on the task dependency structure, so that the task dependency structure can be reused and the data utilization rate is high.

[0105] As an optional implementation of the present disclosure, in step S216, task execution data of the target task is determined based on the scheduling period and target time interval of the target task, specifically including the following contents: determining at least one task execution time based on the scheduling period and target time interval of the target task; taking the task execution time as the execution time of the target task, and obtaining at least one task execution data of the target task.

[0106] The task execution time refers to the specific time when the task is executed, specifically the routine time.

[0107] Specifically, the server can determine one or more task execution times based on the scheduling period of the target task and the input target time interval. After obtaining the task execution time, the task execution time can be used as the execution time of the target task, thereby obtaining at least one task execution data of the target task.

[0108] For example, assuming that the input target time interval is the previous three days, if the scheduling period of the target task is 4 o'clock every day, three routine times can be obtained. By combining the three routine times with the target task, three task instances can be obtained.

[0109] The technical solution provided by the embodiment of the present disclosure generates task execution data of the target task within the target time interval according to the scheduling period and target time interval of the target task, which is beneficial to improving the accuracy of the task execution data of the determined target task, thereby ensuring the success rate of subsequent task executions, and at the same time avoiding the defect of missing the task execution data of the target task, resulting in task execution failure and the need to re-execute the task, which leads to extended task processing time.

[0110] As an optional embodiment of the present disclosure, refer to Figure 5 The above step S212, before obtaining the task corresponding to the task identifier identical to the initial task identifier from the task dependency structure as the current task, further specifically includes the following steps:

[0111] In step S510, task metadata is obtained; the task metadata includes a task identifier, a scheduling period, and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task.

[0112] In step S512, a task dependency equation is generated according to the upstream dependency descriptor and the scheduling period of the upstream task.

[0113] In step S514, a task dependency structure is generated according to the task identifiers and dependency formulas corresponding to the tasks.

[0114] Among them, the task metadata is the basic content of the task and is the data source of the entire task dependency structure (such as the task instance lineage model) of this embodiment. Figure 6The task metadata may include the task name (taskId), the task type (taskType, such as hive task or spark task, etc.), the task scheduling period (cronExp, which is a cron rule expression, such as specifying what time to run every day / week), the task creation time (createTime) and the task's upstream dependency data (upstreams). The upstream dependency data includes the upstream task name and dependency descriptor of the upstream task. The dependency descriptor can be empty, which means that the upstream and downstream tasks are in the same period.

[0115] Among them, the task dependency descriptor format is:

[0116] {"op":"[or|empty|and|max|min]","value":"[p_date-m|p_date|p_date-m,p_date-n]"}.

[0117] Specifically, the dependency descriptor is divided into two parts:

[0118] (1) op (operator): The default value is "empty". Like "and", all conditions must be met. "or" means at least one condition must be met. "max" gets the maximum value in the interval. "min" gets the minimum value in the interval.

[0119] (2) value (time-dependent): corresponds to time or time interval.

[0120] Assuming that both upstream and downstream tasks are day-level tasks, the dependency descriptor of the downstream task on the upstream task can be:

[0121] {"op":"and","value":"p_date-n"} and {"op":"","value":"p_date-n"} are equivalent: they depend on the specified dependency date.

[0122] {"op":"and","value":"p_date-m,p_date-n"}(m>n): depends on all dependent dates within the specified interval.

[0123] {"op":"or","value":"p_date-m,p_date-n"}(m>n): Depends on any dependent time within the specified interval;

[0124] {"op":"max","value":"p_date-m,p_date-n"}(m>n): The maximum (ready) dependency time that meets the conditions in the specified interval.

[0125] {"op":"max","value":"p_date-n"}(<=p_date-n): the maximum dependency time that meets the dependency condition.

[0126] Specifically, the dependency formula between tasks can be derived based on the dependency descriptors and scheduling cycles corresponding to the tasks, and after obtaining the task dependency formula, the task dependency formula and task identifier can be saved together in the task dependency structure template according to the task dependency, thereby obtaining the task dependency structure.

[0127] The technical solution provided by the embodiment of the present disclosure, by acquiring the task dependency structure, is conducive to subsequently obtaining the task corresponding to the task identifier identical to the initial task identifier (i.e., the current task) and the downstream task corresponding to the task through the task dependency structure; only acquiring the tasks and downstream tasks of interest is conducive to improving the flexibility of task processing and making effective use of server resources.

[0128] As an optional implementation of the present disclosure, the upstream dependency descriptor includes an operator and a dependency time. In step S512, a dependency formula of the task is generated according to the upstream dependency descriptor and the scheduling period of the upstream task, which may specifically include the following contents: obtaining a bitmap; the bitmap includes the assignment positions of the scheduling period, the operator and the dependency time; according to the conversion relationship for the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship for the operator, the operator is converted into a second result value; according to the conversion relationship for the dependency time, the dependency time is converted into a third result value; according to the assignment positions of the scheduling period, the operator and the dependency time, the first result value, the second result value and the third result value are respectively imported into the corresponding positions in the bitmap to obtain the task dependency relationship of the task.

[0129] In this embodiment, the dependency descriptor can be converted into a dependency expression. The dependency expression is a simplified expression of the dependency descriptor. A bitmap (corresponding to a long integer long, occupying 8 bytes) can be used to express complex task dependencies. The use of bitmap greatly saves storage space and reduces memory usage. At the same time, bitmap bit processing is efficient and fast.

[0130] The bitmap sets corresponding assignment positions for the scheduling cycle, operator and dependency time respectively. After converting the scheduling cycle, operator and dependency time according to the corresponding conversion relationships, the corresponding first result value, second result value and third result value can be obtained; the first result value, the second result value and the third result value are imported into the corresponding assignment positions in turn, and the dependency relationship formula of this embodiment can be obtained.

[0131] For example, see Figure 7 , the meaning of the assignment corresponding to each bit in the 8-byte (64-bit) dependency expression (i.e., dependency expression) is:

[0132] 1. According to the scheduling period of the upstream task, the calculated result value is assigned to the 0-1 position of the dependency expression.

[0133] 2. According to the second part of the dependency descriptor (value, dependency time), the calculated result value is assigned to bits 2-60 and 63 of the dependency expression.

[0134] 3. According to the first part of the dependency descriptor (op, operator), the calculated result data is assigned to bits 61-62 of the dependency expression.

[0135] Among them, the 63rd bit of the dependency expression indicates whether it depends on the past task or the future task, 0 indicates dependence on the past, and 1 indicates dependence on the future.

[0136] Specifically, the dependency descriptor conversion bit dependency expression can be the following examples:

[0137] Example 1: The dependency descriptor "{p_date-0}" can be derived into a dependency expression: 000...000 (64 zeros), and the corresponding integer value is 0, indicating that the two tasks are executed in the same cycle.

[0138] Example 2: The dependency descriptor "{p_date-1}" can be used to derive the dependency expression: 000...0100, with a corresponding integer value of 4, indicating that the downstream task depends on the previous scheduling cycle of the upstream task.

[0139] Example 3: The dependency descriptor "{p_date-4, p_date-1}" can derive the dependency expression: 00...0111100, and the corresponding integer value is 60, indicating that the downstream task depends on the first four consecutive scheduling cycles of the upstream task.

[0140] The technical solution provided by the embodiment of the present disclosure expresses complex task dependencies through a simple dependency formula, which is conducive to saving storage space, thereby reducing server memory usage, thereby making server processing calculations more efficient and faster, and further improving the processing efficiency of subsequent tasks; at the same time, by generating a dependency formula for the task, it is beneficial to subsequently combine the task identifiers corresponding to the tasks to generate a task dependency structure.

[0141] As an optional implementation of the present disclosure, the above step S514, after generating the task dependency structure according to the task identifier and dependency formula corresponding to the task, also includes the following contents: obtaining incremental task metadata and corresponding operation behaviors; updating the task dependency structure according to the operation behaviors and incremental task metadata to obtain an updated task dependency structure.

[0142] Among them, the incremental task metadata is the task metadata input by the user through the terminal, which is used to update the task dependency structure (such as the task instance lineage model). Its corresponding operation behaviors may include operation types such as addition, deletion, and update. The task dependency structure is updated in real time according to the incremental task metadata, which can correspondingly improve the real-time performance of the obtained task dependency structure.

[0143] Among them, the incremental task metadata includes the incremental task identifier (such as the incremental task name), incremental scheduling period and incremental upstream dependency data corresponding to the incremental task; the incremental upstream dependency data includes the incremental upstream task name and incremental upstream dependency descriptor of the incremental upstream task corresponding to the incremental task.

[0144] Specifically, the server obtains the incremental task metadata and corresponding operation behaviors uploaded by the terminal; according to the operation behaviors and based on the incremental task metadata, the server performs corresponding update processing on the historical task dependency structure to obtain an updated task dependency structure.

[0145] For example, the terminal responds to the user's operation behavior on the task processing interface and obtains the incremental task metadata input by the user; generates a task dependency structure update request based on the incremental task metadata input by the user and the triggered operation behavior, and sends the task dependency structure update request to the corresponding server; the server parses the received task dependency structure update request to obtain the incremental task metadata and the triggered operation behavior, and according to the triggered operation behavior and based on the incremental task metadata, performs corresponding update processing on the historical task dependency structure to obtain an updated task dependency structure.

[0146] The technical solution provided by the disclosed embodiment performs corresponding updates on the historical task dependency structure based on the incremental task metadata according to the triggered operation behavior, thereby ensuring the timeliness of the task dependency structure, and facilitating improving the accuracy of the subsequent task execution data structure obtained based on the task dependency structure, avoiding the use of the old task dependency structure, which may lead to defects in task processing failure, and thereby improving the success rate of task execution.

[0147] As an optional implementation mode of the present disclosure, the task dependency structure is updated according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure, which specifically includes the following contents: when the operation behavior is a new operation behavior, the dependency formula of the incremental task is determined according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; the incremental task identifier and the dependency formula corresponding to the incremental task are added to the task dependency structure to obtain an updated task dependency structure.

[0148] Specifically, when the operation behavior is a new operation behavior, the dependency formula of the incremental task is generated according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task, and the incremental task identifier and dependency formula are added at the corresponding position in the task dependency structure to obtain an updated task dependency structure.

[0149] For example, the terminal responds to the user's new operation behavior on the task processing interface and obtains the incremental task metadata input by the user; generates a task dependency structure update request based on the incremental task metadata input by the user and the triggered new operation behavior, and sends the task dependency structure update request to the corresponding server; the server parses the received task dependency structure update request to obtain the incremental task metadata and the triggered new operation behavior, and extracts the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task from the incremental task metadata; calls the dependency formula generation instruction to generate the dependency formula of the incremental task based on the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; obtains the incremental task identifier of the incremental task and the dependency formula of the incremental task in the task dependency structure. The position label, according to the position label, adds the incremental task identifier and the dependency formula of the incremental task to the corresponding position in the task dependency structure to obtain an updated task dependency structure.

[0150] The technical solution provided by the embodiment of the present disclosure, when the operation behavior is a new operation behavior, performs corresponding update processing on the historical task dependency structure based on the incremental task metadata, thereby ensuring the timeliness of the task dependency structure; at the same time, by updating the task dependency structure in real time, it is beneficial to improve the accuracy of the subsequent task execution data structure obtained based on the task dependency structure.

[0151] As an optional implementation mode of the present disclosure, the task dependency structure is updated according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure, which specifically includes the following contents: when the operation behavior is an update operation behavior, the incremental task identifier in the incremental task metadata is obtained; the task identifier in the task dependency structure that is the same as the incremental task identifier is determined as the update task identifier; the incremental dependency formula of the incremental task is determined according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; the dependency formula of the task corresponding to the update task identifier is replaced with the incremental dependency formula to obtain an updated task dependency structure.

[0152] Specifically, when the operation behavior is an update operation behavior, the incremental task identifier is first obtained from the incremental task metadata, and then the task identifier consistent with the incremental task identifier in the task dependency structure is determined as the update task identifier, and the incremental dependency formula of the incremental task is generated according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task. The dependency formula corresponding to the original update task identifier in the task dependency structure is deleted and replaced with the incremental dependency formula, thereby obtaining an updated task dependency structure.

[0153] For example, the terminal responds to the user's update operation behavior on the task processing interface and obtains the incremental task metadata input by the user; generates a task dependency structure update request based on the incremental task metadata input by the user and the triggered update operation behavior, and sends the task dependency structure update request to the corresponding server; the server parses the received task dependency structure update request to obtain the incremental task metadata and the triggered update operation behavior, extracts the incremental task identifier, the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task from the incremental task metadata; determines the task identifier corresponding to the incremental task identifier in the task dependency structure as the update task identifier; calls the dependency formula generation instruction to generate the incremental dependency formula of the incremental task based on the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; constructs a replacement relationship between the incremental dependency formula and the dependency formula, and according to the replacement relationship, replaces the dependency formula of the update task identifier in the task dependency structure with the incremental dependency formula to obtain an updated task dependency structure.

[0154] The technical solution provided by the embodiment of the present disclosure, when the operation behavior is an update operation behavior, performs corresponding update processing on the historical task dependency structure based on the incremental task metadata, thereby ensuring the timeliness of the task dependency structure; at the same time, it avoids the defect of using the old task dependency structure, which leads to task processing failure, thereby improving the success rate of task execution.

[0155] As an optional implementation of the present disclosure, the task dependency structure is updated according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure, which specifically includes the following contents: when the operation behavior is a deletion operation behavior, the incremental task identifier in the incremental task metadata is obtained; the task identifier in the task dependency structure that is the same as the incremental task identifier is determined as the deletion task identifier; the dependency formula of the task corresponding to the deletion task identifier in the task dependency structure is deleted to obtain an updated task dependency structure.

[0156] Specifically, when the operation behavior is a deletion operation behavior, the incremental task identifier is first obtained from the incremental task metadata, and then the task identifier that is consistent with the incremental task identifier in the task dependency structure is determined as the deletion task identifier. Finally, the dependency formula corresponding to the deletion task identifier in the task dependency structure is deleted to obtain an updated task dependency structure.

[0157] For example, the terminal responds to the user's deletion operation on the task processing interface and obtains the incremental task metadata input by the user; generates a task dependency structure update request based on the incremental task metadata input by the user and the triggered deletion operation, and sends the task dependency structure update request to the corresponding server; the server parses the received task dependency structure update request to obtain the incremental task metadata and the triggered deletion operation, and extracts the incremental task identifier from the incremental task metadata based on the deletion operation; determines the task identifier corresponding to the incremental task identifier in the task dependency structure as the deletion task identifier; deletes the dependency formula of the task corresponding to the deletion task identifier in the task dependency structure to obtain an updated task dependency structure.

[0158] The technical solution provided by the disclosed embodiment updates the task dependency structure in real time so that the subsequently constructed task execution data structure is also the latest, thereby solving the problem that the current task execution data structure update lacks real-time performance. At the same time, through the updated task dependency structure, the task execution data structure at any time can be abstracted to execute the corresponding task execution data, so that the task dependency structure can be reused and the data utilization rate is high.

[0159] In order to enable those skilled in the art to better understand the implementation process of constructing the task instance lineage model graph in real time in this embodiment, a specific example is used for illustration below. In the following example, the task instance lineage model graph is referred to as the lineage model graph, and the task instance lineage graph is referred to as the lineage graph. Figure 8, is a block diagram of the overall solution of a lineage model diagram of this embodiment, wherein the Workflow task table is a basic table of the workflow scheduling system, mainly including some basic information of the task (task metadata), Figure 8 The whole consists of three parts:

[0160] 1. Batch offline synchronization: Perform unified metadata processing on the acquired Workflow task table, pull task metadata in batches and sections, and store them in the Task metadata table. Specifically, batch offline synchronization can be implemented through Java code, with the aim of achieving the following: 1) In order to avoid a large number of read operations on the database in a short period of time, the batch processing is divided into several sections according to the time interval and processed in batches; 2) According to the time period, filter according to the creation time in the Workflow task table to filter out the Task list; 3) gradually perform task metadata processing on the filtered task list (because the current workflow task table includes two scheduling systems, airflow (open source scheduling system) and kwafilow (self-developed scheduling system), unified task metadata processing is required to align task types, align scheduling cycles, etc., to facilitate subsequent unified processing. Among them, batch offline synchronization only needs to be processed once in the early stage.

[0161] 2. Incremental real-time synchronization: For operation behaviors involving tasks in the scheduling system (operation types include addition, update, and deletion), they will be synchronized, parsed, and uniformly processed through MYSQL binlog in real time, and finally the incremental results (incremental task metadata) will be synchronized to the Kafka platform for real-time consumption by the task instance lineage model system.

[0162] 3. Task instance lineage model system: The task instance lineage model system initially obtains the task metadata from the task metadata table for initialization, performs model generation processing, and obtains the task instance lineage model graph. Subsequently, the task instance lineage model system can consume the incremental task metadata of the Kafka platform in real time, and distribute and process it according to the operation type of the task, so as to update the task instance lineage model graph in real time according to the operation type and the corresponding incremental task metadata. In this way, the user of this embodiment can use the latest task instance lineage model graph to obtain the task root node specified by the user (the user selects the task he cares about) and the task instance lineage graph of the target time interval.

[0163] Among them, refer to Fig. 9 First, create an object to save the bloodline model graph. The object type is a graph structure.<Task,Long> ,The updating process of the task instance lineage model graph is as follows:

[0164] 1. Try to pull tasks (incremental task metadata) and operation behaviors (default new behaviors);

[0165] 2. Determine whether the task is pulled; if not, return to step 1; if so, go to step 3;

[0166] 3. Get the task and operation behavior, and perform locking processing;

[0167] 4. Forward according to the operation behavior. If it is a newly added behavior, execute step 5; if it is a newly added behavior, execute step 7; if it is a newly added behavior, execute step 5;

[0168] 5. Generate dependency expressions based on the task's dependency descriptor;

[0169] 6. Add the dependency expressions between nodes (edges) to the blood relationship model graph, and release the lock to execute step 11;

[0170] 7. Obtain the existing dependency relationship from the bloodline model graph according to the taskId corresponding to the task;

[0171] 8. Delete the existing dependency from the bloodline model diagram and execute step 5;

[0172] 9. Obtain the existing dependencies from the bloodline model graph based on taskId;

[0173] 10. Delete the old dependency from the bloodline model graph, release the lock and execute step 11;

[0174] 11. Get the bloodline model diagram.

[0175] Among them, refer to Fig.10 ,The process of generating a blood relationship graph based on the blood relationship model graph is as follows:

[0176] 1. After obtaining the input root node taskid, startDate (start time), and endDate (end time), initialize the current set and the next set;

[0177] 2. Determine whether the root node taskid exists in the lineage model graph; if so, execute step 3; if not, obtain the lineage graph;

[0178] 3. Store the root node taskid in the current set;

[0179] 4. Determine whether the current set is empty; if so, obtain the blood relationship graph; if not, execute step 5;

[0180] 5. Judge i < current.size; where i represents the index corresponding to each taskid in the current set, initially assigned i = 0, with the index incremented each time, and current.size represents the maximum index of the current set; if i < current.size, it means the traversal of the current set has not been completed, then execute step 6, if i ≥ current.size, it means the traversal of the current set is completed, then after clearing the current set (current.clear()), obtaining the next root node taskid and storing it in the current set (current.addAll(next)), and clearing the next set (next.clear()), return to step 4;

[0181] 6. e = current.get(i), that is, obtain the directly downstream nodes downNodes of the current node e and store downNodes in the next set;

[0182] 7. Judge j < downNodes.size; where j represents the index corresponding to each taskid of downNodes in the next set, initially assigned j = 0, with the index incremented each time, and downNodes.size represents the maximum index of the next set; if j < downNodes.size, it means the traversal of the next set has not been completed, then execute step 8;

[0183] 8. d = downNodes.get(j), that is, obtain the downstream node d in the next set;

[0184] 9. Generate a task instance list of the downstream node d based on the start time and end time input by the user and the scheduling period of the downstream node d;

[0185] 10. Generate a task instance list of the node e based on the start time and end time input by the user and the scheduling period of the node e;

[0186] 11. Generate an instance dependency relationship (d_i, e_i) between the task instances of the node d and the node e according to the dependency expression of d -> e;

[0187] 12. Save the task instance dependency relationship between the task instances into the lineage graph.

[0188] The application of this embodiment can have at least the following advantages: Real-time: The update delay of the task instance lineage graph is reduced from the original 15 minutes to the current 1 minute; Memory usage: By using bitmap to store dependencies, the memory occupied by the lineage model graph is very small, and the overall memory usage is reduced from the original 16G to the current 4G (which can well meet the generation of the task instance lineage graph, and too low memory is not conducive to high concurrency). Database storage: The storage scale has been reduced from the original million-level tasks to the current 100,000-level, and the storage delay has been reduced from 5 minutes to less than 1 minute. It supports the acquisition of the lineage graph of task instances at any routine time, so it better supports the re-running system to re-run historical routine times, supports re-running at the task instance level, refines the granularity of re-running, and improves the efficiency of re-running.

[0189] It should be understood that although Figure 2 and Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 and Figure 5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0190] Fig.11 is a block diagram of a task processing device according to an exemplary embodiment. Fig.11 The device includes an input data acquisition unit 1110, a current task acquisition unit 1112, a target task acquisition unit 1114, a task execution data determination unit 1116, a data dependency determination unit 1118 and a data execution unit 1120.

[0191] An input data acquisition unit 1110 is configured to acquire an input initial task identifier and a target time interval;

[0192] The current task acquisition unit 1112 is configured to acquire a task corresponding to a task identifier identical to the initial task identifier from a task dependency structure as the current task; the task dependency structure includes task identifiers, scheduling cycles, and task dependencies corresponding to multiple tasks;

[0193] The target task acquisition unit 1114 is configured to acquire the downstream task corresponding to the current task from the task dependency structure, and take the current task and the downstream task corresponding to the current task as the target task;

[0194] The task execution data determining unit 1116 is configured to determine the task execution data of the target task according to the scheduling period and the target time interval of the target task;

[0195] The data dependency determining unit 1118 is configured to determine the data dependency between the task execution data according to the task dependency of the target task;

[0196] The data execution unit 1120 is configured to generate a task execution data structure including the task execution data according to the data dependency relationship, and execute the task execution data according to the task execution data structure.

[0197] In one of the embodiments, the task execution data determination unit 1116 is further configured to determine at least one task execution time according to the scheduling period and target time interval of the target task; use the task execution time as the execution time of the target task, and obtain at least one task execution data of the target task.

[0198] In one of the embodiments, the task processing device also includes a task dependency structure generation unit, which is configured to execute the acquisition of task metadata; the task metadata includes a task identifier, a scheduling period and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task; a dependency formula of the task is generated according to the upstream dependency descriptor and the scheduling period of the upstream task; and a task dependency structure is generated according to the task identifier and the dependency formula corresponding to the task.

[0199] In one of the embodiments, the upstream dependency descriptor includes an operator and a dependency time, and the task dependency structure generation unit is further configured to execute and obtain a bitmap; the bitmap includes the assignment positions of the scheduling period, the operator and the dependency time; according to the conversion relationship for the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship for the operator, the operator is converted into a second result value; according to the conversion relationship for the dependency time, the dependency time is converted into a third result value; according to the assignment positions of the scheduling period, the operator and the dependency time, the first result value, the second result value and the third result value are respectively imported into the corresponding positions in the bitmap to obtain the task dependency of the task.

[0200] In one of the embodiments, the task processing device also includes an incremental processing unit, which is configured to execute the acquisition of incremental task metadata and corresponding operation behaviors; based on the operation behaviors and incremental task metadata, the task dependency structure is updated to obtain an updated task dependency structure.

[0201] In one of the embodiments, the incremental task metadata includes the incremental task identifier, incremental scheduling period and incremental upstream dependency data corresponding to the incremental task; the incremental upstream dependency data includes the incremental upstream task identifier and incremental upstream dependency descriptor of the incremental upstream task corresponding to the incremental task.

[0202] In one of the embodiments, the incremental processing unit is further configured to determine the dependency formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task when the operation behavior is a new operation behavior; add the incremental task identifier and dependency formula corresponding to the incremental task to the task dependency structure to obtain an updated task dependency structure.

[0203] In one of the embodiments, the incremental processing unit is further configured to execute, when the operation behavior is an update operation behavior, obtaining the incremental task identifier in the incremental task metadata; determining the task identifier that is the same as the incremental task identifier in the task dependency structure as the update task identifier; determining the incremental dependency formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; and replacing the dependency formula of the task corresponding to the update task identifier with the incremental dependency formula to obtain an updated task dependency structure.

[0204] In one of the embodiments, the incremental processing unit is further configured to execute, when the operation behavior is a deletion operation behavior, obtaining the incremental task identifier in the incremental task metadata; determining the task identifier in the task dependency structure that is the same as the incremental task identifier as the deletion task identifier; deleting the dependency formula of the task corresponding to the deletion task identifier in the task dependency structure to obtain an updated task dependency structure.

[0205] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0206] Fig.12 1 is a block diagram of a device 1200 for executing the above-mentioned task processing method according to an exemplary embodiment. For example, the device 1200 may be a server. Fig.12, the device 1200 includes a processing component 1220, which further includes one or more processors, and a memory resource represented by a memory 1222 for storing instructions that can be executed by the processing component 1220, such as an application. The application stored in the memory 1222 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1220 is configured to execute instructions to perform the above-mentioned task processing method.

[0207] The device 1200 may also include a power component 1224 configured to perform power management of the device 1200, a wired or wireless network interface 1226 configured to connect the device 1200 to a network, and an input / output (I / O) interface 1228. The device 1200 may operate based on an operating system stored in the memory 1222, such as Windows Server, Mac OSX, Unix, Linux, FreeBSD, or the like.

[0208] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 1122 including instructions, which can be executed by a processor of the device 1100 to perform the above method. The storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0209] In an exemplary embodiment, a computer program product is also provided, which includes a computer program stored in a readable storage medium, and at least one processor of a device reads and executes the computer program from the readable storage medium, so that the device performs the task processing method described in any one of the embodiments of the present disclosure.

[0210] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0211] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A task processing method, characterized in that: include: Get the input initial task ID and target time interval; From the task dependency structure, a task corresponding to a task identifier identical to the initial task identifier is obtained as the current task; the task dependency structure includes task identifiers, scheduling periods and task dependencies corresponding to a plurality of tasks; the task dependency of each task is obtained by respectively importing a first result value of the scheduling period of the upstream task corresponding to each task, a second result value of the operator of the upstream task corresponding to each task and a third result value of the dependency time of the upstream task corresponding to each task into the assignment position of the scheduling period of the upstream task, the assignment position of the operator of the upstream task and the assignment position of the dependency time of the upstream task included in the bitmap; Acquire the downstream task corresponding to the current task from the task dependency structure, and use the current task and the downstream task corresponding to the current task as target tasks; Determining task execution data of the target task according to the scheduling period and the target time interval of the target task; Determining a data dependency relationship between the task execution data according to the task dependency relationship of the target task; A task execution data structure including the task execution data is generated according to the data dependency, and the task execution data is executed according to the task execution data structure.

2. The task processing method according to claim 1, characterized in that: The determining the task execution data of the target task according to the scheduling period and the target time interval of the target task includes: Determining at least one task execution time according to the scheduling period of the target task and the target time interval; The task execution time is used as the execution time of the target task to obtain at least one task execution data of the target task.

3. The task processing method according to claim 1, characterized in that: Before obtaining, from the task dependency structure, a task corresponding to a task identifier identical to the initial task identifier as the current task, the method further includes: Acquire task metadata; the task metadata includes a task identifier, a scheduling period, and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task; Generate a task dependency relationship of the task according to the upstream dependency descriptor and the scheduling period of the upstream task; The task dependency structure is generated according to the task identifier corresponding to the task and the task dependency.

4. The task processing method according to claim 3, characterized in that: The upstream dependency descriptor includes an operator and a dependency time, and generating the task dependency relationship of the task according to the upstream dependency descriptor and the scheduling period of the upstream task includes: Obtaining a bitmap; the bitmap includes the scheduling period, the operator and the assignment position of the dependent time; According to the conversion relationship for the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship for the operator, the operator is converted into a second result value; according to the conversion relationship for the dependent time, the dependent time is converted into a third result value; According to the scheduling period, the operator and the assignment position of the dependent time, the first result value, the second result value and the third result value are respectively imported into the corresponding positions in the bitmap to obtain the task dependency relationship of the task.

5. The task processing method according to claim 3, characterized in that: After generating the task dependency structure according to the task identifier corresponding to the task and the task dependency, the method further includes: Get incremental task metadata and corresponding operation behaviors; The task dependency structure is updated according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure.

6. The task processing method according to claim 5, characterized in that: The incremental task metadata includes the incremental task identifier, incremental scheduling period and incremental upstream dependency data corresponding to the incremental task; the incremental upstream dependency data includes the incremental upstream task identifier and incremental upstream dependency descriptor of the incremental upstream task corresponding to the incremental task.

7. The task processing method according to claim 6, characterized in that: The updating of the task dependency structure according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure includes: When the operation behavior is a newly added operation behavior, determining a dependency relationship formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; The incremental task identifier corresponding to the incremental task and the dependency relationship formula are added to the task dependency relationship structure to obtain an updated task dependency relationship structure.

8. The task processing method according to claim 6, characterized in that: The updating of the task dependency structure according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure includes: When the operation behavior is an update operation behavior, obtaining the incremental task identifier in the incremental task metadata; Determine, in the task dependency structure, a task identifier that is identical to the incremental task identifier as an update task identifier; Determining an incremental dependency relation formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; The incremental dependency relationship formula is used to replace the dependency relationship formula of the task corresponding to the updated task identifier to obtain an updated task dependency relationship structure.

9. The task processing method according to claim 6, characterized in that: The updating of the task dependency structure according to the operation behavior and the incremental task metadata to obtain an updated task dependency structure includes: When the operation behavior is a deletion operation behavior, obtaining the incremental task identifier in the incremental task metadata; Determine, in the task dependency structure, a task identifier that is the same as the incremental task identifier as a deletion task identifier; The dependency relationship formula of the task corresponding to the deleted task identifier in the task dependency relationship structure is deleted to obtain an updated task dependency relationship structure.

10. A task processing device, characterized in that: include: An input data acquisition unit is configured to execute and acquire an input initial task identifier and a target time interval; The current task acquisition unit is configured to execute, from a task dependency structure, acquiring a task corresponding to a task identifier identical to the initial task identifier as the current task; the task dependency structure includes task identifiers, scheduling periods and task dependencies corresponding to a plurality of tasks; the task dependency of each task is obtained by respectively importing a first result value of the scheduling period of an upstream task corresponding to each task, a second result value of an operator of the upstream task corresponding to each task and a third result value of a dependency time of the upstream task corresponding to each task into an assignment position of the scheduling period of the upstream task, an assignment position of the operator of the upstream task and an assignment position of the dependency time of the upstream task included in a bitmap; a target task acquisition unit, configured to acquire a downstream task corresponding to the current task from the task dependency structure, and use the current task and the downstream task corresponding to the current task as target tasks; a task execution data determination unit, configured to determine the task execution data of the target task according to the scheduling period and the target time interval of the target task; a data dependency determining unit, configured to determine the data dependency between the task execution data according to the task dependency of the target task; The data execution unit is configured to generate a task execution data structure including the task execution data according to the data dependency relationship, and execute the task execution data according to the task execution data structure.

11. The task processing device according to claim 10, characterized in that: The task execution data determination unit is also configured to determine at least one task execution time according to the scheduling period of the target task and the target time interval; use the task execution time as the execution time of the target task to obtain at least one task execution data of the target task.

12. The task processing device according to claim 10, characterized in that: The device also includes a task dependency structure generating unit, which is configured to execute the acquisition of task metadata; the task metadata includes a task identifier, a scheduling period and upstream dependency data corresponding to the task; the upstream dependency data includes an upstream task identifier and an upstream dependency descriptor of an upstream task corresponding to the task; the task dependency of the task is generated according to the upstream dependency descriptor and the scheduling period of the upstream task; and the task dependency structure is generated according to the task identifier corresponding to the task and the task dependency.

13. The task processing device according to claim 12, characterized in that: The upstream dependency descriptor includes an operator and a dependency time, and the task dependency structure generation unit is further configured to execute an acquisition bitmap; The bitmap includes the scheduling period, the operator and the assignment position of the dependent time; according to the conversion relationship for the scheduling period, the scheduling period of the upstream task is converted into a first result value; according to the conversion relationship for the operator, the operator is converted into a second result value; according to the conversion relationship for the dependent time, the dependent time is converted into a third result value; according to the assignment position of the scheduling period, the operator and the dependent time, the first result value, the second result value and the third result value are respectively imported into the corresponding positions in the bitmap to obtain the task dependency relationship of the task.

14. The task processing device according to claim 12, characterized in that: The device also includes an incremental processing unit configured to execute and obtain incremental task metadata and corresponding operation behaviors; update the task dependency structure according to the operation behaviors and the incremental task metadata to obtain an updated task dependency structure.

15. The task processing device according to claim 14, characterized in that: The incremental task metadata includes the incremental task identifier, incremental scheduling period and incremental upstream dependency data corresponding to the incremental task; the incremental upstream dependency data includes the incremental upstream task identifier and incremental upstream dependency descriptor of the incremental upstream task corresponding to the incremental task.

16. The task processing device according to claim 15, characterized in that: The incremental processing unit is also configured to execute, when the operation behavior is a new operation behavior, determine the dependency formula of the incremental task according to the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; add the incremental task identifier corresponding to the incremental task and the dependency formula to the task dependency structure to obtain an updated task dependency structure.

17. The task processing device according to claim 15, characterized in that: The incremental processing unit is further configured to obtain the incremental task identifier in the incremental task metadata when the operation behavior is an update operation behavior; Determine in the task dependency structure, a task identifier that is identical to the incremental task identifier as an updated task identifier; determine the incremental dependency formula of the incremental task based on the incremental upstream dependency descriptor and the incremental scheduling period of the incremental upstream task; and use the incremental dependency formula to replace the dependency formula of the task corresponding to the updated task identifier to obtain an updated task dependency structure.

18. The task processing device according to claim 15, characterized in that: The incremental processing unit is further configured to obtain the incremental task identifier in the incremental task metadata when the operation behavior is a deletion operation behavior; Determine in the task dependency structure, a task identifier that is the same as the incremental task identifier as a deletion task identifier; delete the dependency formula of the task corresponding to the deletion task identifier in the task dependency structure to obtain an updated task dependency structure.

19. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the task processing method according to any one of claims 1 to 9.

20. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of the server, the server is enabled to execute the task processing method as described in any one of claims 1 to 9.

21. A computer program product, the program product comprising a computer program, characterized in that The computer program is stored in a readable storage medium, and at least one processor of the device reads and executes the computer program from the readable storage medium, so that the device performs the task processing method according to any one of claims 1 to 9.

Images