Task processing method and device

By determining the status of upstream tasks in the data warehouse and executing the target task only after the upstream task has been completed, the problem of data consistency between data warehouse layers is solved, and accurate and timely data synchronization is achieved.

CN118568102BActive Publication Date: 2026-11-10BOE TECHNOLOGY GROUP CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310215160.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-28
Publication Date
2026-11-10
Estimated Expiration
2043-02-28

AI Technical Summary

Technical Problem

In the enterprise production process, the data warehouse has complex data processing dependencies between different layers, making it difficult to guarantee data consistency, especially during incremental synchronization, which can easily lead to data loss and delays.

Method used

By obtaining the execution information of the target task, the execution status of its upstream tasks is determined. The target task is only executed when the upstream task is completed and the start time of its execution information is after or the same as the end time of the target task, thus ensuring data consistency.

Benefits of technology

This effectively avoids data loss and delays caused by incomplete upstream tasks, ensuring the accuracy and timeliness of data at all levels in the data warehouse.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118568102B_ABST
    Figure CN118568102B_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure provide a task processing method and device, relating to the technical field of data processing, and used for ensuring the data consistency between layers in a database. The method comprises: obtaining execution information of a target task at a current time, the target task being used for performing a processing operation on target data, the execution information of the target task comprising a start time and an end time of the target data; determining an upstream task of the target task and execution information of the upstream task; and if the target task and the upstream task are located in a same task execution cycle, and the start time of the data in the execution information of the target task is located after the end time of the target data or is the same as the end time of the target data, then executing the target task according to the execution information of the target task at the current time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method and apparatus for processing tasks. Background Technology

[0002] As businesses grow, various departments within a manufacturing company (such as business lines, production lines, and product lines) will build various information systems to process their respective product data. For example, the information system built for the production line can integrate and analyze product data uploaded by equipment and manually uploaded data, thus facilitating developers to determine the causes of product defects based on the analysis results.

[0003] Due to the complex and lengthy production processes, large volumes of data involved in product manufacturing, a data warehouse can be used to layer the data for better management. The data processed at each layer has dependencies on each other. Ensuring data consistency between these layers is a critical issue that needs to be addressed. Summary of the Invention

[0004] The purpose of the embodiments disclosed herein is to provide a task processing method and apparatus for ensuring data consistency between layers in a database.

[0005] To achieve the above objectives, the embodiments of this disclosure provide the following technical solutions:

[0006] On the one hand, a task processing method is provided. This method includes: obtaining the execution information of a target task at the current time, the target task being used to process target data, the execution information of the target task including the start and end times of the target data; determining the upstream task of the target task and its execution information; if the target task and the upstream task are within the same task execution cycle, and the start time of the data in the execution information of the target task is after or the same as the end time of the target data, then the target task is executed according to its current execution time.

[0007] In the above method, after obtaining the execution information of the target task, the upstream task and its execution information can be determined. It is then determined whether the target task and the upstream task are within the task execution cycle, and whether the start time of the data in the upstream task's execution information is after the end time of the data in the target task's execution information, or whether they are the same. If the target task and the upstream task are within the task execution cycle, it means that both are in the execution phase. If the start time of the data in the upstream task's execution information is after the end time of the data in the target task's execution information, or is the same as the end time of the target data, it means that the upstream task has been successfully executed. In this way, when executing the target task, the execution data of the target task can be obtained from the processing results of the upstream task, ensuring data consistency.

[0008] In some embodiments, the method further includes: if the target task and / or its upstream task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task of the target task is before the end time of the target data, then the target task is not executed.

[0009] In some embodiments, the method further includes: in response to the completion of the target task, updating the execution information of the target task, wherein the updated execution information includes the start time and end time of the data to be processed after the current time.

[0010] In some embodiments, the above-mentioned "determining the upstream task of the target task" specifically includes: determining the upstream task of the target task according to a preset correspondence. The preset correspondence includes upstream and downstream relationships between multiple tasks, wherein the multiple tasks include the target task.

[0011] In some embodiments, the method further includes: in response to the completion of the target task, storing the actual execution time of the target task, wherein the actual execution time of the target task includes the start time and end time of the target task.

[0012] In some embodiments, the execution information of the target task may also include the execution time interval and the number of executions of the target task. The above-mentioned "execute the target task according to the execution information of the target task at the current time" includes: executing the target task according to the execution time interval, and updating the number of executions of the target task after the target task is completed.

[0013] In some embodiments, the method further includes: if the target task and its upstream task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task is before the end time of the target data, outputting a prompt message to indicate the reason for not executing the target task.

[0014] In some embodiments, the target task is an ETL task.

[0015] On the other hand, a task processing apparatus is provided. The processing apparatus includes: an acquisition unit, a determination unit, and a processing unit.

[0016] The acquisition unit is configured to acquire the execution information of the target task at the current time. This target task is used to process target data, and its execution information includes the start and end times of the target data. The determination unit is configured to determine the upstream task of the target task and its execution information. The processing unit is configured to execute the target task if the target task and its upstream task are within the same task execution cycle, and the start time of the data in the target task's execution information is after or the same as the end time of the target data.

[0017] In some embodiments, the processing unit is further configured to: if the target task and / or the upstream task of the target task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task of the target task is before the end time of the target data, then the target task is not executed.

[0018] In some embodiments, the processing unit is further configured to: update the execution information of the target task in response to the completion of the target task, wherein the updated execution information includes the start and end times of the data to be processed after the current time.

[0019] In some embodiments, the determining unit is specifically configured to: determine the upstream task of the target task according to a preset correspondence. The preset correspondence includes upstream and downstream relationships between multiple tasks, wherein the multiple tasks include the target task.

[0020] In some embodiments, the apparatus further includes a storage unit. The storage unit is configured to: in response to the completion of the execution of the target task, store the actual execution time of the target task, the actual execution time of the target task including the start time and end time of the execution of the target task.

[0021] In some embodiments, the execution information of the target task further includes the execution time interval and the number of executions of the target task. The processing unit is specifically configured to: execute the target task according to the execution time interval, and update the number of executions of the target task after the target task is completed.

[0022] In some embodiments, the apparatus further includes an output unit. The output unit is configured to output a prompt message indicating the reason for not executing the target task if the target task and / or its upstream task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task is before the end time of the target data.

[0023] In some embodiments, the target task is an ETL task.

[0024] In another aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program instructions that, when executed on a computer, cause the computer to perform the processing method as described in any of the above embodiments.

[0025] In another aspect, a computer program product is provided. The computer program product includes computer program instructions, which, when executed on a computer, cause the computer to perform the processing method as described in any of the above embodiments.

[0026] In another aspect, a computer program is provided. When the computer program is executed on a computer, it causes the computer to perform the processing method as described in any of the above embodiments.

[0027] In another aspect, a chip is provided, the chip including a processor and a communication interface, the communication interface and the processor being coupled, the processor being used to run computer programs or instructions to implement the method as described in any of the above embodiments.

[0028] Specifically, the chip provided in this application also includes a memory for storing computer programs or instructions.

[0029] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the device, or it may be packaged separately from the processor of the device; this application does not impose any limitation on this.

[0030] In this application, the names of the aforementioned devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those in this application, they fall within the scope of the claims of this application and their equivalents. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this disclosure, the accompanying drawings used in some embodiments of this disclosure will be briefly described below. Obviously, the drawings described below are only drawings of some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings. In addition, the drawings described below can be regarded as schematic diagrams and are not intended to limit the actual size of the product, the actual flow of the method, the actual timing of the signals, etc. involved in the embodiments of this disclosure.

[0032] Figure 1 This is a structural diagram of a data warehouse according to some embodiments;

[0033] Figure 2 This is a schematic diagram illustrating the dependencies between tasks according to some embodiments;

[0034] Figure 3 This is a schematic diagram illustrating the dependencies between tasks according to some embodiments;

[0035] Figure 4 This is a schematic diagram illustrating the dependencies between tasks in an actual production process according to some embodiments;

[0036] Figure 5 This is a structural diagram of an ETL system according to some embodiments;

[0037] Figure 6 A flowchart of a task processing method according to some embodiments;

[0038] Figure 7 A flowchart of a task processing method according to some embodiments;

[0039] Figure 8 A flowchart of a task processing method according to some embodiments.

[0040] Figure 9 This is a structural diagram of a task processing apparatus according to some embodiments;

[0041] Figure 10 This is a structural diagram of a task processing apparatus according to some embodiments. Detailed Implementation

[0042] The technical solutions in some embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments provided in this disclosure are within the scope of protection of this disclosure.

[0043] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as open-ended and encompassing, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples.

[0044] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0045] "At least one of A, B and C" has the same meaning as "at least one of A, B or C", both including the following combinations of A, B and C: only A, only B, only C, combinations of A and B, combinations of A and C, combinations of B and C, and combinations of A, B and C.

[0046] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0047] As used herein, depending on the context, the term “if” may optionally be interpreted as meaning “when”, “in the event of”, “in response to determination”, or “in response to detection”. Similarly, depending on the context, the phrase “if it is determined that…” or “if [the stated condition or event] is detected” may optionally be interpreted as meaning “in the event of determination that…”, “in response to determination that…”, “when [the stated condition or event] is detected”, or “in response to the detection of [the stated condition or event]”.

[0048] The use of “applies to” or “configured to” in this article implies an open and inclusive language that does not preclude applicability to or configuration to devices that perform additional tasks or steps.

[0049] In addition, the use of “based on” implies openness and inclusivity, because processes, steps, calculations or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0050] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0051] The following explanations of the terms used in the embodiments of this application are provided to facilitate the reader's understanding.

[0052] Extract-transform-load (ETL): This term describes the process of extracting, transforming, and loading data from a source to a destination.

[0053] In one example, the source and destination can be different databases. For example, the databases could include: a manufacturing execution system (MES) database, an operational data store (ODS) database, and a manufacturing data warehouse (MDW) database.

[0054] The MES database can be connected to the production equipment. The MES database can be used to store manufacturing data from the production equipment, such as product quantity, product qualification rate, and production efficiency.

[0055] It should be noted that, to ensure data security and reliability, the MES database can be configured with access control or protection measures. That is, without the necessary permissions, the MES database cannot perform operations such as adding, deleting, modifying, or querying the stored data.

[0056] The ODS database can be used to store raw data. The data in the ODS database is consistent with the data generated by the source system. For example, ODS data can be extracted from the MES database and stored, summarized, and analyzed during the production process. In other words, the ODS database can back up data in the MES database. This avoids data anomalies caused by data loss or tampering in the MES database.

[0057] The MDW database is used to summarize, store, and analyze data from the manufacturing process. In other words, the MDW database responds to user actions, organizing and summarizing data to create multiple types of tables. The data in these tables can then be used as a data source and foundation for subsequent data processing.

[0058] In one scenario, when building a database, a company that manufactures products can divide the data into multiple layers according to business needs. For example, the data in an MDW database can be divided into an operational data store (ODS) layer (also known as a data preparation layer, which will be referred to as the ODS layer for ease of description), a data warehouse detail layer (DWD) (which will be referred to as the DWD layer for ease of description), and a data warehouse summary layer (DWS) (which will be referred to as the DWS layer for ease of description).

[0059] The ODS layer can be used to store data from the MES database. The DWD layer can be used to store data from the ODS layer after it has been correlated and cleaned. The DWS layer can be used to store data from the DWD layer after it has been summarized.

[0060] In conjunction with the above application scenarios, such as Figure 1 As shown, the MDW database can include multiple layers, and data can be transferred and processed between these layers via ETL. For example, data in the MES database can be stored in MES tables 1 to 3. Data in the MDW database can be stored in ODS tables 4 to 6, DWD tables 7 to 9, and DWS tables 10 to 11.

[0061] Among them, ODS tables 4 to ODW tables 6 are used to store data in the ODS layer. DWD tables 7 to DWD tables 9 are used to store data in the DWD layer. DWS tables 10 to DWS tables are used to store data in the DWS layer.

[0062] Task: This can refer to the process of extracting, transforming, and loading data from tables at different levels. For example, a task can be an ETL task.

[0063] In this embodiment, a task may include an upstream task and a downstream task. Upstream and downstream tasks are inextricably linked. A downstream task depends on an upstream task. One downstream task may correspond to one or more upstream tasks. An upstream task may also correspond to one or more downstream tasks.

[0064] In one example, if the data in table A originates from table B, then in the relationship between table A and table B, table B is called the upstream of table A, and table A is called the downstream of table B. Therefore, the task corresponding to table A is called the downstream task, and the task corresponding to table B is called the upstream task.

[0065] For example, combining Figure 1 Tasks in the DWD layer depend on tasks in the ODS layer. Tasks in the DWS layer depend on tasks in the DWD layer. Furthermore, there are both multiple dependencies and single dependencies between tasks.

[0066] For example, the upstream task of the task corresponding to ODS Table 4 can be the task corresponding to MES Table 1. The upstream task of the task corresponding to DWD Table 7 can be the task corresponding to ODS Table 4. The upstream task of the task corresponding to DWD Table 8 can be the tasks corresponding to ODS Table 4 and ODS Table 5. The upstream task of the task corresponding to DWS Table 11 can be the tasks corresponding to DWD Table 8 and DWD Table 9.

[0067] The task corresponding to the table can refer to a task that requires processing the data in the table. For example, it could be updating data in the table or inserting data into the table.

[0068] In one example, the task corresponding to ODS table 4 can process the data in ODS table 4 and write the processed data back into ODS table 4. Similarly, the task corresponding to DWD table 7 can process the data in the ODS table and write the processed data back into DWD table 7. In other words, the data in DWD table 7 originates from ODS table 4.

[0069] It should be noted that, Figure 1 The arrows in the diagram indicate the direction of data flow. The left side of the arrow represents upstream, and the right side represents downstream. The diagram only illustrates the case of a downstream task depending on one upstream task and a downstream task depending on two upstream tasks. In actual factory production processes, the dependencies between tasks are much more complex.

[0070] Because the production process in a factory generates a large amount of data, incremental synchronization is typically used to synchronize data between different layers. However, incremental synchronization can lead to errors due to various reasons, such as server issues or source database problems. The following section explains incremental synchronization and potential errors using a specific scenario.

[0071] Scenario 1: No dependencies are set between tasks, and tasks run independently.

[0072] In this context, dependency can refer to the association between tasks. For example, taking task A and task B as an example, task A is used to process data in table A, and task B is used to process data in table B. The data in table B (i.e., the downstream table) comes from table A (i.e., the upstream table).

[0073] In one example, at time 202206280710 (i.e., 07:10 on June 28, 2022), both Task A and Task B need to execute data between time 20220628070500 and time 20220628071000, and the scheduling information of Task A and Task B is updated every 5 minutes, incrementally synchronizing data from 5 minutes prior to the current time.

[0074] In one possible scenario, if task A encounters an error or fails to execute successfully after task B begins execution, the following problem may occur even if task B executes successfully because task A and task B do not have a dependency relationship:

[0075] 1. When task B is executed, table A does not yet contain data after time 20220628070500. However, since the data in table B originates from table A, table B will also not contain data after time 20220628070500. Therefore, even if task B executes successfully, the incrementally retrieved data will be empty.

[0076] 2. Since task B was executed successfully this time, no error log will be generated. Therefore, it is not easy to find the problem of missing data in table B.

[0077] 3. The data range for the next execution of Task B is from time 20220628071000 to time 20220628071500. Due to the time misalignment between upstream and downstream tasks, the incremental data acquired by Task B may all be empty.

[0078] It should be noted that in this embodiment, when a task scheduling fails, the scheduled task will be executed repeatedly until it succeeds. For example, if task A fails to execute data between 20220628070500 and 20220628071000, but task B succeeds, since the data executed by task B comes from the data executed by task A, and task A failed, the data executed by task B is empty. Thus, even if task B succeeds, no data will be generated, resulting in subsequent task B executions always having empty data.

[0079] Based on the above example, the execution status of tasks A and B can be shown in Tables 1 to 3. Table 1 shows the status before tasks A and B are executed at time 202206280710; Table 2 shows the status after tasks A and B are executed at time 202206280710; and Table 3 shows the status after tasks A and B are executed at time 202206280715.

[0080] Table 1

[0081] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 Ready Task B 20220628070500 20220628071000 READY

[0082] Table 2

[0083] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 Error (ERROR) Task B 20220628071000 20220628071500 READY

[0084] It should be noted that in Table 2, ERROR indicates that task A failed to execute at time 2022062807110.

[0085] Table 3

[0086] Task Name Start time End time Task execution status Task A 20220628071000 20220628071500 READY Task B 20220628071500 20220628072000 READY

[0087] As shown in Tables 2 and 3 above, due to the failure of Task A, the data executed by Task A in Table 3 cannot be synchronized with the data executed by Task B. Therefore, when Task B executes data from 20220628071500 to 20220628072000, since this time period's data originates from the time period from 20220628071500 to 20220628072000 executed by Task A, but Task A has not yet executed this time period's data, the data executed by Task B is empty, meaning the data executed by Task B is inaccurate.

[0088] Scenario 2: Tasks are set to have dependencies on each other. The tasks are independent of each other, and the downstream task can only be executed after the upstream task is executed successfully.

[0089] In one example, such as Figure 2 As shown, Task A and Task B are upstream tasks of Task C, and Task A and Task B are independent of each other.

[0090] In this task, the data for execution comes from tables A and B. There is no correlation between the data in table A and table B, and table A corresponds to task A, while table B corresponds to task B. Task C can only be executed after tasks A and B have both executed successfully.

[0091] In one scenario, at time 202206280710, task A fails to retrieve data between 20220628070500 and 20220628071000, while task B successfully retrieves data between 20220628070500 and 20220628071000. Therefore, task C will also fail to retrieve data between 20220628070500 and 20220628071000. At time 202206280715, if task A successfully retrieves data between 20220628070500 and 20220628071000, while task B fails, then task C will still fail to retrieve data between 20220628070500 and 20220628071000. Therefore, task C experiences significant execution delays.

[0092] Based on the above examples, the execution status of tasks A, B, and C can be shown in Tables 4 to 6. Table 4 shows the status when tasks A, B, and C have not been executed at time 2022062807110; Table 5 shows the execution status of tasks A, B, and C at time 2022062807110; and Table 6 shows the execution status of tasks A, B, and C at time 202206280715.

[0093] Table 4

[0094] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 READY Task B 20220628070500 20220628071000 READY Task C 20220628070500 20220628071000 READY

[0095] Table 5

[0096] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 ERROR Task B 20220628071000 20220628071500 READY Task C 20220628070500 20220628071000 ERROR

[0097] It should be noted that in Table 5, ERROR indicates that task A failed to execute at time 2022062807110.

[0098] Table 6

[0099]

[0100]

[0101] As shown in Tables 5 and 6 above, the failure of either Task A or Task B will cause Task C to fail. Furthermore, as shown in Table 6, since Task A failed at time 2022062807110 and Task B failed at time 202206280715, the execution time of Task C was delayed by 10 minutes.

[0102] Scenario 3: Dependencies are set between tasks. There are dependencies between upstream tasks, and downstream tasks can only be executed after the upstream task is executed successfully.

[0103] In one example, such as Figure 3 As shown, Task A and Task B are upstream tasks of Task C, and there is a dependency relationship between Task A and Task B.

[0104] The data for task C comes from tables A and B. There is no correlation between the data in table A and the data in table B. Table A corresponds to task A and table B corresponds to task B.

[0105] In one scenario, if both Task A and Task B fail at time 202206280710, Task C will also fail. If Task A succeeds at time 202206280715, but Task B fails, Task C will also fail. If both Task A and Task B succeed at time 202206280720, Task C will also succeed, but its execution time will be delayed.

[0106] Based on the above examples, the execution status of tasks A, B, and C can be shown in Tables 7 to 10. Table 7 shows the status when tasks A, B, and C have not been executed at time 202206280710; Table 8 shows the execution status of tasks A, B, and C at time 202206280710; Table 9 shows the execution status of tasks A, B, and C at time 202206280715; and Table 10 shows the execution status of tasks A, B, and C at time 202206280720.

[0107] Table 7

[0108] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 READY Task B 20220628070500 20220628071000 READY Task C 20220628070500 20220628071000 READY

[0109] Table 8

[0110] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 ERROR Task B 20220628070500 20220628071000 ERROR Task C 20220628070500 20220628071000 ERROR

[0111] In Table 8, because task A failed to execute, both task B and task C also failed to execute.

[0112] Table 9

[0113] Task Name Start time End time Task execution status Task A 20220628071000 20220628071500 READY Task B 20220628070500 20220628071000 ERROR Task C 20220628070500 20220628071000 ERROR

[0114] In Table 9, task A was executed successfully, but task B failed, which caused all tasks C to fail.

[0115] Table 10

[0116] Task Name Start time End time Task execution status Task A 20220628071500 20220628072000 READY Task B 20220628071000 20220628071500 READY Task C 20220628071000 20220628071500 READY

[0117] In Table 10, after Task A, Task B, and Task C are executed at time 202206280720, data scheduling should be performed between time 202206280720 and time 202206280725. However, the latest end time for Task A and Task B is time 202206250715. That is, the execution time of the upstream task is delayed by 10 minutes, and the execution time of Task C (the downstream task) is delayed by 10 minutes.

[0118] In a practical application scenario, during the construction of a manufacturing data warehouse, the production status of products can be analyzed based on indicators such as input-output ratio, defect rate, and first-pass yield. For example... Figure 4 As shown, this illustrates the dependencies between tasks in the actual production process. Figure 4 In the diagram, the task (or table) on the right side of the arrow depends on the task (or table) on the left side of the arrow. Therefore, there are cases of single dependency and multiple dependencies.

[0119] like Figure 4 The task shown uses the task dependency in Scenario 1 above. When the data in the ODS_LOTHISTORY table is delayed, all subsequent subject table data that depends on the ODS_LOTHISTORY table will be missing, resulting in the factory output and defective number being less than the actual production, and also causing the yield index to be inaccurate.

[0120] If there is a dependency relationship between tasks JOB_DWS_FPY_LOT, JOB_DWS_INOUT_LOT, and JOB_DWS_NG_LOT as described in Scenario 2 above, the failure of task JOB_DWS_INOUT_LOT or task JOB_DWS_NG_LOT will cause a delay in the data in the DWS_FPY_LOT table, resulting in a lag in the calculation of yield metrics.

[0121] If there are dependencies between tasks JOB_DWS_FPY_LOT, JOB_DWS_INOUT_LOT, and JOB_DWS_NG_LOT as shown in Scenario 3 above, that is, tasks in the input-output topic depend on tasks in the bad topic, and the pass rate task depends on both tasks in the input-output topic and tasks in the bad topic, it may cause serious data lag in the input-output topic table and the pass rate topic table.

[0122] To address the aforementioned issues, this application provides a method that does not restrict upstream and downstream relationships in ETL tools, but instead determines whether a downstream task can be executed based on the data corresponding to upstream and downstream tasks and the relationship between upstream and downstream tasks.

[0123] In one example, in this application, when a task begins execution, the execution information of the task, such as the start and end times of the task's execution data, is retrieved from the `config` table. The upstream tasks of the task are retrieved from the `mapping` table. If the execution information of all upstream tasks for the task is retrieved from the `config` table, and the start time of each upstream task is after the end time of the task, then the task is executed. After the task execution is completed, the execution log is written to the `config hist` table, and the execution information in the `config` table is updated. If all upstream information for the task cannot be retrieved from the `mapping` table, and / or the start time of the upstream information for the task is before the end time of the task, then the task execution is stopped, and the reason for the task execution failure is output.

[0124] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0125] like Figure 5 The diagram shown is a structural schematic of an ETL system provided in an embodiment of this application. The ETL system may include multiple databases. These databases are interconnected. For example, the multiple databases may include an MES database and an MDW database. The details of these multiple databases are as described above and will not be repeated here.

[0126] In one example, the database may be configured with an ETL tool. This ETL tool can be used to schedule tasks on the data in the database. For example, the ETL tool can be used to perform the methods described in the following embodiments.

[0127] The following is combined Figure 1 The multiple layers shown and Figure 5 The system shown describes the method provided in the embodiments of this application.

[0128] like Figure 6 As shown, the method may include S601 to S603.

[0129] S601. Obtain the execution information of the target task at the current time.

[0130] The target task can be used to process the target data, such as scheduling the target data. For example, the target task can be an ETL task.

[0131] In one example, combining Figure 1 as well as Figure 2 The ETL tool is configured for the MDW database. The target task can be... Figure 1The task in the DWS layer can target data that is also data from the DWS layer. For example, the target data can be stored in the DWS layer in tabular form. Specifically, the target data could be data from DWS table 10 or DWS table 11.

[0132] In one possible implementation, the database can have a `config` table. This `config` table can include execution information for multiple tasks, and these tasks must be executed within the same period. These multiple tasks can include target tasks. The execution information of the target tasks can include time information of the target data (such as start time (START_TIME) and end time (END_TIME). In this way, ETL tools can obtain the execution information of the target data at the current time from this `config` table.

[0133] The start and end times of the target data can be referred to as the timestamps of the target data. In other words, ETL tools can perform scheduling tasks on data within this timestamp interval.

[0134] Of course, the execution information of the target task can also include other information, such as the name of the target task (JOB_NAME), the actual execution time of the target task (also known as the update time (UPDATE_TIME)), the execution time interval (INCREMENT_MI), the number of executions (BATCH_ID), and the execution status (JOB_STATUS).

[0135] The actual execution time of a task can refer to the time it takes for the task to complete. The execution time interval of a task can refer to the data that needs to be processed each time the task is executed. The number of executions can refer to the total number of times the task has been executed.

[0136] For example, Table 11 shows a config table provided in an embodiment of this application. This table can be used to record information such as the range of data that multiple tasks need to synchronize at the current time.

[0137] Table 11

[0138]

[0139] It should be noted that Table 11 only shows three tasks and their execution information. Of course, more tasks and their corresponding execution information can be included, without limitation.

[0140] S602. Determine the upstream task of the target task and the execution information of the upstream task.

[0141] The upstream tasks of the target task can be referred to in the above description and will not be repeated here. The execution information of the upstream tasks can be referred to in the above description of the execution information of the target task and will not be repeated here.

[0142] In one possible implementation, the ETL tool can determine the upstream tasks of the target task based on a pre-defined mapping. This pre-defined mapping can include upstream and downstream relationships between multiple tasks, including the target task.

[0143] In one example, the preset mapping can be pre-configured in the ETL tool or database. For instance, the preset mapping can be stored in the ETL tool or database in the form of a table (such as a mapping table). For example, the preset mapping can be as shown in Table 12.

[0144] Table 12

[0145] Task Name Table Name Source table name Interface time job_dwd_inout_lot dwd_inout_lot ods_lothistory 20220506 15:50:15 job_dws_fpy_move_d dws_fpy_move_d dws_inout_lot_d 20220507 13:25:03 job_dws_fpy_move_d dws_fpy_move_d dws_ng_lot_d 20220507 13:25:03 job_dws_fpy_move_d dws_fpy_move_d dws_palletpacking_lot_d 20220507 13:25:03 job_dws_inout_lot_d dws_inout_lot_d dws_inout_lot_h 20220507 08:42:13

[0146] It should be noted that in Table 12, "task name" can refer to the name of a downstream task. "Table name" can refer to the name of the table corresponding to the downstream task (i.e., the downstream table name). "Source table name" can refer to the name of the upstream table that the downstream table depends on. "Interface time" refers to the insertion time of the correspondence between the task and its upstream task. For example, for the second row of data in Table 12, "interface time (20220506 15:50:15)" means that the row of data was inserted into the table at 15:50:15 on 20220506.

[0147] It's important to note that the table names and task names in Table 12 are related. The task name identifies the corresponding table, and vice versa. For example, the task `job_dwd_inout_lot` in the second row of Table 12 has a source table named `ods_lothistory`. Therefore, the upstream task of `job_dwd_inout_lot` is `job_ods_lothistory`. In other words, adding `job_` before the source table name will give you the upstream task.

[0148] After identifying the upstream task of the target task, the ETL tool can obtain the execution information of the upstream task based on its name. Specifically, refer to the method for obtaining the execution information of the target task in S501 above; details will not be repeated here.

[0149] S603. If the target task and its upstream task are within the task execution cycle, and the start time of the data in the execution information of the upstream task is after the end time of the target data, or the same as the end time of the target data, then the target task is executed according to the execution information of the target task at the current time.

[0150] The phrase "the target task and its upstream task are within the execution cycle" can also be described as "the target task and its upstream task need to be executed", "the target task and its upstream task are within the task execution list", or "the target task and its upstream task need to be executed synchronously".

[0151] In one example, the ETL tool or database may be pre-configured with one or more tables, each containing multiple tasks. For example, this table could be the `config` table mentioned above (i.e., Table 11). If the target task and its upstream task are located in the `config` table, it means that the target task and its upstream task are within the same task execution cycle.

[0152] If the start time of the data in the execution information of the upstream task is after the end time of the target data, it means that the upstream task has been completed and the target data has been obtained.

[0153] In one example, referring to Table 11 above, if the target task is `job_dwd_inout_lot`, and the upstream task is `job_ods_lothistory`, and the target task's end time is 20220627183000, while the upstream task's start time is 20220627191000, since 20220627191000 is after 20220627183000, it means the upstream task has already completed executing the data prior to 20220627183000. In other words, the upstream task has already provided the data source for the target task. Therefore, when executing the target task, the ETL tool can obtain the data (i.e., the target data) with a timestamp interval between 20220627173000 and 20220627183000.

[0154] In one possible implementation, after determining that the target task and its upstream task are within the task execution cycle, if the start time of the data in the execution information of the upstream task is after the end time of the target data in the execution information of the target task, or if the start time of the data in the execution information of the upstream task is the same as the end time of the target data in the execution information of the target task, the ETL tool can execute the target task based on the execution information of the target data at the current time. For example, the ETL tool can schedule and process the data between timestamp intervals in the aforementioned config table based on the execution information of the target task in that table.

[0155] Furthermore, if the execution information of the target task also includes the number of times the task has been executed, then the execution count in the execution information can be updated after the target task is completed. For example, the execution count in the execution information can be increased by a first value (such as 1). For instance, if the execution count in the execution information of the target task is A when the target task has not been executed, then after the target task is completed, the execution count in the execution information of the target task can be updated to A+1.

[0156] Conversely, if the target task and / or upstream task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task is before the end time of the target data, then the target task will not be executed.

[0157] In one example, if either the target task or its upstream task is not listed in the config table, it indicates that there are tasks that are not being executed. To avoid the target task failing or lacking data, it can be temporarily suspended until the config table is updated and contains both the target task and its upstream task.

[0158] In another example, if the start time of the data in the execution information of the target task in the config table is earlier than the end time of the upstream task, it means that the upstream task has not yet been completed. In this case, if the target task is executed, it will be unable to obtain the data. Therefore, the target task can be skipped.

[0159] based on Figure 6 The technical solution, after obtaining the execution information of the target task, can determine the upstream task and its execution information, and determine whether the target task and the upstream task are within the task execution cycle, and whether the start time of the data in the upstream task's execution information is after the end time of the data in the target task's execution information, or whether the two are the same. If the target task and the upstream task are within the task execution cycle, it means that both the target task and the upstream task are in the execution phase. If the start time of the data in the upstream task's execution information is after the end time of the data in the target task's execution information, or the two are the same, it means that the upstream task has been successfully executed. In this way, when executing the target task, the data required for the execution of the target task can be obtained from the processing results of the upstream task, ensuring data consistency.

[0160] In some embodiments, such as Figure 7 As shown, the method provided in this application embodiment may further include S701.

[0161] S701. In response to the completion of the target task, update the execution information of the target task.

[0162] The updated execution information includes the start and end times of the data to be processed after the current time.

[0163] In one possible implementation, in response to the completion of the target task, the ETL tool can update the target task's execution information based on the timestamp interval of the data in the target task's execution information.

[0164] In one example, if the timestamp interval of the data in the execution information of the target task is 60 minutes, and the start time of the target data in the execution information of the target task at the current time is 20220627173000 and the end time is 20220627183000, then the start time of the target data in the updated execution information of the target task can be 20220627173000 and the end time can be 20220627183000.

[0165] In one scenario, if the target task fails to execute successfully, the ETL tool can re-execute the target task until it succeeds, and then update the execution information of the target task after it succeeds.

[0166] For example, referring to Table 11 above, if the task in Table 11 is executed successfully, the ETL tool can update the start time and end time in Table 11 according to the timestamp interval corresponding to the task.

[0167] based on Figure 7 The technical solution allows for updating the start and end times of the data in the execution information of the target task after the target task is completed, so that the target task can accurately obtain the data to be processed based on the updated execution information when it is executed next time.

[0168] In some embodiments, the method provided in this application may further include: in response to the completion of the target task, the ETL tool may store the actual execution time of the target task.

[0169] The actual execution time of the target task can include the start time and end time of the target task.

[0170] In one example, in response to the completion time of the target task, the ETL tool can store the actual execution time of the target task in a database. For example, the actual execution time of the target task can be stored in the form of a table. This table could be a `configHist` table. The `configHist` table can be used to record logs of executed / running tasks.

[0171] For example, Table 13 shows a config Hist table provided in an embodiment of this application. This config Hist table may include task name, number of task executions, start and end times of the task execution data, timestamp interval of the task, task status, and actual execution time of the task, etc.

[0172] Table 13

[0173]

[0174] It should be noted that Table 13 only shows the logs for one task executed 9 times. Of course, more logs can be included without limitation. In Table 13, the start time and end time of the task are the same. This is because in some scenarios, such as when the amount of data being synchronized is small, the execution time of the task may be in the microsecond range. This would result in the start time and end time appearing to be the same in the table, but in reality, there is still a time interval between the two.

[0175] Based on this embodiment, after the target task is completed, the ETL tool can store the execution log of the target task, which makes it easier for subsequent administrators to check whether the task execution is abnormal.

[0176] In some embodiments, the method provided in this application may further include: if the target task and the upstream task are not within the same task execution cycle, and / or the end time of the data in the execution information of the upstream task is after the start time of the target data, outputting a prompt message.

[0177] This prompt message can be used to indicate the reason for not performing the target task.

[0178] In one scenario, if the target task and / or upstream task are not within the task execution cycle, the output prompt information can be used to indicate that there are non-executing tasks among the target task and upstream tasks. For example, the reason for not executing the target task may include that the upstream task of the target task is not executing or that the execution status of the upstream task of the target task is unknown.

[0179] In another scenario, if the start time of the data in the execution information of the upstream task is before the end time of the target data, the output prompt information can be used to indicate that the upstream task has not been completed or that the target task has no data source.

[0180] Based on the technical solution of this embodiment, when the ETL tool does not execute a task, it can output a prompt message indicating that the target task is not executed, so that maintenance personnel can accurately determine the reason for the task not being executed based on the prompt message.

[0181] In some embodiments, the target task can have multiple upstream tasks. In this case, the ETL tool can obtain the execution information of each upstream task and execute the target task when these multiple upstream tasks are in the same task execution cycle as the target task, and the start time of the data in the execution information of each upstream task is before the end time of the target task. Thus, even when the target task has multiple upstream tasks, the ETL tool can still guarantee the successful execution of the target task and that the data source is not empty.

[0182] The method provided in the embodiments of this application will be described below with reference to specific examples.

[0183] In one example, the target task is task C, which has two upstream tasks, task A and task B. For example... Figure 8 As shown, the method provided in this application embodiment may include:

[0184] S801. In response to the start of task C, retrieve the execution information of task C from the config table.

[0185] The execution information may include the start time, end time, and timestamp interval of the data that task C needs to process.

[0186] S802. Store the execution information when executing task C.

[0187] In one example, the execution information of this task C can be stored in a Kettle variable.

[0188] S803. Read the task names of the upstream tasks that task C depends on from the mapping table.

[0189] In one example, the upstream task names that task C depends on can be task A and task B.

[0190] S804. Based on the task names of the upstream tasks that task C depends on, retrieve the execution information of task A and task C from the config table.

[0191] If the start time of the data in the execution information of task A and the start time of the data in the execution information of task B are greater than or equal to the end time of the data in the execution information of task C, then execute S805 to S806; otherwise, execute S807.

[0192] Wherein, the start time of the data in the execution information of task A and the start time of the data in the execution information of task B are actually greater than or equal to the end time of the data in the execution information of task C. This can mean that the start time of the data in the execution information of task A and the start time of the data in the execution information of task B are both after the end time of the data in the execution information of task C, or both are the same as the end time of the data in the execution information of task C.

[0193] S805. Perform data processing based on the execution information during the execution of task C.

[0194] Data processing can include data transformation, data storage, etc.

[0195] S806. Write the log of this execution of task C to the config Hist table, and update the relevant information of task C in the config table to obtain the time information of the data required for the next execution of task C (that is, the start time and end time of the data).

[0196] Among them, the relevant information for task C can refer to the start and end times of the data that task C needs to process, as shown in the config table.

[0197] S807. Terminate task C and output a prompt message.

[0198] In one possible implementation, S804, S805, and S807 can be implemented through code.

[0199] In one example, the code could look like this.

[0200]

[0201]

[0202] It should be noted that the above code is merely an example, and other code can also be used to achieve the same result; there are no restrictions.

[0203] Based on the above embodiments, when executing a task, the upstream task can be obtained through the mapping table, and the execution information of the upstream task can be obtained from the config table. In this way, it can be determined whether to execute the task based on the execution information of the upstream task and the execution information of the current task, avoiding data inconsistencies during task execution.

[0204] In some embodiments, to address the data inconsistency issue in Scenario 1 above, based on the method provided in this application, the execution status of Task A and Task B can be shown in Tables 14 to 16 below. Table 14 shows the situation when Task A and Task B have not been executed at time 202206280710; Table 15 shows the situation after Task A and Task B have been executed at time 2022062807110; and Table 16 shows the situation after Task A and Task B have been executed at time 202206280715.

[0205] Table 14

[0206] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 READY Task B 20220628070500 20220628071000 READY

[0207] Table 15

[0208] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 ERROR Task B 20220628070500 20220628071000 ERROR

[0209] As shown in Table 15, since task A failed to execute, task B also failed to execute, and the data is accurate.

[0210] Table 16

[0211] Task Name Start time End time Task execution status Task A 20220628071000 20220628071500 READY Task B 20220628071000 20220628071500 READY

[0212] As shown in Table 16, when task A is executed successfully, task B is also executed successfully, indicating that the data is accurate.

[0213] To address the significant delay in downstream task execution in Scenario 2 described above, the execution status of Task A, Task B, and Task C, based on the method provided in this application embodiment, is shown in Tables 17 to 19 below. Table 17 shows the status when Task A, Task B, and Task C have not been executed at time 2022062807110; Table 18 shows the execution status of Task A, Task B, and Task C at time 2022062807110; and Table 19 shows the execution status of Task A, Task B, and Task C at time 202206280715.

[0214] Table 17

[0215]

[0216]

[0217] Table 18

[0218] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 ERROR Task B 20220628071000 20220628071500 READY Task C 20220628070500 20220628071000 ERROR

[0219] Table 19

[0220] Task Name Start time End time Task execution status Task A 20220628071000 20220628071500 READY Task B 20220628071000 20220628071500 ERROR Task C 20220628071000 20220628071500 READY

[0221] As shown in Table 19 above, based on the method provided in the embodiments of this application, task C is executed only after task A and task B have successfully processed the data from time 20220628070500 to time 20220628071000. The execution time of task C is only delayed by 5 minutes. Compared with the execution result of task C in Table 6 (delayed by 10 minutes), the timeliness of the data of the downstream task in this application is higher.

[0222] To address the significant delay in the execution time of downstream tasks in Scenario 3 described above, the execution status of Task A, Task B, and Task C, based on the method provided in this application embodiment, can be shown in Tables 20 to 23 below. Table 20 shows the situation when Task A, Task B, and Task C have not been executed at time 202206280710; Table 21 shows the execution status of Task A, Task B, and Task C at time 202206280710; Table 22 shows the execution status of Task A, Task B, and Task C at time 202206280715; and Table 23 shows the execution status of Task A, Task B, and Task C at time 202206280720.

[0223] Table 20

[0224] Task Name Start time End time Task execution status Task A 20220628070500 20220628071000 READY Task B 20220628070500 20220628071000 READY Task C 20220628070500 20220628071000 READY

[0225] Table 21

[0226]

[0227]

[0228] As shown in Table 21, the start time of task A (20220628070500) is less than the end time of task C in Table 20 (20220628071000). Therefore, the task execution status of task C is ERROR.

[0229] Table 22

[0230] Task Name Start time End time Task execution status Task A 20220628071000 20220628071500 READY Task B 20220628071000 20220628071500 ERROR Task C 20220628071000 20220628071500 READY

[0231] As shown in Table 22, the start time of Task A (20220628071000) and the start time of Task B (20220628071000) are greater than the end time of Task C (20220628071000) in Table 20. Therefore, Task C in Table 22 is executed. That is, Task C is executed on the data between 20220628071000 and 20220628071500.

[0232] Table 23

[0233] Task Name Start time End time Task execution status Task A 20220628071500 20220628072000 READY Task B 20220628071500 20220628072000 READY Task C 20220628071500 20220628072000 READY

[0234] As shown in Table 23, the start time of Task A (20220628071500) and the start time of Task B (20220628071500) are both equal to the end time of Task C (20220628071500) in Table 22. Therefore, Task C in Table 23 can be executed. That is, Task C is executed on the data between 20220628071500 and 20220628072000.

[0235] It should be noted that the various embodiments of this application can be referenced or learned from each other. For example, the same or similar steps, method embodiments, system embodiments and device embodiments can be referenced from each other without limitation.

[0236] This application embodiment can divide the task processing device into functional modules or functional units according to the above method examples. For example, each function can be divided into a separate functional module or functional unit, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or in software functional modules or functional units. The module or unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0237] like Figure 9 The diagram shown is a schematic diagram of a task processing device provided in an embodiment of this application. The device includes: an acquisition unit 901, a determination unit 902, and a processing unit 903.

[0238] The acquisition unit 901 is configured to acquire the execution information of the target task at the current time. This target task is used to process target data, and the execution information includes the start and end times of the target data. The determination unit 902 is configured to determine the upstream task of the target task and its execution information. The processing unit 903 is configured to execute the target task if the target task and its upstream task are within the same task execution cycle, and the start time of the data in the target task's execution information is after or the same as the end time of the target data.

[0239] In some embodiments, the processing unit 903 is further configured to: if the target task and / or the upstream task of the target task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task of the target task is before the end time of the target data, then the target task is not executed.

[0240] In some embodiments, the processing unit 903 is further configured to: update the execution information of the target task in response to the completion of the target task, wherein the updated execution information includes the start time and end time of the data to be processed after the current time.

[0241] In some embodiments, the determining unit 902 is specifically configured to: determine the upstream task of the target task according to a preset correspondence. The preset correspondence includes upstream and downstream relationships between multiple tasks, wherein the multiple tasks include the target task.

[0242] In some embodiments, the apparatus further includes a storage unit 904. The storage unit 904 is configured to store the actual execution time of the target task in response to the completion of the target task, the actual execution time of the target task including the start time and end time of the target task.

[0243] In some embodiments, the execution information of the target task further includes the execution time interval and the number of executions of the target task. The processing unit 903 is specifically configured to: execute the target task according to the execution time interval, and update the number of executions of the target task when the target task is completed.

[0244] In some embodiments, the apparatus further includes an output unit 905. The output unit 905 is configured to output a prompt message indicating the reason for not executing the target task if the target task and / or its upstream task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task is before the end time of the target data.

[0245] In some embodiments, the target task is an ETL task.

[0246] When implemented in hardware, the acquisition unit 901 in this embodiment can be integrated onto the communication interface, and the determination unit 902 and processing unit 903 can be integrated onto the processor. Specific implementation methods are as follows: Figure 10 As shown.

[0247] Figure 10 A schematic diagram of another possible device structure for the task processing apparatus involved in the above embodiments is shown. The device includes a processor 1002 and a communication interface 1003. The processor 1002 is used to control and manage the operation of the device, for example, executing the steps performed by the determination unit 902 and processing unit 903, and / or performing other processes of the technology described herein. The communication interface 1003 is used to support communication between the device and other network entities, for example, executing the steps performed by the acquisition unit 901. The device may also include a memory 1001 and a bus 1004, the memory 1001 being used to store the device's program code and data.

[0248] The memory 1001 may be a memory in the device, and the memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include a combination of the above types of memory.

[0249] The processor 1002 described above can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0250] Bus 1004 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1004 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0251] Figure 10 The device in the middle can also be a chip. The chip includes one or more processors 1002 and a communication interface 1003.

[0252] Optionally, the chip also includes a memory 1001, which may include read-only memory and random access memory, and provides operation instructions and data to the processor 1002. A portion of the memory 1001 may also include non-volatile random access memory (NVRAM).

[0253] In some implementations, memory 1001 stores elements such as execution modules or data structures, or subsets thereof, or extended sets thereof.

[0254] In this embodiment of the application, the corresponding operation is executed by calling the operation instructions stored in the memory 1001 (the operation instructions can be stored in the operating system).

[0255] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0256] Some embodiments of this disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer, cause the computer to perform a method for constructing an ETL system as described in any of the above embodiments.

[0257] For example, the aforementioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., CDs (Compact Disks), DVDs (Digital Versatile Disks), etc.), smart cards, and flash memory devices (e.g., EPROMs (Erasable Programmable Read-Only Memory), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this disclosure may represent one or more devices for storing information and / or other machine-readable storage media. The term "machine-readable storage media" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0258] Some embodiments of this disclosure also provide a computer program product, for example, stored on a non-transitory computer-readable storage medium. The computer program product includes computer program instructions that, when executed on a computer, cause the computer to perform the methods described in the above embodiments.

[0259] Some embodiments of this disclosure also provide a computer program. When executed on a computer, the computer program causes the computer to perform the methods described in the embodiments above.

[0260] The beneficial effects of the computer-readable storage medium, computer program product, and computer program described above are the same as the beneficial effects of the methods described in some of the above embodiments, and will not be repeated here.

[0261] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0262] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0263] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0264] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for processing a task, characterized in that, include: Obtain the execution information of the target task at the current time. The target task is used to process target data. The execution information includes the start time and end time of the target data. Determine the upstream task of the target task and the execution information of the upstream task; If the target task and the upstream task are within the same task execution cycle, and the start time of the data in the execution information of the upstream task is after the end time of the target data, or the same as the end time of the target data, then the target task is executed according to the execution information of the target task at the current time.

2. The method according to claim 1, characterized in that, The method further includes: If the target task and / or the upstream task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task is before the end time of the target data, then the target task will not be executed.

3. The method according to claim 1 or 2, characterized in that, The method further includes: In response to the completion of the target task, the execution information of the target task is updated. The updated execution information includes the start and end times of the data to be processed after the current time.

4. The method according to claim 1 or 2, characterized in that, Determining the upstream task of the target task includes: Based on a preset correspondence, the upstream task of the target task is determined. The preset correspondence includes upstream and downstream relationships between multiple tasks, wherein the multiple tasks include the target task.

5. The method according to claim 3, characterized in that, The method further includes: In response to the completion of the target task, the actual execution time of the target task is stored, including the start time and end time of the target task.

6. The method according to claim 5, characterized in that, The execution information also includes the execution time interval and execution count of the target task. Executing the target task based on its execution information at the current time includes: The target task is executed according to the execution time interval, and the execution count of the target task is updated after the target task is completed.

7. The method according to claim 1 or 2, characterized in that, The method further includes: If the target task and / or its upstream task are not within the task execution cycle, and / or the start time of the data in the execution information of the upstream task is before the end time of the target data, a prompt message is output, which indicates the reason for not executing the target task.

8. The method according to claim 1 or 2, characterized in that, The target task is an Extract-Transform-Load ETL task.

9. A task processing apparatus, characterized in that, The device includes: The acquisition unit is configured to: acquire the execution information of a target task at the current time, wherein the target task is used to process target data, and the execution information includes the start time and end time of the target data; The determining unit is configured to: determine the upstream task of the target task and the execution information of the upstream task; The processing unit is configured to: if the target task and the upstream task are within the task execution cycle, and the start time of the data in the execution information of the upstream task is after the end time of the target data, or is the same as the end time of the target data, then execute the target task according to the execution information of the target task at the current time.

10. A processing apparatus, characterized in that, include: A processor and a communication interface; the communication interface is coupled to the processor, the processor being configured to run computer programs or instructions to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the method described in any one of claims 1-8.

Citation Information

Patent Citations

  • Scheduling scheme generation method and device

    CN109871270A

  • Task delay monitoring method and device, equipment and storage medium

    CN115629903A