Data processing method, data processing device, and computer-readable storage medium
By parsing the execution information of the directed acyclic graph from the binary logs of the database, the task list and upstream and downstream relationships are obtained, solving the problem of untimely updates of lineage data in the metadata center and achieving timely updates and accuracy of lineage data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JUHAOKAN TECH CO LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-06-19
AI Technical Summary
In existing technologies, the mechanism by which the Airflow plugin automatically collects lineage information and reports it to the metadata center after task execution lacks an active perception and synchronization mechanism. This results in the metadata center being unable to update in a timely manner, causing lineage information residues and invalid associations, which affects the accuracy of data traceability and quality monitoring.
By acquiring the binary logs of the database, parsing the operational information of the directed acyclic graph, obtaining the task list and upstream and downstream relationships, and controlling the metadata center to update in a timely manner, the accuracy of lineage data is ensured.
It enables timely updates of lineage data in the metadata center during task execution, resolving issues of residual lineage information and invalid associations, and improving the accuracy of data traceability and quality monitoring.
Smart Images
Figure CN122240250A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to a data processing method, a data processing apparatus, and a computer-readable storage medium. Background Technology
[0002] In big data processing scenarios, the streaming workflow platform Airflow acts as a task scheduling system responsible for orchestrating and executing data tasks. Its task lineage (including the association between tasks and data tables, and the upstream and downstream relationships between data tables) is the core data of the metadata center.
[0003] Existing solutions use the Airflow plugin to automatically collect lineage information and report it to the metadata center after the actual task execution. However, this solution has a fundamental flaw: because the lineage collection mechanism is only triggered during task execution, the system lacks a proactive sensing and synchronization mechanism to notify the metadata center to perform corresponding data cleanup when a task is deleted or the directed acyclic graph (DAG) goes offline. This one-way incremental update mode prevents the metadata center from timely learning about structural changes in the upstream scheduling system, resulting in residual lineage information of deleted tasks and the inability to clean up invalid inter-table relationships. Ultimately, this causes the lineage data in the metadata center to gradually become distorted, affecting the accuracy of key functions such as data traceability and quality monitoring.
[0004] Therefore, how to update the lineage data in the metadata center in a timely manner has become an urgent problem to be solved. Summary of the Invention
[0005] To address the aforementioned technical problems, this disclosure provides a data processing method, a data processing apparatus, and a computer-readable storage medium.
[0006] In a first aspect, this disclosure provides a data processing device, comprising: a communicator configured to: acquire binary logs of a database; and a controller configured to: acquire operational information of each directed acyclic graph (DAG) based on the binary logs acquired by the communicator; wherein the operational information includes a status identifier, the status identifier being used to indicate the operational status of the DAG, the operational status including enabled; acquiring a first task list of the first DAG in the current period and a first task list of the previous period, wherein the first task list includes at least one actual task and a task structured query language corresponding to each actual task, and the first DAG includes any item in the directed acyclic graph; parsing a target structured query language to determine the upstream and downstream information of the tasks in the current period; wherein the target structured query language includes any item in the task structured query language; and when the first DAG is not executed for the first time, controlling a metadata center to delete the upstream and downstream information of the tasks and / or the actual tasks based on the upstream and downstream information of the tasks in the previous period, the upstream and downstream information of the tasks in the current period, the first task list of the current period, and the first task list of the previous period, to obtain an updated first DAG.
[0007] Secondly, this disclosure provides a data processing method, comprising: acquiring binary logs of a database; acquiring operational information of each directed acyclic graph (DAG) based on the binary logs; wherein the operational information includes a status identifier, the status identifier being used to indicate the operational status of the DAG, the operational status including enabled; acquiring a first task list of the first DAG in the current period and a first task list of the previous period, wherein the first task list includes at least one actual task and a task structured query language corresponding to each actual task, and the first DAG includes any item in the directed acyclic graph; parsing the target structured query language to determine the upstream and downstream information of the tasks in the current period; wherein the target structured query language includes any item in the task structured query language; when the first DAG is not executed for the first time, controlling the metadata center to delete the upstream and downstream information of the tasks and / or the actual tasks based on the upstream and downstream information of the tasks in the previous period, the upstream and downstream information of the tasks in the current period, the first task list of the current period, and the first task list of the previous period, to obtain an updated first DAG.
[0008] Thirdly, this disclosure provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, the computer program being executed by a controller using a data processing method as provided in any of the second aspects.
[0009] Fourthly, this disclosure provides a computer program product that, when run on a computer, causes the computer to perform any of the data processing methods provided in the second aspect.
[0010] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be encapsulated together with the controller of the data processing device, or it may be encapsulated separately from the controller of the data processing device; this disclosure does not impose any limitations on this.
[0011] The descriptions of the second, third, and fourth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, and fourth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.
[0012] In this disclosure, the names of the aforementioned data processing devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of this disclosure and its equivalents.
[0013] These or other aspects of this disclosure will become more readily apparent in the following description.
[0014] The technical solution provided in this disclosure has the following advantages compared with the prior art: The data processing device and communicator provided in this disclosure are configured to: acquire binary logs from a database; and a controller is configured to: acquire runtime information of each directed acyclic graph (DAG) based on the binary logs acquired by the communicator; acquire the first task list of the first DAG in the current period and the first task list of the previous period for the first DAG in the current period when the runtime status is enabled; parse the target structured query language to determine the upstream and downstream information of the tasks in the current period; and, when the first DAG is not being executed for the first time, control the metadata center to delete the upstream and downstream information of the tasks and / or the actual tasks based on the upstream and downstream information of the tasks in the previous period, the upstream and downstream information of the tasks in the current period, the first task list of the current period, and the first task list of the first period, to obtain an updated first DAG. It can be seen that the data processing device provided in this disclosure, by monitoring the binary logs of the database, for the first DAG in the current period when the runtime status is enabled, based on the first task list of the first DAG in the previous period and the first task list of the current period, the first task list of the previous period, the upstream and downstream information of the tasks in the previous period, and the upstream and downstream information of the tasks in the current period, perceive whether the first DAG and the actual tasks in the first DAG have changed. In this way, the first acyclic graph can be updated in a timely manner each time it is executed, thus solving the problem of how to update the lineage data in the metadata center in a timely manner. Attached Figure Description
[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 One of the flowcharts illustrating the data processing method provided in the embodiments of this application; Figure 2 A system architecture diagram illustrating the application of the data processing method provided in the embodiments of this application; Figure 3 A second schematic flowchart illustrating the data processing method provided in this application embodiment; Figure 4 The third schematic flowchart of the data processing method provided in the embodiments of this application; Figure 5 The fourth flowchart illustrating the data processing method provided in this application embodiment; Figure 6 Fifth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application; Figure 8 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation
[0018] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0019] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0021] In some examples, MySQL in this disclosure is a relational database management system.
[0022] In some examples, PostgreSQL in this disclosure is a powerful object-relational database management system (ORDBMS).
[0023] In some examples, the associated entity in this disclosure refers to a data carrier, such as a database table, file, dataset, etc. They are data producers, consumers, or storage points.
[0024] In some examples, the lineage in this disclosure refers to the complete flow path of data from its source to its final use, including which tasks it passes through, which entities consume it, and the dependencies between tasks.
[0025] The data processing device provided in this disclosure can be a server. When the server executes the data processing method provided in this disclosure, it can be the server's processor.
[0026] In the following embodiments, the execution subject of the data processing method provided in the embodiments of this disclosure is the aforementioned server, which will be used as an example to illustrate the method of the embodiments of this application.
[0027] This application provides a data processing method, such as... Figure 1 As shown, the data processing method may include S11-S15.
[0028] S11. Obtain the binary log of the database.
[0029] In some examples, the binary log can be a Binlog.
[0030] In some examples, the database can be MySQL, PostgreSQL, etc.
[0031] In some examples, the data processing methods provided in this disclosure are applied to, for example... Figure 2 The system architecture shown includes an Airflow scheduler 1, a data processing device 2 that executes the data processing method provided in this embodiment, and a metadata center 3. The data processing device 2 monitors changes in the binary logs of the database to identify the execution of actual tasks. Simultaneously, the data processing device 2 also monitors the running status and scheduling cycle of each DAG.
[0032] S12. Based on the binary log, obtain the running information of each directed acyclic graph; wherein, the running information includes a status identifier, which is used to indicate the running status of the directed acyclic graph, and the running status includes enabled.
[0033] In some examples, the runtime information also includes the scheduling cycle, and the runtime status includes "shutdown". The scheduling cycle can be daily, weekly, etc.
[0034] In some examples, when the server obtains the running information of the DAG, it can obtain the DAG's status identifier, such as "active status". If the active status is off, that is, active=false.
[0035] In some examples, when obtaining runtime information for each directed acyclic graph based on binary logs, the binary logs can be parsed to determine the changes in the DAG table (used to store basic DAG information) and task list (such as the Task instance table, used to store basic information of actual tasks) recorded in the binary logs.
[0036] S13. Obtain the first task list of the first acyclic graph in the current cycle and the first task list of the previous cycle, where the running status is enabled; wherein, the first task list includes at least one actual task and the task structured query language corresponding to each actual task, and the first acyclic graph includes any item in the directed acyclic graph.
[0037] In some examples, a task identifier is assigned to an actual task, and an acyclic graph is assigned to an acyclic graph identifier.
[0038] In some examples, the first task list records one or more task identifiers.
[0039] In some examples, the server can retrieve the task SQL corresponding to the actual task through the target interface of Airflow scheduler 1.
[0040] In some examples, the list of the first task in the previous cycle and the upstream and downstream information of the tasks in the previous cycle can be retrieved from the server's cache.
[0041] S14. Parse the target structured query language to determine the upstream and downstream information of the task in the current cycle; wherein, the target structured query language includes any item in the task structured query language; S15. When the first acyclic graph is not executed for the first time, based on the upstream and downstream information of the tasks in the previous cycle, the upstream and downstream information of the tasks in the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, the metadata center is controlled to delete the upstream and downstream information of the tasks and / or the actual tasks to obtain the updated first acyclic graph.
[0042] In some examples, the server initiates a deletion request to the metadata center to delete upstream and downstream information and / or actual tasks. The metadata center then sends a response to the server confirming the deletion. Next, the server sends an update to the metadata center, updating the first acyclic graph (DAG) identifier, which corresponds to the first acyclic graph. The metadata center then updates the mapping between Tasks and their corresponding upstream and downstream information in the DAG corresponding to the first acyclic graph identifier, as well as the mapping between each actual task in the first task list and the mapping between the first task list and the upstream and downstream information of the Tasks within that list. Finally, the metadata center returns the updated first acyclic graph to the server. Thus, the server obtains the updated first acyclic graph.
[0043] In some examples, when the first acyclic graph is executed for the first time, the server needs to associate the upstream and downstream information of the tasks in the first acyclic graph and record the inter-table relationships as the current task. Afterwards, the server takes a snapshot of the first task list of the first acyclic graph to serve as a basis for determining whether the actual tasks or upstream and downstream information of the first acyclic graph have changed when it is executed next time.
[0044] As described above, the data processing method provided in this disclosure obtains the binary log of the database; based on the binary log obtained by the communicator, it obtains the running information of each directed acyclic graph (DAG); it obtains the first task list of the first DAG in the current cycle and the first task list of the previous cycle for the first DAG with the running status "enabled"; it parses the target structured query language to determine the upstream and downstream information of the tasks in the current cycle; and when the first DAG is not being executed for the first time, based on the upstream and downstream information of the tasks in the previous cycle, the upstream and downstream information of the tasks in the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, it controls the metadata center to delete the upstream and downstream information of the tasks and / or the actual tasks, thereby obtaining the updated first DAG. It can be seen that the data processing device provided in this disclosure, by monitoring the binary log of the database, for the first DAG with the running status "enabled", based on the first task list of the first DAG in the previous cycle and the current cycle, the first task list of the previous cycle, the upstream and downstream information of the tasks in the previous cycle, and the upstream and downstream information of the tasks in the current cycle, perceives whether the first DAG and the actual tasks in the first DAG have changed. In this way, the first acyclic graph can be updated in a timely manner each time it is executed.
[0045] In some feasible examples, runtime information also includes scheduling cycles, and runtime status includes shutdown; combined with Figure 1 ,like Figure 3 As shown, the data processing method provided in this embodiment of the disclosure further includes: S16-S19.
[0046] S16. Obtain the execution time of the last execution of the second acyclic graph whose running state is closed; wherein, the second acyclic graph includes any item in the directed acyclic graph; S17. When the time difference between the current time and the execution time is greater than the scheduling period, obtain the acyclic graph identifier of the second acyclic graph and the second task list associated with the acyclic graph identifier. In some examples, the data processing method provided in this disclosure determines whether to delete the second acyclic graph by judging the relationship between the time difference between the current time and the execution time and the scheduling cycle, thereby avoiding accidental deletion.
[0047] In some examples, the second task list includes at least one actual task and a Task Structured Query Language corresponding to each actual task.
[0048] S18. Based on the acyclic graph identifier and the second task list, determine the associated entities and related lineages; In some examples, when determining associated entities and lineages based on the acyclic graph identifier and the second task list, the server sends a query to the metadata center containing the acyclic graph identifier and the second task list. The metadata center, based on this query, determines the associated entities and lineages corresponding to both the acyclic graph identifier and the second task list. Then, the metadata center sends feedback information containing these associated entities and lineages to the server. In this way, the server obtains the associated entities and lineages corresponding to both the acyclic graph identifier and the second task list.
[0049] S19. Control the metadata center to delete associated entities and related lineages.
[0050] In some examples, the server sends deletion information to the metadata center, instructing the deletion of the associated entities and lineages corresponding to both the acyclic graph identifier and the second task list. Based on this deletion information, the metadata center deletes the associated entities and lineages corresponding to both the acyclic graph identifier and the second task list, and sends feedback information to the server indicating that the associated entities and lineages have been deleted. Thus, the server completes the process of controlling the metadata center to delete associated entities and lineages.
[0051] As described above, the data processing method provided in this disclosure obtains the binary log of the database; based on the binary log obtained by the communicator, it obtains the running information of each directed acyclic graph (DAG); it obtains the execution time of the last execution of the second DAG whose running state is closed; when the time difference between the current time and the execution time is greater than the scheduling cycle, it obtains the acyclic graph identifier of the second DAG and the second task list associated with the acyclic graph identifier; based on the acyclic graph identifier and the second task list, it determines the associated entities and related lineages; and it controls the metadata center to delete the associated entities and related lineages. It can be seen that the data processing device provided in this disclosure, by monitoring the binary log of the database, indicates that the second DAG whose running state is closed needs to be deleted when the time difference between the current time and the execution time is greater than the scheduling cycle; at this time, by obtaining the acyclic graph identifier of the second DAG and the second task list associated with the acyclic graph identifier, it determines the associated entities and related lineages based on the acyclic graph identifier and the second task list; then, it controls the metadata center to delete the associated entities and related lineages, thus completing the cleanup of the second DAG whose running state is closed.
[0052] In some feasible examples, combining Figure 1 ,like Figure 4 As shown, the above S15 can be implemented by the following S150.
[0053] S150. If the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list in the previous cycle is different from that in the current cycle, the control metadata center will delete the upstream and downstream information of the tasks in the previous cycle and the actual tasks that are different from the first task list in the current cycle, and establish the correspondence between the first task list in the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
[0054] As described above, the data processing method provided in this disclosure obtains the binary log of the database; based on the binary log obtained by the communicator, it obtains the running information of each directed acyclic graph; it obtains the first task list of the first acyclic graph in the current cycle and the first task list of the previous cycle for the first acyclic graph whose running status is enabled; it parses the target structured query language to determine the upstream and downstream information of the tasks in the current cycle; when the first acyclic graph is not executed for the first time, if the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list of the previous cycle is different from that in the current cycle, the control metadata center deletes the upstream and downstream information of the tasks in the previous cycle and the actual tasks that are different from the first task list of the current cycle, and establishes the correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph. As can be seen, the data processing device provided in this disclosure, by monitoring the binary logs of the database, can detect whether the first acyclic graph (ACR) has changed, based on the first task list of the previous and current cycles, the first task list of the previous cycle, the upstream and downstream information of the tasks in the previous cycle, and the upstream and downstream information of the tasks in the current cycle. In this way, the ACR can be updated promptly each time it is executed.
[0055] In some feasible examples, combining Figure 1 ,like Figure 5 As shown, the above S15 can be implemented in the following S151.
[0056] S151. If the upstream and downstream information of the tasks in the previous cycle is the same as that in the current cycle, and the first task list in the previous cycle is different from the first task list in the current cycle, the control metadata center will delete the actual tasks that are different from the first task list in the current cycle, and establish the correspondence between the first task list in the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
[0057] As described above, the data processing method provided in this disclosure obtains the binary logs of the database; based on the binary logs obtained by the communicator, it obtains the running information of each directed acyclic graph (DAG); it obtains the first task list of the first DAG in the current cycle and the first task list of the previous cycle for the first DAG with the running status "enabled"; it parses the target structured query language to determine the upstream and downstream information of the tasks in the current cycle; if the upstream and downstream information of the tasks in the previous cycle is the same as that in the current cycle, and the first task list of the previous cycle is different from that in the current cycle, it controls the metadata center to delete the actual tasks that are different from the first task list of the current cycle, and establishes the correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first DAG. It can be seen that the data processing device provided in this disclosure, by monitoring the binary logs of the database, for the first DAG with the running status "enabled", based on the first task list of the first DAG in the previous cycle and the current cycle, the first task list of the previous cycle, the upstream and downstream information of the tasks in the previous cycle, and the upstream and downstream information of the tasks in the current cycle, perceives the first DAG and whether the actual tasks in the first DAG have changed. In this way, the first acyclic graph can be updated in a timely manner each time it is executed.
[0058] In some feasible examples, combining Figure 1 ,like Figure 6 As shown, the above S15 can be implemented in the following S152.
[0059] S152. If the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list in the previous cycle is the same as that in the current cycle, the control metadata center deletes the upstream and downstream information of the tasks in the previous cycle and establishes a correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
[0060] In some examples, the first acyclic graph is not updated when the upstream and downstream information of the tasks in the previous cycle is the same as that in the current cycle, and the first task list in the previous cycle is the same as that in the current cycle.
[0061] As described above, the data processing method provided in this disclosure obtains the binary log of the database; based on the binary log obtained by the communicator, it obtains the running information of each directed acyclic graph (DAG); it obtains the first task list of the first DAG in the current cycle and the first task list of the previous cycle for the first DAG with the running status "enabled"; it parses the target structured query language to determine the upstream and downstream information of the tasks in the current cycle; if the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list of the previous cycle is the same as that in the current cycle, it controls the metadata center to delete the upstream and downstream information of the tasks in the previous cycle and establishes a correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first DAG. It can be seen that the data processing device provided in this disclosure, by monitoring the binary log of the database, for the first DAG with the running status "enabled", based on the first task list of the first DAG in the previous cycle and the current cycle, the first task list of the previous cycle, the upstream and downstream information of the tasks in the previous cycle, and the upstream and downstream information of the tasks in the current cycle, perceives the first DAG and whether the actual tasks in the first DAG have changed. In this way, the first acyclic graph can be updated in a timely manner each time it is executed.
[0062] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0063] This application embodiment can divide the data processing device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0064] like Figure 7 As shown in the diagram, an embodiment of this application provides a schematic diagram of a data processing device. It includes a communicator 101 and a controller 102.
[0065] Communicator 101 is configured to: retrieve the binary log of the database; Controller 102 is configured as follows: Based on the binary log obtained by communicator 101, the running information of each directed acyclic graph is obtained; wherein, the running information includes a status identifier, which is used to indicate the running status of the directed acyclic graph, and the running status includes enabled. Get the first task list of the first acyclic graph in the current period and the first task list of the previous period, which are in the running status of the first acyclic graph. The first task list includes at least one actual task and the task structured query language corresponding to each actual task. The first acyclic graph includes any item in the directed acyclic graph. The target structured query language is parsed to determine the upstream and downstream information of the task in the current cycle; wherein, the target structured query language includes any item in the task structured query language; When the first acyclic graph is not executed for the first time, based on the upstream and downstream information of the tasks in the previous cycle, the upstream and downstream information of the tasks in the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, the control metadata center deletes the upstream and downstream information of the tasks and / or the actual tasks to obtain the updated first acyclic graph.
[0066] In some feasible examples, the runtime information also includes the scheduling cycle, and the runtime status includes shutdown; Before executing the process of acquiring the first task list of the first acyclic graph in the current cycle and the first task list of the previous cycle, where the running status is enabled, controller 102 is also configured as follows: Obtain the execution time of the last execution of the second acyclic graph whose running state is closed; wherein the second acyclic graph includes any item in the directed acyclic graph; When the time difference between the current moment and the execution moment is greater than the scheduling period, obtain the acyclic graph identifier of the second acyclic graph and the second task list associated with the acyclic graph identifier; Based on the acyclic graph identifier and the second task list, related entities and related lineages are identified; Control the metadata center to delete associated entities and related lineages.
[0067] In some implementable examples, when the controller 102 executes the first acyclic graph not for the first time, it controls the metadata center to delete the task upstream and downstream information and / or actual tasks based on the task upstream and downstream information of the previous cycle, the task upstream and downstream information of the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, to obtain an updated first acyclic graph. This is further configured to: If the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list in the previous cycle is different from that in the current cycle, the control metadata center will delete the upstream and downstream information of the tasks in the previous cycle and the actual tasks that are different from the first task list in the current cycle, and establish the correspondence between the first task list in the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
[0068] In some implementable examples, when the controller 102 executes the first acyclic graph not for the first time, it controls the metadata center to delete the task upstream and downstream information and / or actual tasks based on the task upstream and downstream information of the previous cycle, the task upstream and downstream information of the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, to obtain an updated first acyclic graph. This is further configured to: If the upstream and downstream information of the tasks in the previous cycle is the same as that in the current cycle, and the first task list of the previous cycle is different from the first task list of the current cycle, the control metadata center will delete the actual tasks that are different from the first task list of the current cycle, and establish the correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
[0069] In some implementable examples, when the controller 102 executes the first acyclic graph not for the first time, it controls the metadata center to delete the task upstream and downstream information and / or actual tasks based on the task upstream and downstream information of the previous cycle, the task upstream and downstream information of the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, to obtain an updated first acyclic graph. This is further configured to: If the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list in the previous cycle is the same as that in the current cycle, the control metadata center will delete the upstream and downstream information of the tasks in the previous cycle and establish a correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
[0070] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and their functions will not be repeated here.
[0071] Of course, the data processing device provided in this application embodiment includes, but is not limited to, the modules described above. For example, the data processing device may also include a memory 103. The memory 103 may be used to store the program code of the data processing device, and may also be used to store data generated by the data processing device during operation, such as data in write requests.
[0072] like Figure 8As shown, this application embodiment also provides a chip system that can be applied to the data processing device in the foregoing embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 may be the processor in the aforementioned data processing device. The processor 1501 and the interface circuit 1502 are interconnected via a line. The processor 1501 can receive and execute computer instructions from the memory of the aforementioned data processing device through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the data processing device can perform the various steps executed by the data processing device in the foregoing embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.
[0073] This application also provides a computer-readable storage medium for storing computer instructions for operating the aforementioned data processing device.
[0074] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing device, characterized in that, include: The communicator is configured to retrieve the binary logs of the database. The controller is configured as follows: Based on the binary log obtained by the communicator, the running information of each directed acyclic graph is obtained; wherein, the running information includes a status identifier, the status identifier is used to indicate the running status of the directed acyclic graph, and the running status includes enabled; Obtain the first task list of the first acyclic graph in the current cycle and the first task list of the previous cycle, where the running status is enabled; wherein, the first task list includes at least one actual task and a task structured query language corresponding to each actual task, and the first acyclic graph includes any item in the directed acyclic graph. The target structured query language is parsed to determine the upstream and downstream information of the task in the current cycle; wherein, the target structured query language includes any item of the task structured query language; When the first acyclic graph is not being executed for the first time, based on the upstream and downstream information of the tasks in the previous cycle, the upstream and downstream information of the tasks in the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, the metadata center is controlled to delete the upstream and downstream information of the tasks and / or the actual tasks to obtain the updated first acyclic graph.
2. The data processing device according to claim 1, characterized in that, The operation information also includes a scheduling cycle, and the operation status also includes being shut down; Before the controller retrieves the first task list of the first acyclic graph in the current cycle and the first task list of the previous cycle, where the running state is enabled, it is further configured as follows: Obtain the execution time of the last execution of the second acyclic graph whose running state is closed; wherein, the second acyclic graph includes any item in the directed acyclic graph; When the time difference between the current time and the execution time is greater than the scheduling period, obtain the acyclic graph identifier of the second acyclic graph and the second task list associated with the acyclic graph identifier; Based on the acyclic graph identifier and the second task list, related entities and related lineages are determined; Control the metadata center to delete the associated entity and the associated lineage.
3. The data processing device according to claim 1, characterized in that, When the controller executes the first acyclic graph not for the first time, based on the upstream and downstream task information of the previous cycle, the upstream and downstream task information of the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, it controls the metadata center to delete the upstream and downstream task information and / or actual tasks to obtain an updated first acyclic graph. This is further configured as follows: If the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list in the previous cycle is different from that in the current cycle, the control metadata center will delete the upstream and downstream information of the tasks in the previous cycle and the actual tasks that are different from the first task list in the current cycle, and establish a correspondence between the first task list in the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
4. The data processing device according to claim 1, characterized in that, When the controller executes the first acyclic graph not for the first time, based on the upstream and downstream task information of the previous cycle, the upstream and downstream task information of the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, it controls the metadata center to delete the upstream and downstream task information and / or actual tasks to obtain an updated first acyclic graph. This is further configured as follows: If the upstream and downstream information of the tasks in the previous cycle is the same as that in the current cycle, and the first task list of the previous cycle is different from the first task list of the current cycle, the control metadata center will delete the actual tasks that are different from the first task list of the current cycle, and establish the correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
5. The data processing device according to claim 1, characterized in that, When the controller executes the first acyclic graph not for the first time, based on the upstream and downstream task information of the previous cycle, the upstream and downstream task information of the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, it controls the metadata center to delete the upstream and downstream task information and / or actual tasks to obtain an updated first acyclic graph. This is further configured as follows: If the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list of the previous cycle is the same as that in the current cycle, the control metadata center deletes the upstream and downstream information of the tasks in the previous cycle and establishes a correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
6. A data processing method, characterized in that, include: Retrieve the binary log of the database; Based on the binary log, the running information of each directed acyclic graph is obtained; wherein, the running information includes a status identifier, the status identifier is used to indicate the running status of the directed acyclic graph, and the running status includes enabled; Obtain the first task list of the first acyclic graph in the current cycle and the first task list of the previous cycle, where the running status is enabled; wherein, the first task list includes at least one actual task and a task structured query language corresponding to each actual task, and the first acyclic graph includes any item in the directed acyclic graph. The target structured query language is parsed to determine the upstream and downstream information of the task in the current cycle; wherein, the target structured query language includes any item of the task structured query language; When the first acyclic graph is not being executed for the first time, based on the upstream and downstream information of the tasks in the previous cycle, the upstream and downstream information of the tasks in the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, the metadata center is controlled to delete the upstream and downstream information of the tasks and / or the actual tasks to obtain the updated first acyclic graph.
7. The data processing method according to claim 6, characterized in that, The operation information also includes a scheduling cycle, the operation status also includes being off, and the method further includes: Obtain the execution time of the last execution of the second acyclic graph whose running state is closed; wherein, the second acyclic graph includes any item in the directed acyclic graph; When the time difference between the current time and the execution time is greater than the scheduling period, obtain the acyclic graph identifier of the second acyclic graph and the second task list associated with the acyclic graph identifier; Based on the acyclic graph identifier and the second task list, related entities and related lineages are determined; Control the metadata center to delete the associated entity and the associated lineage.
8. The data processing method according to claim 6, characterized in that, When the first acyclic graph is not being executed for the first time, based on the upstream and downstream task information of the previous cycle, the upstream and downstream task information of the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, the metadata center is controlled to delete the upstream and downstream task information and / or the actual tasks to obtain an updated first acyclic graph, including: If the upstream and downstream information of the tasks in the previous cycle is different from that in the current cycle, and the first task list in the previous cycle is different from that in the current cycle, the control metadata center will delete the upstream and downstream information of the tasks in the previous cycle and the actual tasks that are different from the first task list in the current cycle, and establish a correspondence between the first task list in the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
9. The data processing method according to claim 6, characterized in that, When the first acyclic graph is not being executed for the first time, based on the upstream and downstream task information of the previous cycle, the upstream and downstream task information of the current cycle, the first task list of the current cycle, and the first task list of the previous cycle, the metadata center is controlled to delete the upstream and downstream task information and / or the actual tasks to obtain an updated first acyclic graph, including: If the upstream and downstream information of the tasks in the previous cycle is the same as that in the current cycle, and the first task list of the previous cycle is different from the first task list of the current cycle, the control metadata center will delete the actual tasks that are different from the first task list of the current cycle, and establish the correspondence between the first task list of the current cycle and the upstream and downstream information of the tasks in the current cycle, thus obtaining the updated first acyclic graph.
10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method as described in any one of claims 6-9.