Data lineage parsing method and system

Abstract

The present invention provides a data lineage parsing method and system, for use in parsing data lineage information of a task instance. The data lineage parsing method comprises: monitoring the state of a task instance, and on the basis of parameters of the task instance, selecting a corresponding parsing method, wherein the parameters comprise at least the type of the task instance; using the parsing method to parse the task instance to obtain an input table / output table; and constructing data lineage information on the basis of the input table / output table. According to the solution provided by the present invention, the compatibility of the data lineage technology can be improved, so that the data lineage technology has wider applicability and exhibits better data processing precision.

Description

A data lineage analysis method and system Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data lineage analysis method and system. Background Art

[0002] In the age of intelligence, the demand for data processing is increasing. Accurately tracking and tracing data can effectively identify data issues in big data systems and improve data quality. Data lineage technology is a method for tracking and recording the flow and changes of data throughout its lifecycle. Currently, the following methods are commonly used to generate data lineage:

[0003] 1. Manual data lineage entry: Data developers write documents to record the lineage relationships between data tables, conversion logic, and other information. Data entities and their lineage relationships are manually created and managed through drag-and-drop operations and form input. This method relies on manual operation, is prone to errors and omissions, and is difficult to maintain and update.

[0004] 2. Leverage the logging and auditing capabilities of data warehouses: Some data warehouses and database management systems (such as Snowflake, Redshift, and MySQL) provide logging and auditing features that record data operations and change history. Scripts or tools can also be written to analyze these logs and audit records to extract data lineage information. However, this approach is limited to specific systems that support logging and auditing features and has limited compatibility.

[0005] 3. Parsing SQL to Generate Data Lineage: This method automatically extracts and generates data lineage information by analyzing SQL statements in data processing tasks. This method can infer data lineage relationships by parsing table names, field names, and join operations in SQL statements. However, it relies on correct SQL statements and accurate table and field naming conventions and is not applicable to other types of tasks or development languages.

[0006] 4. Use data exploration tools: Data exploration tools (such as Trifacta and DataRobot) can automatically analyze the statistical information, distribution, and association characteristics of a dataset and infer their lineage relationships. These tools can automatically explore and infer data lineage, reducing the workload of manual input and parsing. However, these tools may have compatibility issues with specific development languages ​​or data warehouses, and lack integration with scheduling systems, making it impossible to use lineage information to control task scheduling.

[0007] It can be seen that the above existing technical solutions have some inherent disadvantages, such as:

[0008] 1. Low automation and difficult maintenance: Manually entering lineage information or relying on log analysis methods requires manual operation or script writing, making it difficult to maintain and update lineage information.

[0009] 2. Only for specific development languages ​​or data warehouses, poor compatibility: Some methods are only applicable to specific development languages ​​or data warehouses and have limited compatibility with other systems.

[0010] 3. It is not connected to the scheduling system, and lineage information cannot be used to control task scheduling: Existing technical solutions lack close integration between lineage information and the task scheduling system, and cannot control task scheduling and execution through lineage information.

[0011] Therefore, how to provide a method for data lineage analysis that is highly adaptable and accurate in addressing the shortcomings of the above-mentioned data lineage technology is an urgent problem to be solved in this field.

[0012] Summary of the Invention

[0013] The present invention provides a data lineage analysis method and system, the purpose of which is to improve the compatibility of data lineage technology so that it has a wider range of applicability and better data processing accuracy.

[0014] In order to achieve the above-mentioned purpose, a technical solution adopted by the present invention is: a data lineage parsing method, which is used to parse the data lineage information of a task instance, including: monitoring the status of the task instance and selecting a corresponding parsing method according to the parameters of the task instance; the parameters include at least the type of the task instance; using the parsing method to parse the task instance to obtain an input table / output table; and constructing data lineage information based on the input table / output table.

[0015] In a preferred embodiment, the parsing method includes at least a first parsing method for task instances based on non-database management and operation languages; the database management and operation languages ​​include at least SQL-like languages.

[0016] In a preferred embodiment, the first parsing method includes: generating a logic execution plan tree according to the state of the task instance; traversing the logic execution plan tree and finding a first related node, and generating the input table / output table according to the first related node.

[0017] In a preferred embodiment, the parsing method includes at least a second parsing method for task instances based on the database management and operation language.

[0018] In a preferred embodiment, the second parsing method includes: constructing a language parser; using the language parser to parse the task instance into an abstract syntax tree (AST); traversing the abstract syntax tree (AST) and finding a second related node, and generating the input table / output table based on the second related node.

[0019] In a preferred embodiment, the parsing method includes a third parsing method for the task instance of the log, and the third parsing method includes: calling the underlying computing engine, querying and parsing the execution information of the task instance; generating the input table / output table based on the execution information.

[0020] In a preferred embodiment, constructing data lineage information based on the input table / output table includes: storing data lineage information via a relational database and / or a graph database.

[0021] In order to achieve the above-mentioned purpose, another technical solution adopted by the present invention is: a data lineage analysis system, which is used to execute any of the above-mentioned methods; wherein, the data lineage analysis system selects a corresponding analysis method according to the type of the task instance, and the corresponding analysis methods are each independently designed in the form of a lineage analysis plug-in.

[0022] In order to achieve the above-mentioned purpose, another technical solution adopted by the present invention is: a scheduling system using any of the above-mentioned methods or the above-mentioned data lineage analysis system, and the scheduling system uses the above-mentioned method or the above-mentioned data lineage analysis system to parse the data lineage information of the task instance.

[0023] In a preferred embodiment, when the dependency rule of the task instance includes data-based dependency, the scheduling system parses the dependency rule of the task instance according to the data lineage information.

[0024] Compared with the existing technology, the advantages of the present invention are: (1) the present invention can easily support multiple data development languages ​​and multiple data sources by classifying tasks and formulating corresponding parsing strategies; (2) the present invention can parse tasks based on non-database management and operation languages, breaking through the limitations of task types; (3) the present invention can implement column-level lineage parsing, improving the accuracy of data processing task execution; (4) the present invention closely combines lineage parsing with the scheduling system, and realizes the automatic construction of the complete lineage relationship of the entire data warehouse; (5) compared with the results of merely displaying the data lineage relationship, the combination of lineage parsing and scheduling system can efficiently apply data lineage relationship to various scenarios of the big data platform, thereby improving data development. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] FIG1 is a flow chart of a method for data lineage analysis according to an embodiment of the present invention.

[0027] Figure 2: A flowchart of a data lineage analysis method provided by an embodiment of the present invention.

[0028] FIG3 is a flow chart showing a scheduling system determining task instance dependency rules according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0030] As used herein, "at least one" means one or more than one.

[0031] In this document, terms such as "include", "comprising", "containing" and "having" are open-ended and indicate the presence of the described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0032] In this article, a "task instance" refers to a specific task operation performed to complete required data processing, calculations, analysis, or other business logic. A task instance is a specific execution unit of a task, responsible for executing a predefined task flow or operation and generating the corresponding results. For example, in a data warehouse platform, task instances include but are not limited to data cleansing tasks, data analysis tasks, data import / export tasks, resource scheduling tasks, and business process tasks.

[0033] In this article, a "scheduling system" refers to software or services used to manage and coordinate the execution of tasks in a computer system. It is responsible for triggering, scheduling, and monitoring the execution of tasks according to predefined rules and conditions, on a set schedule, or triggered by events.

[0034] In this article, "development language" refers to a formal language used to write computer programs, which is a tool used to interact with computers and instruct computers to perform specific tasks, including but not limited to the following elements: grammatical rules, semantic rules, keywords and identifiers, data types, control structures, libraries and frameworks, and compilation or interpretation.

[0035] In this document, "data source" refers to a source or carrier that provides data or information, including an accessible dataset, database, file, sensor, web service or other data provider.

[0036] In this article, "SQL-like language" refers to a standard query language used to manage relational databases, or a programming language with similar syntax and features to SQL (Structured Query Language), which is used to store, retrieve, modify, and manage data in the database.

[0037] In this article, "dependency" refers to the relationship between a task or operation and other tasks, where the execution or completion of one task depends on the status, results, or execution order of other tasks; common dependencies include task-based dependencies, time window-based dependencies, state-based dependencies, etc.

[0038] The present invention proposes a data lineage analysis method and system, which can perform data lineage analysis on any type of task instance without being limited by the development language, data source, etc. of the task instance.

[0039] Furthermore, the present invention divides the data lineage analysis methods into several types, so that a matching data lineage analysis method can be selected according to the type of task instance, thereby avoiding the data lineage analysis being restricted by the task type.

[0040] Furthermore, the present invention proposes a data lineage analysis method for task instances based on non-database management and operation languages. This method can meet the needs of task instances based on non-database management and operation languages, such as unstructured data processing, file processing, API calls, etc.

[0041] Furthermore, the present invention combines the above-mentioned data lineage analysis method for task instances based on non-database management and operation languages ​​with the data lineage analysis method for task instances based on database management and operation languages, as well as the data lineage analysis method for task instances based on logs, so that it can meet the data lineage analysis needs of basically all types of task instances.

[0042] 1 is a flow chart of a data lineage analysis method according to an embodiment of the present invention. The method includes the following steps:

[0043] S1-1: Monitor the status of the task instance and select the corresponding parsing method based on the parameters of the task instance.

[0044] Specifically, this embodiment selects a corresponding parsing method based on the parameters of the task instance, wherein the parameters of the task instance at least include the type of the task instance. At the same time, this embodiment monitors the status of the task instance.

[0045] In an optional embodiment, the parameters of a task instance may typically include the instance ID, task type, task configuration, scheduling configuration, timestamp, owner, etc. These parameters can all serve as the basis for selecting a parsing method. This embodiment divides the data lineage parsing methods according to task type. For example, for Spark computing tasks, this embodiment analyzes the task's physical execution plan to infer the task's input and output tables.

[0046] In an optional embodiment, one or more data lineage analysis methods are preset in the system. After the parameters of the task instance to be analyzed are determined, the corresponding data lineage analysis method can be selected for analysis.

[0047] During the creation and execution of a task instance, there may be multiple states. This embodiment monitors the state so that it can (1) ensure the accuracy of data lineage: if the task fails or an exception occurs, it may cause incomplete or erroneous data lineage information; by monitoring the state of the task instance, problems in the task execution can be discovered and handled in a timely manner to ensure the accuracy of data lineage; (2) track data changes and flows: the state of the task instance can provide detailed information about the data processing process, including the start time, end time, execution progress, etc. of the task; by monitoring the task state, data changes and flow paths can be tracked to understand how the data is converted and processed from the input table to the output table, so as to facilitate the subsequent construction of a complete data lineage relationship. In an optional embodiment, the monitoring of the task instance can be completed by the scheduling system, see the description of the embodiment below for details. In an optional embodiment, monitoring the state of the task instance is a pre-judgment step of the data lineage analysis method, that is, when the task instance runs successfully, the data lineage analysis method is executed; if the task instance fails to run, the data lineage analysis method is not executed.

[0048] S1-2: Use the parsing method to parse the task instance to obtain the input table / output table.

[0049] This embodiment uses a parsing method to parse a task instance to obtain the input table and output table. Specifically, after the task instance is completed, this embodiment parses the task instance according to the selected data lineage parsing method. The parsed content includes the task instance's configuration, definition, metadata, etc., to determine the input table and output table of the task instance, respectively.

[0050] Different types of task instances have different characteristics and requirements, involving different data processing methods and technologies. Consequently, different types of task instances may require different parsing methods and logic to parse the input and output of the task instance. This invention selects a parsing method appropriate for each task type to ensure accurate data lineage information is obtained. This also improves the compatibility and adaptability of data lineage parsing.

[0051] S1-3: Construct data lineage information based on input table / output table.

[0052] This embodiment constructs data lineage information based on input and output tables. Specifically, the relevant information of the input and output tables typically includes the table name, fields, data types, and sources. This embodiment uses this information to analyze the logic, data conversion rules, or other metadata of the task instance and determine the relationship between the input and output tables, thereby obtaining data lineage information.

[0053] In an optional embodiment, constructing data lineage information based on the input table / output table includes storing the data lineage information through a relational database and / or a graph database. Specifically, data lineage information can be expressed and stored in a variety of ways. For example, the source, destination, and conversion path of the data can be visualized by constructing a lineage relationship graph. In an optional embodiment, after obtaining the input table / output table of multiple task instances, the data information in the table (which can be accurate to the table level and field level) is constructed into a global directed acyclic graph and stored in the graph database for use. In an optional embodiment, the data lineage information can at least be used for lineage graph query, lineage graph search, task dependency prompts, impact surface analysis, etc.

[0054] 2 is a flow chart of a data lineage analysis method according to an embodiment of the present invention. The method includes the following steps:

[0055] S2-1: Monitor the status of the task instance and select the corresponding parsing method based on the parameters of the task instance.

[0056] Specifically, this embodiment selects a corresponding parsing method based on the parameters of the task instance, wherein the parameters of the task instance at least include the type of the task instance. At the same time, this embodiment monitors the status of the task instance.

[0057] In this embodiment, the parsing method includes at least a first parsing method for task instances based on non-database management and operation languages, a second parsing method for task instances based on database management and operation languages, and a third parsing method for task instances based on logs; wherein, the database management and operation language includes at least SQL-type languages.

[0058] In an optional embodiment, when the type of the task instance is neither a database management or operation class nor a log class, the parsing method will select the first parsing method and execute steps S2-2 to S2-3; in an optional embodiment, when the type of the task instance is a database management or operation class, the parsing method will select the second parsing method and execute steps S2-4 to S2-6; in an optional embodiment, when the type of the task instance is a log class, the parsing method will select the third parsing method and execute steps S2-7 to S2-8.

[0059] S2-2: Generate a logical execution plan tree based on the status of the task instance.

[0060] In this embodiment, generating a logical execution plan tree according to the state of the task instance includes: obtaining the state of the task instance, parsing the configuration of the task instance, and constructing the logical execution plan tree.

[0061] Specifically, this embodiment monitors the status of task instances. Therefore, when a task instance is completed, its current status, including its running status, execution progress and process, and execution results, can be easily obtained. The configuration of a task instance typically includes input parameters, output parameters, data source connections, transformation rules, etc. By analyzing the configuration of a task instance and combining it with the running process of the task instance, a logical execution plan tree can be constructed.

[0062] In an optional embodiment, a logical execution plan tree of the task instance is constructed based on the configuration and state information of the task instance. The logical execution plan tree is composed of operation nodes and edges of the task instance, where each node represents a task operation step and the edges represent the dependency relationship between task operations.

[0063] S2-3: Traverse the logical execution plan tree and find the first related node, and generate an input table / output table according to the first related node.

[0064] In this embodiment, the first related node may be a node in the logical execution plan tree that reflects the flow and conversion relationship of data in the task instance; after the input table / output table is generated according to the first related node, step S2-9 is executed.

[0065] In an optional embodiment, the first related nodes include at least data operation nodes, data conversion nodes, etc. Among them, the data operation node represents the data reading or writing operation involved in the task instance; for example, the node that reads the input table can be represented as a data operation node, which is used to obtain the input data of the task; similarly, the node that writes the output table can also be represented as a data operation node, which is used to write the output data of the task to the specified target table. The data conversion node represents the data conversion operation in the task instance, which processes, calculates or converts the input data to generate output data. The data conversion node usually involves a series of data processing steps, such as data cleaning, data filtering, data aggregation, data calculation, etc.

[0066] In an optional embodiment, there are multiple ways to traverse the logical execution plan tree, which can be selected based on actual needs, data structures and / or programming language characteristics. For example, it can be implemented through recursion, iteration, and specific data structures (such as stacks or queues).

[0067] S2-4: Build a language parser.

[0068] In this embodiment, for a task instance based on the database management and operation language, a language parser is used to parse the grammatical structure and semantic information of the task instance.

[0069] In an optional embodiment, existing tools and technologies can be selected for construction based on the grammatical and semantic rules of the task instance development language, for example, using tools such as ANTLR and JavaCC to automatically generate parser code based on the grammatical rules.

[0070] S2-5: Use the language parser to parse the task instance into an abstract syntax tree.

[0071] This embodiment uses a language parser to parse the task instance into an abstract syntax tree (AST), wherein each node of the abstract syntax tree represents a grammatical structure of the language, such as a statement, expression, function declaration, etc.

[0072] S2-6: Traverse the abstract syntax tree and find a second related node, and generate the input table / output table according to the second related node.

[0073] In this embodiment, the second related node may be a node in the abstract syntax tree that is related to data reading and writing operations; after the input table / output table is generated according to the second related node, step S2-9 is executed.

[0074] In an optional embodiment, the second related nodes include at least a read node, a write node, an expression node, a function call node, and a variable node. Among them, the read node represents the operation of reading data from the input table or data source; this node usually contains information about the table or data source being read, such as the table name, file path, connection information, etc. The write node represents the operation of writing data to the output table or target data source; this node contains information about the table or data source being written, such as the table name, file path, connection information, etc. Expression nodes represent expressions that operate or process the input table data; these nodes can contain logic for filtering, converting, aggregating, and other operations on columns in the input table. Function call nodes represent operations that call specific functions or methods to process the input table data; these nodes can contain some specific data operations, conversions, or calculations on the input table. Variable nodes represent operations that reference specific variables or columns in the input table; these nodes are used to represent data fields or columns in the input table.

[0075] S2-7: Call the underlying computing engine to query and parse the execution information of the task instance.

[0076] This embodiment uses the underlying computing engine to query and parse the execution information of the task instance. Specifically, after establishing a connection with the underlying computing engine, the task instance is parsed through specific instructions.

[0077] S2-8: Generate input table / output table according to execution information.

[0078] Each step in the execution information provides relevant information, such as input tables, output tables, and join operations. By analyzing the execution information, you can understand the tables involved in the task instance and the operations between them.

[0079] When analyzing execution information, you may need to focus on specific operators (such as JOIN, UNION, etc.) and the operator's input and output tables to understand the input and output of the task instance. After generating the input table / output table, execute step S2-9.

[0080] S2-9: Construct data lineage information based on input table / output table.

[0081] In this embodiment, constructing data lineage information based on the input table / output table includes: storing data lineage information through a relational database and / or a graph database. Specifically, the relevant information of the input table / output table usually includes the table name, field, data type, source, etc. This embodiment uses such information to analyze the logic, data conversion rules or other metadata information of the task instance, and determines the relationship between the input table / output table, thereby obtaining the data lineage information. After obtaining the data lineage analysis results, this embodiment stores the obtained data lineage information in a database for use. Among them, the scheduling system selects one or more methods of relational databases and graph databases for storage according to specific needs and scenarios.

[0082] The data lineage analysis method provided in any embodiment of the present invention is not intended to limit the present invention and is described below using an SQL task instance (representing a task instance based on a database management and operation language), a log task instance, and a Spark task instance (representing a task instance based on a non-database management and operation language) as examples.

[0083] This embodiment selects a corresponding parsing method based on the parameters of the task instance, wherein the parsing method includes at least a first parsing method for task instances based on non-database management and operation languages, a second parsing method for task instances based on database management and operation languages, and a third parsing method for task instances based on logs; wherein the database management and operation languages ​​include at least SQL-type languages.

[0084] Specifically, for the three task instances illustrated in this embodiment, Spark task instance, SQL task instance and log task instance, according to the type of task instance, the first parsing method, the second parsing method and the third parsing method will be used for parsing respectively.

[0085] In an optional embodiment, the method for parsing a Spark task instance using the first parsing method is as follows: the logical execution plan generated during the execution of the Spark engine is obtained by registering the Event Listener of the Spark platform. Specifically, during the execution of the Spark task, a custom Event Listener can be registered to monitor events generated by the Spark engine, which include the generation process of the logical execution plan. This embodiment can obtain the generated logical execution plan tree by registering the Listener. The Visitor pattern of the Spark platform is used to traverse the logical execution plan, obtain the first relevant node of the task, and generate the input table / output table. Specifically, this embodiment traverses the logical execution plan tree of the Spark task by applying the Visitor pattern and visits the nodes therein. During the lineage parsing process, attention is paid to the nodes related to the input and output of the task; these nodes can be data source nodes, conversion operation nodes or output nodes, etc.

[0086] In an optional embodiment, the method of parsing an SQL task instance using a second parsing method includes: using the Antlr4 tool to build a SQL Parser language parser; using the SQL Parser language parser to parse the SQL task instance into an abstract syntax tree (AST); using the Visitor mode to traverse the abstract syntax tree, find the second related node, and generate an input table / output table.

[0087] In an optional embodiment, a method for parsing a log task instance using the third parsing method includes: utilizing the Presto underlying computing engine and using the EXPLAIN command to analyze and query execution information, where the execution information includes the execution details of the Query, to obtain the analysis results of the Query; parsing the analysis log and generating an input table / output table.

[0088] In an optional embodiment, in the above steps, the mutual generation relationship of the input / output columns is determined based on the analysis of the attributes of the input / output nodes, and the data lineage information is constructed. Specifically, once the nodes related to the task input / output are determined, it is necessary to analyze the attributes of these nodes, such as column names, column types, etc. By analyzing the attributes, the mutual generation relationship between data columns can be determined, that is, which columns are generated by other columns. Based on these generation relationships, a data lineage graph can be constructed for storage to represent the lineage relationship between data. In an optional embodiment, the storage of data lineage information adopts a combination of mature relational databases (PostgreSQL), graph databases (such as Neo4j) and full-text search engines (ElasticSearch), which can meet the query and search requirements of different types.

[0089] It should be understood that the above task examples are only intended to illustrate the implementation steps of the present invention, and the method provided by the present invention can actually be applied to a wider range of task examples.

[0090] The present invention also provides a data lineage analysis system for executing the method of any of the above embodiments. The data lineage analysis system can select a corresponding analysis method based on the type of task instance, and the corresponding analysis methods are independently designed as lineage analysis plug-ins.

[0091] In an optional embodiment, the parsing algorithms, logic, or rules required to parse a specific task type are designed into a plug-in. This parsing logic may involve code, scripts, instructions, configuration files, etc. Furthermore, the plug-in design will consider the data processing characteristics of the specific task type, such as data conversion, data migration, and data analysis, to ensure the accuracy and completeness of the parsing.

[0092] In an optional embodiment, the lineage parsing logic for different task types can be designed as independent plug-ins, so that the system has a good modular structure. When a new task type is introduced, it can be expanded by writing a new plug-in without modifying the entire parsing system. At the same time, each plug-in can customize the parsing logic according to the characteristics and requirements of a specific task type, so that the input table and output table of the task can be parsed more accurately, providing more accurate lineage relationships. In an optional embodiment, the plug-in can interact with the scheduling system through a unified interface, so that the parsing results of different task types can be processed and stored in a unified format and manner, thereby providing consistent lineage information output and usage methods.

[0093] The present invention also provides a scheduling system, which utilizes the data lineage parsing method provided by any of the above embodiments or the data lineage parsing system provided by any of the above embodiments to parse the data lineage information of the task instance, thereby completing system scheduling.

[0094] The scheduling system provided by the present invention is applicable to various scenarios requiring task scheduling and task dependency management, such as big data processing, distributed computing, data warehouse construction, etc., and is particularly applicable to cloud server environments. It can help manage and track the lineage relationships between tasks, and accurately determine the dependencies and data flows between task instances. This is of great significance for task scheduling, performance optimization, troubleshooting, and data tracing. In addition, the present invention closely integrates lineage analysis with the scheduling system, and can automatically construct the complete lineage relationship of the entire data management system.

[0095] Refer to FIG3 , which is a flowchart illustrating a scheduling system determining task instance dependency rules according to an embodiment of the present invention.

[0096] In this embodiment, the scheduling system is responsible for creating, scheduling, and monitoring tasks, determining when tasks run and their execution cycles. Task instances are created by the scheduling system based on the scheduling cycle. This invention establishes dependencies between tasks by analyzing their dependency rules. Task instances are then created and executed in the correct order and according to their dependencies. This ensures the correct execution order of tasks, avoids data loss or errors, and improves the reliability and efficiency of the entire task process.

[0097] Specifically, after creating a task instance (such as step S3-1), in order to determine the upstream task instance on which the task instance depends, the scheduling system will parse the dependency rules of the task instance (such as step S3-2). When all the upstream dependencies of the task instance are satisfied (that is, all the dependent task instances are completed), the scheduling system will start running the current task instance. For the avoidance of doubt, the dependencies between tasks and the blood relationship between data described in the present invention are independent of each other (although there is a close relationship between the two). The dependencies only define the execution order between tasks (specifically, task instances), and the blood relationship analysis obtains the derivative relationship between the data produced by the tasks. The various dependency rules mentioned in the present invention all refer to the dependencies between tasks. The dependency of tasks can be different from the dependency of blood relationship. For the scheduling system, it will only schedule tasks based on the dependency of tasks.

[0098] When the dependency rules of the task instance include data-based dependencies, the scheduling system parses the dependency rules of the task instance according to the data lineage information (e.g., step S3-3). Specifically, the dependency rules of this embodiment include at least data-based dependency rules. Among them, data-based dependencies include the relationship between tasks and data. For example, some tasks may need to use data generated by other tasks as input, so it is necessary to ensure the availability of data before these tasks are executed; in this case, the scheduling system will use the data lineage parsing method or system provided by the present invention to analyze the data lineage information between tasks and determine the upstream task instance on which the current task instance depends to ensure the correct availability of data.

[0099] More specifically, when creating a task instance, the scheduling system simultaneously determines the data lineage resolution method for that task instance. The scheduling system includes at least one task scheduler and a dependency resolution engine. The task scheduler is used to schedule the system throughout its lifecycle, including resource allocation, dependency resolution, status tracking, log collection, error analysis, and alerting. The dependency resolution engine, scheduled by the task scheduler, resolves task dependency rules and determines the final task dependencies. In an optional embodiment, the task scheduler can be implemented using a known scheduling framework, such as Argo Workflows or Prefect.

[0100] In an optional embodiment, the scheduling system may optionally include a task execution monitoring mechanism to capture the execution process and related information of the task instance, wherein the monitoring may be implemented through log recording, event monitoring, hook functions, etc.

[0101] The present invention has been introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The above implementation description is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. Changes and improvements to the present invention will be possible without exceeding the concept and scope specified in the appended claims. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A data lineage analysis method, characterized in that: The method is used to parse data lineage information of a task instance, including: Monitoring the state of the task instance and selecting a corresponding parsing method according to the parameters of the task instance; the parameters at least include the type of the task instance; Parse the task instance using the parsing method to obtain an input table / output table; Data lineage information is constructed based on the input table / output table.

2. The method according to claim 1, characterized in that The parsing method includes at least a first parsing method for a task instance based on a non-database management and operation language; the database management and operation language includes at least a SQL-like language.

3. The method according to claim 2, characterized in that The first parsing method comprises: Generate a logical execution plan tree according to the state of the task instance; The logic execution plan tree is traversed to find a first related node, and the input table / output table is generated according to the first related node.

4. The method according to any one of claims 1 to 3, characterized in that: The parsing method includes at least a second parsing method for a task instance based on the database management and operation language.

5. The method according to claim 4, characterized in that The second parsing method comprises: Build language parsers; Parsing the task instance into an abstract syntax tree (AST) using the language parser; The abstract syntax tree (AST) is traversed to find a second related node, and the input table / output table is generated according to the second related node.

6. The method according to any one of claims 1 to 5, characterized in that: The parsing method includes a third parsing method for a task instance of a log, and the third parsing method includes: Calling the underlying computing engine to query and parse the execution information of the task instance; The input table / output table is generated according to the execution information.

7. The method according to any one of claims 1 to 6, characterized in that: The constructing of data lineage information according to the input table / output table includes: storing the data lineage information through a relational database and / or a graph database.

8. A data lineage analysis system, characterized in that: The data lineage analysis system is used to execute the method described in any one of claims 1-7; wherein, the data lineage analysis system selects a corresponding analysis method according to the type of the task instance, and the corresponding analysis methods are each independently designed in the form of a lineage analysis plug-in.

9. A scheduling system using the method according to any one of claims 1 to 7 or the data lineage analysis system according to claim 8, characterized in that: The scheduling system uses the method or the data lineage parsing system to parse the data lineage information of the task instance.

10. The dispatching system according to claim 9, characterized in that: When the dependency rule of the task instance includes a dependency based on data, the scheduling system parses the dependency rule of the task instance according to the data lineage information.

Images