Data lineage analysis method, apparatus, and computer-readable storage medium

Abstract

The application discloses a data blood relationship analysis method and device and a computer readable storage medium. The method comprises the following steps: obtaining job blood relationship data based on a directed graph corresponding to a scheduling task in a scheduling system, and obtaining a first application program identifier based on an execution log corresponding to the scheduling task; when a structured query language (SQL) statement is detected, obtaining field blood relationship data corresponding to the SQL statement and a second application program identifier corresponding to the field blood relationship data; determining a job blood relationship subgraph based on the job blood relationship data and a Neo4j point-edge model, and determining a table field blood relationship subgraph based on the Neo4j point-edge model and the field blood relationship data; and connecting the job blood relationship subgraph and the table field blood relationship subgraph based on the first application program identifier and the second application program identifier. The application can obtain blood relationship data of different types of data systems, adapt the data blood relationship analysis method to different types of data systems, and improve the expansibility of the data blood relationship analysis method.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data lineage analysis method, apparatus, and computer-readable storage medium. Background Technology

[0002] Data lineage analysis is a core function of metadata management and data governance tools. By establishing lineage relationships between data, it's possible to analyze whether changes in upstream data affect related downstream data. Furthermore, by establishing relationships between technical and business metadata, lineage analysis can reveal data flow between different business products and their business relationships. Analyzing data lineage allows for a better understanding and use of data. Currently, Shizhu's MetaOne supports data lineage analysis. MetaOne constructs data lineage chains by parsing SQL (Structured Query Language) scripts. Based on common SQL syntax standards, it decomposes SQL operations into specific operation types and then further breaks down SQL statement modules to obtain the input and output data of a single SQL statement, thus establishing a direct lineage relationship between the input and output data.

[0003] Because MetaOn can only adapt to the general SQL syntax standard, it cannot adapt to the lineage analysis of data between different types of data systems. For example, the open-source data warehouse tool Hive and the SQL engine Spark-SQL have their own syntax characteristics, so MetaOn cannot analyze the lineage relationship of the corresponding data in Hive and Spark-SQL.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a data lineage analysis method, apparatus, and computer-readable storage medium, aiming to solve the technical problem that existing data lineage analysis methods cannot be adapted to different types of data systems.

[0006] To achieve the above objectives, the present invention provides a data lineage analysis method, which includes the following steps:

[0007] Based on the directed graph corresponding to the scheduling task in the scheduling system, obtain job lineage data, and based on the execution log corresponding to the scheduling task, obtain the first application identifier.

[0008] When a Structured Query Language (SQL) statement is detected, the field lineage data corresponding to the SQL statement and the second application identifier corresponding to the field lineage data are obtained.

[0009] Based on the job lineage data and the Neo4j vertex-edge model, a job lineage subgraph is determined, and based on the Neo4j vertex-edge model and the field lineage data, a table field lineage subgraph is determined.

[0010] Based on the first application identifier and the second application identifier, connect the job lineage subgraph and the table field lineage subgraph.

[0011] Furthermore, the steps of obtaining job lineage data based on the directed graph corresponding to the scheduling task in the scheduling system, and obtaining the first application identifier based on the log data corresponding to the scheduling task, include:

[0012] Obtain the directed graph corresponding to the scheduled task from the message queue, and obtain job lineage data based on the directed graph;

[0013] Retrieve the log data corresponding to the scheduled task from the Kafka queue, and obtain the first application identifier based on the log data.

[0014] Furthermore, the step of obtaining the directed graph corresponding to the scheduled task in the message queue, and obtaining job lineage data based on the directed graph, includes:

[0015] When job change information is detected, the system obtains the directed graph of the scheduling task corresponding to the job change information and adds the directed graph to the message queue.

[0016] The directed graph is retrieved from the message queue and mapped to the job lineage data.

[0017] Further, the step of obtaining the log data corresponding to the scheduled task from the Kafka queue and obtaining the first application identifier based on the log data includes:

[0018] When the scheduled task is completed, the log data corresponding to the scheduled task is written to the Kafka queue;

[0019] The log service retrieves the log data corresponding to the scheduled task from the Kafka queue, and extracts the first application identifier from the log data.

[0020] Further, the step of obtaining the field lineage data corresponding to the SQL statement and the second application identifier corresponding to the field lineage data includes:

[0021] The lineage data of the field corresponding to the SQL statement is obtained through a hook.

[0022] Obtain the second application identifier corresponding to the bloodline data in the field.

[0023] Furthermore, the step of obtaining the lineage data of the field corresponding to the SQL statement through a hook includes:

[0024] Parse the SQL statement to obtain the database name, table name, and field name in the SQL statement;

[0025] Based on the SQL statement, obtain the field dependencies corresponding to the database name, table name, and field name;

[0026] Based on preset filtering conditions, the field dependencies are filtered, and the field lineage data is determined based on the filtered field dependencies.

[0027] Furthermore, the steps of determining the job lineage subgraph based on the job lineage data and the Neo4j vertex-edge model, and determining the table field lineage subgraph based on the Neo4j vertex-edge model and the field lineage data, include:

[0028] Write the job lineage data into the message queue, and write the field lineage data into the message queue;

[0029] Consume the job lineage data in the message queue, and determine the job lineage subgraph corresponding to the job lineage data through the Neo4j vertex-edge model;

[0030] Consume the field lineage data in the message queue, and determine the table field lineage subgraph corresponding to the field lineage data through the Neo4j point-edge model.

[0031] Furthermore, after the step of connecting the job lineage subgraph and the table field lineage subgraph based on the first application identifier and the second application identifier, the data lineage analysis method further includes:

[0032] Write the connected job lineage subgraph and table field lineage subgraph into the preset database;

[0033] The bloodline data in the preset database is analyzed and statistically analyzed to construct a data asset map.

[0034] Furthermore, to achieve the above objectives, the present invention also provides a data lineage analysis device, the data lineage analysis device comprising:

[0035] A memory, a processor, and a data lineage analysis program stored in the memory and executable on the processor, wherein the data lineage analysis program, when executed by the processor, implements the steps of the aforementioned data lineage analysis method.

[0036] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data lineage analysis program, which, when executed by a processor, implements the steps of the aforementioned data lineage analysis method.

[0037] This invention obtains job lineage data based on the directed graph corresponding to the scheduled tasks in the scheduling system, and obtains a first application identifier based on the execution log of the scheduled tasks. Then, when a Structured Query Language (SQL) statement is detected, the invention obtains the field lineage data corresponding to the SQL statement and a second application identifier corresponding to the field lineage data. Next, based on the job lineage data and the Neo4j vertex-edge model, a job lineage subgraph is determined, and based on the Neo4j vertex-edge model and the field lineage data, a table field lineage subgraph is determined. Finally, based on the first application identifier and the second application identifier, the job lineage subgraph and the table field lineage subgraph are connected. This allows for the acquisition of lineage data from different types of data systems, making the data lineage analysis method adaptable to different types of data systems. Furthermore, by establishing the association between the job lineage subgraph and the table field lineage subgraph, the invention improves the convenience of subsequent lineage data analysis, reduces the complexity of analyzing data lineage relationships, and enhances the scalability of the data lineage analysis method. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the data lineage analysis device in the hardware operating environment involved in the embodiments of the present invention;

[0039] Figure 2 This is a flowchart illustrating the data lineage analysis method or the first embodiment of the present invention.

[0040] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0041] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0042] like Figure 1 As shown, Figure 1 This is a schematic diagram of the data lineage analysis device in the hardware operating environment involved in the embodiments of the present invention.

[0043] The data lineage analysis device in this embodiment of the invention can be a PC, or a mobile terminal device with display function such as a smartphone, tablet computer, or portable computer.

[0044] like Figure 1 As shown, the data lineage analysis device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0045] Optionally, the data lineage analysis device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. Of course, the data lineage analysis device may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated upon here.

[0046] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the data lineage analysis device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0047] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data lineage analysis program.

[0048] exist Figure 1 In the data lineage analysis device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate with the client; and the processor 1001 can be used to call the data lineage analysis program stored in the memory 1005.

[0049] In this embodiment, the data lineage analysis device includes: a memory 1005, a processor 1001, and a data lineage analysis program stored in the memory 1005 and executable on the processor 1001. When the processor 1001 calls the data lineage analysis program stored in the memory 1005, it executes the steps of the data lineage analysis methods in the following embodiments.

[0050] This invention also provides a data lineage analysis method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the data lineage analysis method of the present invention.

[0051] In this embodiment, the data lineage analysis method includes:

[0052] Step S101: Based on the directed graph corresponding to the scheduling task in the scheduling system, obtain job lineage data; based on the execution log corresponding to the scheduling task, obtain the first application identifier.

[0053] In this embodiment, the data lineage analysis method can be applied to a data lineage analysis device, which may include a collection layer, a summarization layer, and an analysis layer. The collection layer is responsible for collecting lineage data and encapsulating the collected lineage data into a unified format before sending it to a message queue (MQ). The collection of lineage data includes the collection of job lineage data.

[0054] In this embodiment, when job change information (such as job addition, job modification, etc.) is detected through WEBUI, the acquisition layer obtains job lineage data based on the directed graph corresponding to the scheduling task in the scheduling system. Specifically, the scheduling task corresponding to the job change information is obtained in the scheduling system, the directed graph (scheduling task DAG) corresponding to the scheduling task is obtained, and the directed graph is mapped to job lineage data.

[0055] Simultaneously, monitor the execution flow of the scheduled task, and when the scheduled task is completed, obtain the execution log corresponding to the scheduled task, and retrieve the first application identifier from the execution log.

[0056] Step S102: When a Structured Query Language (SQL) statement is detected, obtain the field lineage data corresponding to the SQL statement and the second application identifier corresponding to the field lineage data.

[0057] In this embodiment, the collection of lineage data also includes the collection of field lineage data. Specifically, when a Structured Query Language (SQL) statement is detected, the collection layer obtains the field lineage data corresponding to the SQL statement and the second application identifier corresponding to the field lineage data. The SQL statement can be the SQL statement corresponding to a scheduling task. In this case, the second application identifier is the same as the first application identifier, so as to collect field lineage and context parameters through the extended SQL engine parsing optimization method.

[0058] Step S103: Based on the job lineage data and the Neo4j vertex-edge model, determine the job lineage subgraph, and based on the Neo4j vertex-edge model and the field lineage data, determine the table field lineage subgraph.

[0059] In this embodiment, after obtaining the job lineage data and table field lineage data, the aggregation layer cleans and standardizes the lineage data. Specifically, based on the job lineage data and the Neo4j vertex-edge model, a job lineage subgraph is determined, and based on the Neo4j vertex-edge model and the field lineage data, a table field lineage subgraph is determined. Specifically, Neo4j is used to construct the job lineage subgraph based on the job lineage data, and Neo4j is used to construct the table field lineage subgraph based on the field lineage data.

[0060] Step S104: Based on the first application identifier and the second application identifier, connect the job lineage subgraph and the table field lineage subgraph.

[0061] In this embodiment, after obtaining the job lineage subgraph and the table field lineage subgraph, the aggregation layer connects the job lineage subgraph and the table field lineage subgraph based on the first application identifier and the second application identifier. Specifically, the job lineage subgraph and the table field lineage subgraph are connected based on the same application identifier in the first application identifier and the second application identifier, so as to establish the association relationship between the job lineage subgraph and the table field lineage subgraph.

[0062] The lineage collection method in this embodiment is applicable to the lineage collection and construction of SQL-type big data components such as HIVE, SPARK, FLINK, and CLICKHOUSE.

[0063] The data lineage analysis method proposed in this embodiment obtains job lineage data based on the directed graph corresponding to the scheduling tasks in the scheduling system, and obtains a first application identifier based on the execution log corresponding to the scheduling tasks. Then, when a Structured Query Language (SQL) statement is detected, the method obtains the field lineage data corresponding to the SQL statement and the second application identifier corresponding to the field lineage data. Next, based on the job lineage data and the Neo4j vertex-edge model, a job lineage subgraph is determined, and based on the Neo4j vertex-edge model and the field lineage data, a table field lineage subgraph is determined. Finally, based on the first application identifier and the second application identifier, the job lineage subgraph and the table field lineage subgraph are connected. This method can obtain lineage data from different types of data systems, making the data lineage analysis method adaptable to different types of data systems. Furthermore, by establishing the association between the job lineage subgraph and the table field lineage subgraph, the method improves the convenience of subsequent lineage data analysis, reduces the complexity of analyzing data lineage relationships, and enhances the scalability of the data lineage analysis method.

[0064] Based on the first embodiment, a second embodiment of the data lineage analysis method of the present invention is proposed. In this embodiment, step S101 includes:

[0065] Step S201: Obtain the directed graph corresponding to the scheduled task from the message queue, and obtain job lineage data based on the directed graph;

[0066] Step S202: Obtain the log data corresponding to the scheduled task from the Kafka queue, and obtain the first application identifier based on the log data.

[0067] In this embodiment, when collecting job lineage data, a directed graph corresponding to the scheduled task is obtained from the message queue, and job lineage data is obtained based on the directed graph. Further, in one possible implementation, step S201 includes:

[0068] Step S2011: When job change information is detected, the directed graph of the scheduling task corresponding to the job change information is obtained from the scheduling system, and the directed graph is added to the message queue.

[0069] Step S2012: Obtain the directed graph from the message queue and map the directed graph to the job lineage data.

[0070] In this embodiment, when job change information (such as job addition, job modification, etc.) is detected through the WEBUI, the scheduling system generates a scheduling task based on the job change information and updates the directed graph (scheduling task DAG) corresponding to the scheduling task. The scheduling system uploads the directed graph to the message queue, obtains the directed graph corresponding to the scheduling task by consuming the message queue, and determines the job lineage data based on the directed graph corresponding to the scheduling task. That is, the directed graph is mapped, and the mapped data is encapsulated into job lineage data in JSON format to accurately obtain job lineage data. This method of collecting job lineage data is suitable for the lineage collection of SQL-type big data components such as HIVE, SPARK, FLINK, and CLICKHOUSE, realizing comprehensive big data lineage data collection and making the data lineage analysis method adaptable to different types of data systems.

[0071] In this embodiment, log data corresponding to the scheduled task is simultaneously obtained from the Kafka queue, and the first application identifier is obtained based on the log data. Further, in another possible implementation, step S202 includes:

[0072] Step S2021: When the scheduling task is completed, write the log data corresponding to the scheduling task into the Kafka queue.

[0073] Step S2022: Obtain the log data corresponding to the scheduled task from the Kafka queue through the log service, and extract the first application identifier from the log data.

[0074] In this embodiment, the scheduling system sends the scheduling task to the scheduling worker to execute the scheduling task to complete the corresponding job, obtains the log file (log data corresponding to the scheduling task), and writes the log data corresponding to the scheduling task into the Kafka queue. The log service consumes the Kafka queue to obtain the log data corresponding to the scheduling task, and extracts the first application identifier from the log data to accurately obtain the first application identifier through the log data.

[0075] It should be noted that job metadata can also be saved to the Neo4j point-edge model through log data.

[0076] The data lineage analysis method proposed in this embodiment obtains the directed graph corresponding to the scheduled task from the message queue, and obtains job lineage data based on the directed graph; then, it obtains the log data corresponding to the scheduled task from the Kafka queue, and obtains the first application identifier based on the log data. This method can accurately obtain job lineage data and the corresponding first application identifier. This method of collecting job lineage data is applicable to the lineage collection of SQL-type big data components such as HIVE, SPARK, FLINK, and CLICKHOUSE, realizing comprehensive big data lineage data collection and making the data lineage analysis method adaptable to different types of data systems.

[0077] Based on the first embodiment, a third embodiment of the data lineage analysis method of the present invention is proposed. In this embodiment, step S102 includes:

[0078] Step S301: Obtain the lineage data of the field corresponding to the SQL statement through a hook.

[0079] Step S302: Obtain the second application identifier corresponding to the bloodline data in the field.

[0080] In this embodiment, when a Structured Query Language (SQL) statement is detected, the lineage data of the fields corresponding to the SQL statement is obtained through a hook. Specifically, in one possible implementation, step S301 includes:

[0081] Step S3011: Parse the SQL statement to obtain the database name, table name, and field name in the SQL statement;

[0082] Step S3012: Based on the SQL statement, obtain the field dependencies corresponding to the database name, table name, and field name;

[0083] Step S3013: Based on preset filtering conditions, filter the field dependencies and determine the field lineage data based on the filtered field dependencies.

[0084] In this embodiment, the data collection process for field lineage data includes: Parser parses SQL to generate AST, Analyzer optimizes and analyzes AST to generate Query Block, and Logical Plan Generator processes Query Block to generate OperatorTree, so as to extend the SQL engine parsing and optimize the execution process through Hook.

[0085] This involves parsing the SQL statement to obtain the database name, table name, and field name in the SQL statement. Specifically, the SQL statement is parsed from the Parser to generate an AST, and the database name, table name, and field name contained in the PLAN are parsed from the Catalog.

[0086] Next, based on the SQL statement, the field dependencies corresponding to the database name, table name, and field name are obtained; specifically, the Aggregate and Project are parsed to obtain the field dependencies corresponding to the database name, table name, and field name.

[0087] Then, based on preset filtering conditions, the field dependencies are filtered. These preset filtering conditions include whether the SQL statements contain select or insert operators. The field dependencies corresponding to SQL statements that do not contain select or insert operators are filtered to obtain the filtered field dependencies. The field lineage data is then determined based on the filtered field dependencies by encapsulating the filtered field dependencies in JSON format to obtain the field lineage data.

[0088] Simultaneously, obtain the second application identifier corresponding to the lineage data of the field. For example, extract the application identifier of YARN from sparkContext to obtain the second application identifier.

[0089] The data lineage analysis method proposed in this embodiment obtains the field lineage data corresponding to the SQL statement through a hook; then, it obtains the second application identifier corresponding to the field lineage data. This method can accurately obtain the field lineage data and the corresponding second application identifier. This field lineage data collection method is applicable to the lineage collection of SQL-type big data components such as HIVE, SPARK, FLINK, and CLICKHOUSE, realizing comprehensive big data lineage data collection and making the data lineage analysis method adaptable to different types of data systems.

[0090] Based on the first embodiment, a fourth embodiment of the data lineage analysis method of the present invention is proposed. In this embodiment, step S103 includes:

[0091] Step S401: Write the job lineage data into the message queue, and write the field lineage data into the message queue;

[0092] Step S402: Consume the job lineage data in the message queue, and determine the job lineage subgraph corresponding to the job lineage data through the Neo4j point-edge model;

[0093] Step S403: Consume the field lineage data in the message queue, and determine the table field lineage subgraph corresponding to the field lineage data through the Neo4j point-edge model.

[0094] In this embodiment, when the job lineage data and field lineage data are obtained, the acquisition layer writes the job lineage data into the message queue and the field lineage data into the message queue.

[0095] The aggregation layer consumes the job lineage data in the message queue and determines the job lineage subgraph corresponding to the job lineage data using the Neo4j vertex-edge model. Simultaneously, the aggregation layer consumes the field lineage data in the message queue and determines the table field lineage subgraph corresponding to the field lineage data using the Neo4j vertex-edge model.

[0096] The data lineage analysis method proposed in this embodiment writes the job lineage data and the field lineage data into a message queue. Then, it consumes the job lineage data from the message queue and uses the Neo4j vertex-edge model to determine the job lineage subgraph corresponding to the job lineage data. Next, it consumes the field lineage data from the message queue and uses the Neo4j vertex-edge model to determine the table field lineage subgraph corresponding to the field lineage data. By consuming the job lineage data and field lineage data from the message queue, the construction of the job lineage subgraph and the table field lineage subgraph can be achieved, further improving the efficiency of lineage data analysis.

[0097] Based on the above embodiments, a fifth embodiment of the data lineage analysis method of the present invention is proposed. In this embodiment, after step S104, the data lineage analysis method further includes:

[0098] Step S501: Write the connected job lineage subgraph and table field lineage subgraph into the preset database;

[0099] Step S502: Analyze and statistically analyze the bloodline data in the preset database to construct a data asset map.

[0100] In this embodiment, when the connected job lineage subgraph and table field lineage subgraph are obtained, the connected job lineage subgraph and table field lineage subgraph are written into a preset database.

[0101] The analysis layer can obtain lineage data from a preset database, analyze and statistically process the lineage data in the preset database, specifically use existing graph algorithms to analyze the feature data of the lineage link graph, translate the graph feature data into intuitive business data, construct a data asset map, and improve the observability of the lineage link representation.

[0102] The data lineage analysis method proposed in this embodiment writes the connected job lineage subgraph and table field lineage subgraph into a preset database; then, it analyzes and statistically analyzes the lineage data in the preset database to construct a data asset map, thereby improving the observability of lineage link representation through the data asset map.

[0103] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a data lineage analysis program, wherein when the data lineage analysis program is executed by the processor, it implements the steps of the data lineage analysis method as described above.

[0104] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0105] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0107] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data lineage analysis method, characterized in that, The data lineage analysis method includes the following steps: Based on the directed graph corresponding to the scheduling task in the scheduling system, obtain job lineage data, and based on the execution log corresponding to the scheduling task, obtain the first application identifier. When a Structured Query Language (SQL) statement is detected, the field lineage data corresponding to the SQL statement and the second application identifier corresponding to the field lineage data are obtained. Based on the job lineage data and the Neo4j vertex-edge model, a job lineage subgraph is determined, and based on the Neo4j vertex-edge model and the field lineage data, a table field lineage subgraph is determined. Based on the first application identifier and the second application identifier, connect the job lineage subgraph and the table field lineage subgraph; The steps of determining the job lineage subgraph based on the job lineage data and the Neo4j vertex-edge model, and determining the table field lineage subgraph based on the Neo4j vertex-edge model and the field lineage data, include: Write the job lineage data into the message queue, and write the field lineage data into the message queue; Consume the job lineage data in the message queue, and determine the job lineage subgraph corresponding to the job lineage data through the Neo4j vertex-edge model; Consume the field lineage data in the message queue, and determine the table field lineage subgraph corresponding to the field lineage data through the Neo4j point-edge model.

2. The data lineage analysis method as described in claim 1, characterized in that, The steps of obtaining job lineage data based on the directed graph corresponding to the scheduling tasks in the scheduling system, and obtaining the first application identifier based on the log data corresponding to the scheduling tasks, include: Obtain the directed graph corresponding to the scheduled task from the message queue, and obtain job lineage data based on the directed graph; Retrieve the log data corresponding to the scheduled task from the Kafka queue, and obtain the first application identifier based on the log data.

3. The data lineage analysis method as described in claim 2, characterized in that, The steps of obtaining the directed graph corresponding to the scheduled task in the message queue and obtaining job lineage data based on the directed graph include: When job change information is detected, the system obtains the directed graph of the scheduling task corresponding to the job change information and adds the directed graph to the message queue. The directed graph is retrieved from the message queue and mapped to the job lineage data.

4. The data lineage analysis method as described in claim 2, characterized in that, The steps of obtaining the log data corresponding to the scheduled task from the Kafka queue and obtaining the first application identifier based on the log data include: When the scheduled task is completed, the log data corresponding to the scheduled task is written to the Kafka queue; The log service retrieves the log data corresponding to the scheduled task from the Kafka queue, and extracts the first application identifier from the log data.

5. The data lineage analysis method as described in claim 1, characterized in that, The step of obtaining the field lineage data corresponding to the SQL statement and the second application identifier corresponding to the field lineage data includes: The lineage data of the field corresponding to the SQL statement is obtained through a hook. Obtain the second application identifier corresponding to the bloodline data in the field.

6. The data lineage analysis method as described in claim 5, characterized in that, The steps for obtaining the lineage data of the field corresponding to the SQL statement through a hook include: Parse the SQL statement to obtain the database name, table name, and field name in the SQL statement; Based on the SQL statement, obtain the field dependencies corresponding to the database name, table name, and field name; Based on preset filtering conditions, the field dependencies are filtered, and the field lineage data is determined based on the filtered field dependencies.

7. The data lineage analysis method according to any one of claims 1 to 6, characterized in that, Following the step of connecting the job lineage subgraph and the table field lineage subgraph based on the first application identifier and the second application identifier, the data lineage analysis method further includes: Write the connected job lineage subgraph and table field lineage subgraph into the preset database; The bloodline data in the preset database is analyzed and statistically analyzed to construct a data asset map.

8. A data lineage analysis device, characterized in that, The data lineage analysis device includes: a memory, a processor, and a data lineage analysis program stored in the memory and executable on the processor. When the data lineage analysis program is executed by the processor, it implements the steps of the data lineage analysis method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data lineage analysis program, which, when executed by a processor, implements the steps of the data lineage analysis method as described in any one of claims 1 to 7.

Images