Method, system, device and medium for constructing multi-source heterogeneous data lineage relationship

Abstract

The present application relates to a method, system, device and medium for constructing multi-source heterogeneous data lineage relationships. The method designs a unified data lineage relationship construction standard based on DAG plug-in orchestration. First, the DAG drag-and-drop method is used to select data processing plug-ins and configure plug-in parameters. Then, the plug-in relationship is dynamically orchestrated according to the defined data plug-in relationship dynamic orchestration model. The obtained plug-in attribute information is saved to the Task task configuration. Then, the data lineage entity combination sample model is used to parse the data lineage relationship entities. The data lineage entity relationship model is used to parse the entity relationship of the data lineage relationship entities, and then the data lineage is associated with the task and bound. Finally, a data lineage relationship conversion model based on BFS is used to find all data lineage relationship entities and entity lineage relationships from the entity relationship table through a data lineage relationship traversal algorithm to generate a data lineage map.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology and relates to a method, system, device and medium for constructing multi-source heterogeneous data kinship relationships. Background Art

[0002] In the Internet of Things (IoT), industrial equipment types are growing, and system architectures are becoming increasingly complex, exhibiting multi-source heterogeneity. This is particularly true in data governance for complex scenarios, where numerous processing rules and disorganized processes make it difficult to trace abnormal data and ensure data quality. Establishing data lineage relationships is a key step in data governance, and a unified and effective method for establishing data lineage relationships is a crucial prerequisite for achieving multi-source heterogeneous data governance.

[0003] At present, many scholars have conducted in-depth research on methods for constructing data lineage relationships in multi-source heterogeneous systems and have achieved certain research results. For example, traditional technologies such as constructing data lineage relationships by annotating and labeling different types of operators and parsing dependencies, constructing data lineage relationships by parsing the original lineage of SQL statements through parsing functions, constructing data lineage relationships by using tree-shaped syntax trees to construct data lineage relationships for query tables, and constructing data lineage relationships by pre-configuring icon data sources and attributes. However, when faced with the task of constructing complex multi-source heterogeneous data lineage relationships for a large number of industrial equipment, the aforementioned traditional technologies still have the technical problem of excessively high data processing complexity. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned traditional methods, the present invention proposes a method for constructing multi-source heterogeneous data lineage relationships, a system for constructing multi-source heterogeneous data lineage relationships, a computer device and a computer-readable storage medium, which can greatly reduce the data processing complexity of data lineage relationship construction tasks.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] On the one hand, a method for constructing a kinship relationship of multi-source heterogeneous data is provided, comprising the steps of:

[0007] Use the DAG drag-and-drop method to select input plug-ins, data processing plug-ins, and output plug-ins, and configure plug-in parameters. For input plug-ins, select the input data source plug-in, including the Kafka data source plug-in or the MySQL data source plug-in. For data processing plug-ins, select the data processing process plug-in, including the SQL statement plug-in, the data filtering plug-in, the rule calculation plug-in, the field conversion plug-in, or the date conversion plug-in. For output plug-ins, select the output target source plug-in, including the Hive data source plug-in, the DB2 data source plug-in, or the Phoenix data source plug-in.

[0008] Perform dynamic arrangement of plug-in relationships for input plug-ins, data processing plug-ins, and output plug-ins based on the defined data plug-in relationship dynamic arrangement model;

[0009] The plug-in attribute information obtained after the dynamic arrangement of the plug-in relationship is saved to the Task configuration; the plug-in attribute information includes input plug-ins, data processing plug-ins, output plug-ins, plug-in relationships, plug-in data sources, plug-in data tables, and plug-in field collections;

[0010] Utilize the data lineage entity combination sample model to parse data lineage relationship entities according to the Task configuration, bind the data lineage relationship entities to entity definition identifiers and save them to the entity element table; data lineage relationship entities include data source entities, data table entities, field entities and data processing entities, and entity definition identifiers include database identifiers, data set identifiers, field identifiers and processing process identifiers;

[0011] Utilize the data lineage entity relationship model to parse the entity relationships of data lineage relationship entities according to the task configuration, and save the entity relationships to the entity relationship table; entity lineage relationships include database table entity relationships, input table and data processing entity relationships, data processing and output table relationships, and input table and output table entity relationships;

[0012] Bind the data lineage relationship entity and entity lineage relationship to the Task task identifier and save them in the task association table;

[0013] The data lineage relationship transformation model based on BFS is used to determine the retrieval center and all data lineage relationship entities and entity lineage relationships are found from the entity relationship table through the data lineage relationship traversal algorithm.

[0014] On the other hand, a multi-source heterogeneous data lineage relationship construction system is also provided, including:

[0015] The plugin selection module is used to select input plugins, data processing plugins, and output plugins by dragging and dropping DAGs, and to configure plugin parameters. Input plugins can select input data source plugins such as Kafka or MySQL. Data processing plugins can select data processing process plugins such as SQL statement plugins, data filtering plugins, rule calculation plugins, field conversion plugins, or date conversion plugins. Output plugins can select output target source plugins such as Hive, DB2, or Phoenix.

[0016] The relationship arrangement module is used to dynamically arrange the relationship between input plug-ins, data processing plug-ins, and output plug-ins according to the defined data plug-in relationship dynamic arrangement model;

[0017] The task configuration module is used to save the plug-in attribute information obtained after the dynamic arrangement of plug-in relationships to the Task configuration; the plug-in attribute information includes input plug-ins, data processing plug-ins, output plug-ins, plug-in relationships, plug-in data sources, plug-in data tables, and plug-in field sets;

[0018] The entity parsing module is used to use the data lineage entity combination sample model to parse the data lineage relationship entities according to the Task configuration, bind the data lineage relationship entities to the entity definition identifier and save them to the entity element table; the data lineage relationship entities include data source entities, data table entities, field entities and data processing entities, and the entity definition identifier includes database identifier, data set identifier, field identifier and processing process identifier;

[0019] The relationship parsing module is used to use the data lineage entity relationship model to parse the entity relationship of data lineage relationship entities according to the task configuration and save the entity relationship to the entity relationship table; the entity lineage relationship includes the database table entity relationship, the input table and data processing entity relationship, the data processing and output table relationship, and the input table and output table entity relationship;

[0020] The bloodline binding module is used to bind data bloodline relationship entities and entity bloodline relationships to Task task identifiers and save them in the task association table;

[0021] The blood relationship search module is used to use the BFS-based data blood relationship conversion model to determine the retrieval center and find all data blood relationship entities and entity blood relationships from the entity relationship table through the data blood relationship traversal algorithm.

[0022] On the other hand, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned method for constructing the blood relationship of multi-source heterogeneous data are implemented.

[0023] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for constructing multi-source heterogeneous data bloodline relationships are implemented.

[0024] One of the above technical solutions has the following advantages and beneficial effects:

[0025] The above-mentioned multi-source heterogeneous data lineage relationship construction method, system, equipment and medium, by designing a unified data lineage relationship construction standard based on DAG plug-in orchestration, first uses the DAG drag-and-drop method to select data processing plug-ins and configure plug-in parameters, and then dynamically arranges plug-in relationships according to the defined data plug-in relationship dynamic orchestration model, saves the obtained plug-in attribute information to the Task task configuration, and then uses the data lineage entity combination sample model to perform data lineage relationship entity analysis, and uses the data lineage entity relationship model to perform entity relationship analysis of data lineage relationship entities, and then associates and binds data lineage with tasks, and finally uses the BFS-based data lineage relationship conversion model to find all data lineage relationship entities and entity lineage relationships from the entity relationship table through the data lineage relationship traversal algorithm to generate a data lineage map.

[0026] Compared with traditional technologies, the above technical solutions of this application can simultaneously support offline and real-time data processing, integrate multiple heterogeneous data sources, reduce user operation difficulty through visual interface configuration, use plug-in configuration to achieve low-code development, dynamically arrange plug-in dependencies, design blood relationship storage structure, design non-perceptual blood relationship parsing algorithm, and design binary tree breadth-first relationship traversal method BFS to achieve the generation of full life cycle data blood relationship map, reduce external tool dependence to reduce system complexity. Ultimately, the integration of data blood relationship construction in multiple links solves the problem of numerous processing processes and messy relationships, solves the problem of frequent changes in data structure and difficult metadata traceability, simplifies the system architecture to reduce system complexity, reduces user labor costs, and fundamentally solves the problem of high complexity in building multi-source heterogeneous blood relationship of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flowchart of a method for establishing a kinship relationship of multi-source heterogeneous data in one embodiment is shown;

[0029] Figure 2 A schematic diagram of data plug-in and relationship arrangement in one embodiment;

[0030] Figure 3 This is an example diagram of data plug-in and relationship arrangement in one embodiment;

[0031] Figure 4A schematic flow chart of a method for constructing kinship relationships of multi-source heterogeneous data in another embodiment;

[0032] Figure 5 Schematic diagram of a breadth-first traversal of a binary tree in one embodiment;

[0033] Figure 6 A block diagram of the module structure of a system for building a multi-source heterogeneous data lineage relationship in one embodiment. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0036] It should be noted that reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of such a phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments.

[0037] Those skilled in the art will appreciate that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the present specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0038] In the data governance process, data lineage relationship construction and graph retrieval are crucial. Although many scholars have initiated discussions and research, achieving common data governance lineage relationship construction methods can meet the needs of metadata identification, tracking, and traceability. However, the lack of a unified standard model for establishing lineage relationships, which involves complex processes and heterogeneous sources, and is incompatible with multiple heterogeneous systems, presents certain limitations in data governance.

[0039] Currently, traditional methods for constructing data lineage relationships use hard-coded annotations for operator annotations, increasing manual development costs. Traditional methods for processing data lineage relationships require manual intervention in the original lineage of SQL statements and employ graph databases, increasing system complexity. Traditional methods for querying data lineage relationships use tree-structured syntax trees to parse data lineage, making the data lineage construction process overly complex. Traditional methods for visually configuring graphs from multi-source heterogeneous data fail to decouple and orchestrate graph configuration, leading to a certain amount of code duplication. Therefore, a unified and standardized data lineage construction mechanism is needed to eliminate the complexity of data governance in multi-source heterogeneous systems.

[0040] In response to the shortcomings of traditional technologies, the purpose of this invention is to create a standard solution for constructing multi-source heterogeneous data lineage relationships, which is compatible with offline and real-time processing processes, multi-source heterogeneous data sources, DAG visual plug-in configuration and orchestration, low-code development, no need for human participation, non-perceptual lineage analysis, data lineage graph retrieval, lightweight system architecture, support for complex business data lineage construction processes, and effectively reduce project R&D costs and manual operation and maintenance costs.

[0041] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings.

[0042] See also Figure 1 In one embodiment, a method for constructing a kinship relationship of multi-source heterogeneous data is provided, comprising the following processing steps:

[0043] In S1, use the DAG drag-and-drop method to select input plug-ins, data processing plug-ins, and output plug-ins, and configure plug-in parameters. For input plug-ins, select the input data source plug-in, including the Kafka data source plug-in or the MySQL data source plug-in. For data processing plug-ins, select the data processing process plug-in, including the SQL statement plug-in, the data filtering plug-in, the rule calculation plug-in, the field conversion plug-in, or the date conversion plug-in. For output plug-ins, select the output target source plug-in, including the Hive data source plug-in, the DB2 data source plug-in, or the Phoenix data source plug-in.

[0044] It can be understood that for the multi-source heterogeneous data processing scenarios corresponding to various types of current industrial equipment, the data processing system running the method of the present application can carry out various data processing interactive operations through a WEB interface. Various heterogeneous data source plug-ins, such as but not limited to Kafka data source plug-ins, MySQL data source plug-ins, Hive data source plug-ins, DB2 data source plug-ins, and Phoenix data source plug-ins, can be pre-configured according to the actual application situation and execution environment of the system, or can be configured online as needed during the application process, as long as the existing data sources can be encapsulated in the form of plug-ins. Similarly, various data processing process plug-ins, such as but not limited to SQL statement plug-ins, data filtering plug-ins, rule calculation plug-ins, field conversion plug-ins, and date conversion plug-ins, can also be pre-configured according to the actual application situation and execution environment of the system, or can be configured online as needed during the application process, as long as the existing data processing process functions can be encapsulated in the form of plug-ins.

[0045] A Kafka data source uses Apache Kafka as its data source. Apache Kafka is a high-throughput, distributed message queue system widely used in scenarios such as real-time data stream processing, log collection, and event-driven architectures. Kafka's reliability, high throughput, and distributed processing make it a preferred data source for many real-time data processing systems and stream computing frameworks. By combining Kafka with other tools or frameworks, you can build powerful real-time data processing and distributed systems. A MySQL data source uses a MySQL database as its data source. MySQL is an open-source relational database management system widely used in various web applications and data-driven applications. It provides a stable, reliable, and scalable data storage solution that supports transaction processing, data indexing, replication, and backup. By combining MySQL with other applications or data processing tools, you can build a variety of application systems, including web applications, data warehouses, report generation, and data analysis. SQL (Structured Query Language) statements are a standardized language for managing and operating relational databases. It provides a set of commands and syntax rules for defining, operating, and querying databases.

[0046] A Hive data source uses Apache Hive as its data source. Apache Hive is a data warehouse infrastructure built on Hadoop, designed to process large-scale distributed datasets. Hive data sources are widely used in big data processing and data warehousing, and are particularly well-suited for batch processing and offline analysis tasks. By combining Hive with other tools and frameworks, such as Hadoop and Spark, powerful big data processing and analysis systems can be built. A DB2 data source uses an IBM DB2 database as its data source. IBM DB2 is a relational database management system (RDBMS) widely used in enterprise applications and data-driven solutions. DB2 offers high scalability and reliability, supporting features such as transaction processing, data indexing, replication, and backup. It also offers a variety of deployment options, including standalone deployment, distributed clusters, and cloud environments, to accommodate application scenarios of varying scale and requirements. By combining DB2 with other applications or data processing tools, various enterprise application systems can be built, including transaction processing systems, data warehousing, report generation, and data analysis. DB2 also has multiple versions and variants (such as DB2 for z / OS and DB2 for LUW). Specific functions and features may vary depending on the version and configuration, and can be selected based on actual application needs. Hudi (Hadoop Upserts Deletes and Incrementals) is an open source data management framework for building large-scale, high-performance data lakes. A Hudi data source refers to a data source that uses Hudi as its data source. Doris (formerly known as Apache Doris, now renamed Palo) is an open source distributed SQL data warehouse. A Doris data source refers to a data source that uses Doris as its data source.

[0047] Phoenix data source refers to a data source that uses Apache Phoenix as its data source. Apache Phoenix is ​​a relational database engine built on Apache HBase, designed to provide high-performance query and management capabilities for HBase data. With the Phoenix data source, users can use standard SQL statements to query and manipulate HBase data, performing complex queries, aggregations, and joins. Phoenix also provides transaction support, data versioning, and distributed query capabilities, making processing large datasets more convenient and efficient. DAG (Directed Acyclic Graph) dragging is an interactive technique used to manipulate directed acyclic graphs (DAGs) in a graphical user interface (GUI) or similar visual environment. A DAG is a graph structure composed of nodes (or vertices) and edges, where nodes represent operations or data, and edges represent dependencies or flow relationships between nodes. Drag-and-drop refers to the act of dragging nodes from one location to another using a mouse or similar interactive method. In DAG drag-and-drop, users can create, delete, or modify connections between nodes by dragging them within the web interface. By dragging, users can reposition nodes, thereby changing the structure of the graph.

[0048] Specifically, a plugin pre-configuration method has been designed. For example, using a DAG-based drag-and-drop method, you can select an input plugin and configure the input source; select a data processing plugin and configure processing parameters; and select an output plugin and configure the target source. First, select the input plugin and configure its parameters. For example, you can use the DAG-based drag-and-drop method to select the MySQL data source plugin as the input plugin for the multi-source heterogeneous data lineage relationship building process. Then, configure the following parameters: pull the data source Da, pull the input data table Ta, and select certain field sets Ca.

[0049] Next, select the data processing plug-in and configure its parameters. For example, you can use the DAG drag-and-drop method to select the SQL statement plug-in as the data processing plug-in (during the data processing process, you can similarly select other data processing plug-ins depending on the processing process) to input the SQL statements required for the data processing process. Next, select the output plug-in and configure its parameters. For example, you can use the DAG drag-and-drop method to select the Hive data source plug-in as the output plug-in (during the data output process, you can similarly select other output target source plug-ins depending on the output target) and configure parameters for the plug-in, such as pulling the data source Db and pulling the data table Tb.

[0050] S2, dynamically arrange the plug-in relationship of the input plug-in, data processing plug-in and output plug-in according to the defined data plug-in relationship dynamic arrangement model.

[0051] It can be understood that a plug-in relationship arrangement method is designed, such as using a DAG view connection method to dynamically arrange plug-in relationships and generate the corresponding DAG directed acyclic graph. Specifically, Figure 2 and Figure 3 As shown, this application designs three plug-in relationships L, which are dynamically arranged using DAG connection method, for example: input plug-in I→data processing plug-in C, data processing plug-in C→output plug-in O, input plug-in I→output plug-in O.

[0052] The defined data plug-in relationship dynamic orchestration model can be as follows:

[0053]

[0054] Among them, Lineage() represents the lineage, and m1, m2 and m3 are the number of plug-ins that are plug-ins I, C and O respectively.

[0055] S3 saves the plug-in attribute information obtained after the dynamic arrangement of plug-in relationships to the Task configuration; the plug-in attribute information includes input plug-ins, data processing plug-ins, output plug-ins, plug-in relationships, plug-in data sources, plug-in data tables, and plug-in field sets.

[0056] It can be understood that after completing the above steps, the plug-in properties are entered into the Task configuration. Specifically, from the DAG directed acyclic graph, three categories are parsed: input plug-in I, processing plug-in C, and output plug-in O. The plug-in property information such as input plug-in I, processing plug-in C, output plug-in O, plug-in relationship L, as well as the selected data source, data table, and field set are saved to the Task configuration to record the task configuration of these plug-ins and ensure the normal operation of the plug-ins in the data processing task. Among them, the plug-in property conversion task entity model is defined, which can be expressed as follows:

[0057]

[0058] Among them, m1, m2 and m3 are the number of plug-ins of plug-in I, C and O respectively, m4 is the number of L plug-in relationships, and the arrows indicate the direction of task conversion.

[0059] S4, use the data lineage entity combination sample model to parse the data lineage relationship entity according to the Task configuration, bind the data lineage relationship entity to the entity definition identifier and save it to the entity element table; the data lineage relationship entity includes the data source entity, data table entity, field entity and data processing entity, and the entity definition identifier includes the database identifier, data set identifier, field identifier and processing process identifier.

[0060] It can be understood that after completing the task configuration, the input entity, data processing entity, and output entity are parsed from the Task configuration, and the attributes of these data lineage entity are recorded in the entity element table. Specifically, data lineage entity definition identifiers can be created in advance: data source types such as MySQL and Hive are parsed from the Task configuration, and the corresponding database identifiers mysql_db and hive_db, data set identifiers mysql_table and hive_table, field identifiers mysql_column and hive_column, and processing process identifiers mysql_process and hive_process are defined, and these data lineage entity definition identifiers are saved in the entity definition table.

[0061] In this embodiment, plug-ins and orchestration relationships are freely selected according to business needs to form a sample model of multiple data lineage entity combinations. The model can be expressed as follows:

[0062]

[0063] Where n1, n2, and n3 are the number of plug-ins I, C, and O respectively, m1 is the number of input sources Da, h1 is the number of input tables Ta, m2 is the number of SQL statements, m3 is the number of target sources Db, and h3 is the number of target tables Tb.

[0064] Combine the Task configuration with the data lineage entity combination sample model to parse out the data source entities Da and Db, data table entities Ta and Tb, field entity Ca, and data processing entity SQL. Bind these data lineage relationship entities to the above-mentioned entity definition identifiers, and save these entity attributes to the entity element table.

[0065] S5, using the data lineage entity relationship model to parse the entity relationship of the data lineage relationship entity according to the Task configuration, and save the entity relationship to the entity relationship table; the entity lineage relationship includes the database table entity relationship, the input table and data processing entity relationship, the data processing and output table relationship, and the input table and output table entity relationship.

[0066] As you can understand, after completing the task configuration, the entity relationships between the three types of entities (input entities, data processing entities, and output entities) are parsed from the task configuration and recorded in the entity relationship table. In general, there are three types of data lineage relationships: table lineage, field lineage, and task lineage.

[0067] Specifically, this application designs four types of entity relationships, parsing the database table entity relationship, the input table and data processing entity relationship, the data processing and output table entity relationship, and the input table and output table entity relationship from the task configuration, and saving these entity relationships in the entity relationship table. Among them, the defined data lineage entity relationship model can be expressed as follows:

[0068]

[0069] Among them, m1 is the number of entities in the input table Ta, m2 is the number of entities in the data processing C, and m3 is the number of entities in the output table Tb.

[0070] S6: Bind the data lineage relationship entity and the entity lineage relationship to the Task task identifier and save them in the task association table.

[0071] It can be understood that after completing the above-mentioned processing steps, the above-mentioned parsed entity elements and entity relationships are bound to the corresponding Task tasks, and the user information corresponding to the task is bound at the same time, that is, the entity elements, entity relationships, etc. are bound to the Task task identifier, and the user information is bound at the same time, and these binding information are saved in the task association table to eliminate the problem of metadata traceability difficulties caused by frequent changes in data structure.

[0072] S7, using the BFS-based data lineage relationship transformation model, determines the retrieval center and uses the data lineage relationship traversal algorithm to find all data lineage relationship entities and entity lineage relationships from the entity relationship table.

[0073] It is understood that this embodiment also designs a data lineage relationship conversion model and a data lineage relationship traversal algorithm based on BFS to achieve automated and efficient querying of data lineage relationships. BFS (Breadth-First Search) is a traversal algorithm for graphs and trees. BFS starts from the root node of the graph or tree and traverses the nodes layer by layer, first visiting all neighbor nodes of the current node, then visiting the neighbor nodes of the neighbor nodes in turn, and so on, until all nodes are traversed. The data lineage relationship conversion model based on BFS can be defined as follows:

[0074]

[0075] Where m1 is the total number of entity relationships, m2 is the number of entity relationships after BFS traversal, E is the entity element, L is the entity relationship, and the arrow indicates the BFS transformation direction. Four data lineage relationship traversal algorithms can be designed, such as the direct data lineage relationship traversal algorithm, the pre-data lineage relationship traversal algorithm, the post-data lineage relationship traversal algorithm, and the pre- and post-data lineage relationship traversal algorithm. A traversal algorithm can be selected based on task requirements, using a database, dataset, or task identifier as the search center and using the BFS-based data lineage relationship transformation model to find all required entity elements and their lineage relationships.

[0076] The above-mentioned multi-source heterogeneous data lineage relationship construction method designs a unified data lineage relationship construction standard based on DAG plug-in orchestration. First, the DAG drag-and-drop method is used to select the data processing plug-in and configure the plug-in parameters. Then, the plug-in relationship is dynamically orchestrated according to the defined data plug-in relationship dynamic orchestration model. The obtained plug-in attribute information is saved to the Task task configuration. Then, the data lineage entity combination sample model is used to perform data lineage relationship entity analysis. The data lineage entity relationship model is used to perform entity relationship analysis of data lineage relationship entities, and then the data lineage is associated with the task and bound. Finally, the BFS-based data lineage relationship conversion model is used to find all data lineage relationship entities and entity lineage relationships from the entity relationship table through the data lineage relationship traversal algorithm to generate a data lineage map.

[0077] Compared with traditional technologies, the above technical solutions of this application can simultaneously support offline and real-time data processing, integrate multiple heterogeneous data sources, reduce user operation difficulty through visual interface configuration, use plug-in configuration to achieve low-code development, dynamically arrange plug-in dependencies, design blood relationship storage structure, design non-perceptual blood relationship parsing algorithm, and design binary tree breadth-first relationship traversal method BFS to achieve the generation of full life cycle data blood relationship map, reduce external tool dependence to reduce system complexity. Ultimately, the integration of data blood relationship construction in multiple links solves the problem of numerous processing processes and messy relationships, solves the problem of frequent changes in data structure and difficult metadata traceability, simplifies the system architecture to reduce system complexity, reduces user labor costs, and fundamentally solves the problem of high complexity in building multi-source heterogeneous blood relationship of data.

[0078] In one embodiment, the above-mentioned method for establishing bloodline relationships of multi-source heterogeneous data may further include the following pre-steps:

[0079] Create a data lineage relationship structure storage table for a multi-source heterogeneous data processing system; the data lineage relationship structure storage table includes a data source table, a data set table, an entity definition table, an entity element table, an entity relationship table, and a task association table;

[0080] Create multiple heterogeneous data source plug-ins and data processing plug-ins; heterogeneous data source plug-ins include Kafka data source plug-in, MySQL data source plug-in, Hive data source plug-in, DB2 data source plug-in, and Phoenix data source plug-in;

[0081] Configure the data source connection information of the heterogeneous data source plug-in through the WEB interface and save it to the data source table;

[0082] Pull the table metadata of the data source type of the heterogeneous data source plug-in through the web interface and save it to the dataset table; the table metadata includes the table name, remarks, field name, field type, field length, precision, and primary key;

[0083] Parse the data source type of the heterogeneous data source plug-in from the Task configuration and create an entity definition identifier.

[0084] It can be understood that the data lineage structure storage tables required for system operation can be pre-created and can include two metadata tables, such as the data source table and the dataset table, and four lineage relationship tables, such as the entity definition table, the entity element table, the entity relationship table, and the task association table. Each heterogeneous data source is then created as a corresponding heterogeneous data source plug-in, such as the Kafka data source plug-in, the MySQL data source plug-in, the Hive data source plug-in, the DB2 data source plug-in, and the Phoenix data source plug-in, as well as data processing plug-ins, such as the SQL statement plug-in, the data filtering plug-in, the rule calculation plug-in, the field conversion plug-in, and the date conversion plug-in.

[0085] Then, the connection information of data sources such as MySQL and Hive is configured through the system's WEB interface and saved in the data source table for backup; then, the table metadata of types such as MySQL and Hive, such as table name, remarks, field name, field type, length, precision and primary key, are pulled through the WEB interface, and these table metadata are saved in the dataset table for backup, thereby facilitating support for low-code development, simplifying the process of building data lineage relationships, supporting multiple heterogeneous data sources, eliminating the technical barriers to lineage relationships between different types of data, supporting offline and real-time data processing processes, and solving the limitations of traditional methods in multiple computing engines.

[0086] In one embodiment, Figure 4 As shown, the above-mentioned method for constructing bloodline relationships of multi-source heterogeneous data may further include the following steps:

[0087] S8, generating a data lineage graph based on all the found data lineage relationship entities and entity lineage relationships;

[0088] S9, output the data lineage map to the front end for WEB visualization display.

[0089] It can be understood that after finding all the required data lineage entities and entity lineage relationships, the corresponding data lineage map can be directly generated and can be output to the front-end device through the provided interface, such as but not limited to the API interface, so as to realize the WEB visualization of the data lineage map through the front-end device. The data lineage map can be displayed in the form of but not limited to Echarts (an open source chart library based on JavaScript for visualizing data on web pages) charts, thereby achieving further data lineage map visualization retrieval and display effects.

[0090] In one embodiment, the data lineage relationship traversal algorithm is a data lineage direct relationship traversal algorithm. Regarding the process of determining the search center and searching all data lineage relationship entities and entity lineage relationships from the entity relationship table using the data lineage relationship traversal algorithm in step S7, the process may specifically include the following processing steps:

[0091] Taking the data lineage relationship entity as the retrieval center, all the direct preceding and following node data lineage relationship entities and entity lineage relationships of the retrieval center are found from the entity relationship table.

[0092] Specifically, according to the data lineage relationship conversion model of BFS, in the entity relationship stored in the entity relationship table, a certain entity can be used as the central node (i.e., the retrieval center) to search for all direct predecessor and successor node entities (parent and child nodes) and their lineage relationships, thereby obtaining all required data lineage relationship entities and entity lineage relationships.

[0093] In one embodiment, the data lineage relationship traversal algorithm is a data lineage predecessor relationship traversal algorithm. Regarding the process of determining the search center and searching all data lineage relationship entities and entity lineage relationships from the entity relationship table using the data lineage relationship traversal algorithm in step S7, the process may specifically include the following processing steps:

[0094] Taking the data lineage relationship entity as the retrieval center, find the data lineage relationship entity and entity lineage relationship of the direct predecessor node of the retrieval center from the entity relationship table;

[0095] The binary tree breadth-first traversal algorithm is used to search for the predecessor node layer by layer from top to bottom and from left to right, with the blood relationship entity of the direct predecessor node data as the center, until there is no predecessor node recursively terminated;

[0096] Save all data lineage entities and entity lineage relationships on the search link.

[0097] Specifically, according to the data lineage relationship conversion model of BFS, in the entity relationship stored in the entity relationship table, a certain entity can be used as the central node to find the direct predecessor node entity and its lineage relationship; and the binary tree breadth-first traversal algorithm BFS is used to take these predecessor nodes as the central nodes, and traverse them layer by layer from top to bottom and from left to right until there are no predecessor nodes and recursively terminate, and save all entity elements and their lineage relationships on this link, so as to obtain all the required data lineage relationship entities and entity lineage relationships.

[0098] In one embodiment, the data lineage relationship traversal algorithm is a data lineage postposition relationship traversal algorithm. Regarding the process of determining the search center and searching all data lineage relationship entities and entity lineage relationships from the entity relationship table using the data lineage relationship traversal algorithm in step S7, the process may specifically include the following processing steps:

[0099] Taking the data lineage relationship entity as the retrieval center, find the data lineage relationship entity and entity lineage relationship directly following the retrieval center from the entity relationship table;

[0100] The binary tree breadth-first traversal algorithm is used to search for the subsequent nodes from top to bottom and from left to right, with the blood relationship entity of the direct subsequent node data as the center, until there is no subsequent node recursively terminated;

[0101] Save all data lineage entities and entity lineage relationships on the search link.

[0102] Specifically, according to the data lineage relationship conversion model of BFS, in the entity relationship stored in the entity relationship table, a certain entity can be the central node. Similarly, through the above central node + post-relationship, the binary tree breadth-first traversal algorithm BFS is used to obtain all entity elements and lineage relationships, and all required data lineage relationship entities and entity lineage relationships can also be obtained.

[0103] In one embodiment, the data lineage relationship traversal algorithm is a data lineage forward and backward relationship traversal algorithm. Regarding the process of determining the search center and searching all data lineage relationship entities and entity lineage relationships from the entity relationship table using the data lineage relationship traversal algorithm in step S7, the process may specifically include the following processing steps:

[0104] Taking the data lineage relationship entity as the retrieval center, find the data lineage relationship entities and entity lineage relationships of the direct preceding and succeeding nodes of the retrieval center from the entity relationship table;

[0105] The binary tree breadth-first traversal algorithm is used to search for the predecessor and successor nodes from top to bottom and from left to right, with the data blood relationship entity of the immediate predecessor and successor nodes as the center, until there are no predecessor and successor nodes left.

[0106] Save all data lineage entities and entity lineage relationships on the search link.

[0107] Specifically, according to the data lineage relationship conversion model of BFS, in the entity relationship stored in the entity relationship table, a certain entity can be the central node. Similarly, through the above central node + previous and next relationships, the binary tree breadth-first traversal algorithm BFS is used to obtain all entity elements and lineage relationships, and all required data lineage relationship entities and entity lineage relationships can also be obtained. The processing flow of the binary tree breadth-first traversal can be as follows Figure 5 shown.

[0108] It should be understood that although the above process Figure 1 、 Figure 4 and Figure 5 The steps in the flowchart are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1 、 Figure 4 and Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0109] In one embodiment, Figure 6As shown, a multi-source heterogeneous data lineage relationship construction system 100 is provided, including a plug-in selection module 01, a relationship orchestration module 02, a task configuration module 03, an entity resolution module 04, a relationship resolution module 05, a lineage binding module 06 and a lineage search module 07. Among them, the plug-in selection module 01 is used to select input plug-ins, data processing plug-ins and output plug-ins by using the DAG drag-and-drop method, and configure plug-in parameters; the input data source plug-in selected by the input plug-in includes a Kafka data source plug-in or a MySQL data source plug-in, the data processing process plug-in selected by the data processing plug-in includes an SQL statement plug-in, a data filtering plug-in, a rule calculation plug-in, a field conversion plug-in or a date conversion plug-in, and the output target source plug-in selected by the output plug-in includes a Hive data source plug-in, a DB2 data source plug-in or a Phoenix data source plug-in. The relationship orchestration module 02 is used to dynamically orchestrate the plug-in relationship of the input plug-in, the data processing plug-in and the output plug-in according to the defined data plug-in relationship dynamic orchestration model.

[0110] The task configuration module 03 is used to save the plug-in attribute information obtained after the dynamic arrangement of the plug-in relationship to the Task task configuration; the plug-in attribute information includes the input plug-in, the data processing plug-in, the output plug-in, the plug-in relationship, the plug-in data table and the plug-in field set. The entity resolution module 04 is used to use the data lineage entity combination sample model to resolve the data lineage relationship entity according to the Task task configuration, bind the data lineage relationship entity to the entity definition identifier and save it to the entity element table; the data lineage relationship entity includes the data source entity, the data table entity, the field entity and the data processing entity, and the entity definition identifier includes the database identifier, the data set identifier, the field identifier and the processing process identifier. The relationship resolution module 05 is used to use the data lineage entity relationship model to resolve the entity relationship of the data lineage relationship entity according to the Task task configuration, and save the entity relationship to the entity relationship table; the entity lineage relationship includes the database table entity relationship, the input table and data processing entity relationship, the data processing and output table relationship, and the input table and output table entity relationship. The lineage binding module 06 is used to bind the data lineage relationship entity and the entity lineage relationship to the Task task identifier and save them to the task association table. The blood relationship search module 07 is used to use the BFS-based data blood relationship conversion model to determine the retrieval center and find all data blood relationship entities and entity blood relationships from the entity relationship table through the data blood relationship traversal algorithm.

[0111] The above-mentioned multi-source heterogeneous data lineage relationship construction system designs a unified data lineage relationship construction standard based on DAG plug-in orchestration. It first uses the DAG drag-and-drop method to select data processing plug-ins and configure plug-in parameters, and then dynamically orchestrates plug-in relationships according to the defined data plug-in relationship dynamic orchestration model. The obtained plug-in attribute information is saved to the Task task configuration, and then the data lineage entity combination sample model is used to perform data lineage relationship entity analysis, and the data lineage entity relationship model is used to perform entity relationship analysis of data lineage relationship entities, and then the data lineage is associated with the task and bound. Finally, the BFS-based data lineage relationship conversion model is used to find all data lineage relationship entities and entity lineage relationships from the entity relationship table through the data lineage relationship traversal algorithm to generate a data lineage map.

[0112] Compared with traditional technologies, the above technical solutions of this application can simultaneously support offline and real-time data processing, integrate multiple heterogeneous data sources, reduce user operation difficulty through visual interface configuration, use plug-in configuration to achieve low-code development, dynamically arrange plug-in dependencies, design blood relationship storage structure, design non-perceptual blood relationship parsing algorithm, and design binary tree breadth-first relationship traversal method BFS to achieve the generation of full life cycle data blood relationship map, reduce external tool dependence to reduce system complexity. Ultimately, the integration of data blood relationship construction in multiple links solves the problem of numerous processing processes and messy relationships, solves the problem of frequent changes in data structure and difficult metadata traceability, simplifies the system architecture to reduce system complexity, reduces user labor costs, and fundamentally solves the problem of high complexity in building multi-source heterogeneous blood relationship of data.

[0113] In one embodiment, the modules of the multi-source heterogeneous data lineage relationship construction system 100 can also be used to implement the functions of the steps of the corresponding subsidiary embodiments of the multi-source heterogeneous data lineage relationship construction method.

[0114] Regarding the specific limitations of the multi-source heterogeneous data lineage relationship construction system 100, please refer to the corresponding limitations of the multi-source heterogeneous data lineage relationship construction method above, which will not be repeated here. The various modules in the above-mentioned multi-source heterogeneous data lineage relationship construction system 100 can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of a device with data processing functions in the form of hardware, or can be stored in the memory of the aforementioned device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. The aforementioned device can be, but is not limited to, various types of data calculation and processing equipment already available in the art.

[0115] In one embodiment, a computer device is also provided, including a memory and a processor, the memory stores a computer program, and the processor implements the following processing steps when executing the computer program: using the DAG drag-and-drop method to select input plug-ins, data processing plug-ins and output plug-ins, and configure plug-in parameters; the input data source plug-in selected by the input plug-in includes a Kafka data source plug-in or a MySQL data source plug-in, the data processing process plug-in selected by the data processing plug-in includes an SQL statement plug-in, a data filtering plug-in, a rule calculation plug-in, a field conversion plug-in or a date conversion plug-in, and the output target source plug-in selected by the output plug-in includes a Hive data source plug-in, a DB2 data source plug-in or a Phoenix data source plug-in; the input plug-in, the data processing plug-in and the output plug-in are dynamically orchestrated according to the defined data plug-in relationship dynamic orchestration model; the plug-in attribute information obtained after the dynamic orchestration of the plug-in relationship is saved to the Task task configuration; the plug-in attribute information includes the input plug-in, the data processing plug-in, the output plug-in, the plug-in relationship, the plug-in data source, the plug-in , a collection of fields of data tables and plug-ins; using the data lineage entity combination sample model to parse out data lineage relationship entities according to the Task configuration, binding the data lineage relationship entities to the entity definition identifier and saving them to the entity element table; data lineage relationship entities include data source entities, data table entities, field entities and data processing entities, and the entity definition identifiers include database identifiers, data set identifiers, field identifiers and processing process identifiers; using the data lineage entity relationship model to parse out the entity relationships of data lineage relationship entities according to the Task configuration, saving the entity relationships to the entity relationship table; entity lineage relationships include database table entity relationships, input table and data processing entity relationships, data processing and output table relationships, and input table and output table entity relationships; binding data lineage relationship entities and entity lineage relationships to the Task identifier and saving them to the task association table; using the data lineage relationship transformation model based on BFS, determine the retrieval center and find all data lineage relationship entities and entity lineage relationships from the entity relationship table through the data lineage relationship traversal algorithm.

[0116] It can be understood that in addition to the memory and processor mentioned above, the above-mentioned computer device also includes other software and hardware components not listed in this specification. The specific components can be determined according to the model of the specific computer device in different application scenarios. This specification will not list them one by one in detail.

[0117] In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added in each embodiment of the above-mentioned multi-source heterogeneous data bloodline relationship construction method.

[0118] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the following processing steps are implemented: using a DAG drag-and-drop method to select input plug-ins, data processing plug-ins and output plug-ins, and configure plug-in parameters; the input data source plug-in selected by the input plug-in includes a Kafka data source plug-in or a MySQL data source plug-in, the data processing process plug-in selected by the data processing plug-in includes an SQL statement plug-in, a data filtering plug-in, a rule calculation plug-in, a field conversion plug-in or a date conversion plug-in, and the output target source plug-in selected by the output plug-in includes a Hive data source plug-in, a DB2 data source plug-in or a Phoenix data source plug-in; the input plug-in, the data processing plug-in and the output plug-in are dynamically orchestrated according to the defined data plug-in relationship dynamic orchestration model; the plug-in attribute information obtained after the dynamic orchestration of the plug-in relationship is saved to the Task task configuration; the plug-in attribute information includes the input plug-in, the data processing plug-in, the output plug-in, the plug-in relationship, the plug-in data source, the plug-in data The field collection of tables and plug-ins; using the data lineage entity combination sample model to parse the data lineage relationship entities according to the Task configuration, bind the data lineage relationship entities to the entity definition identifier and save them to the entity element table; the data lineage relationship entities include data source entities, data table entities, field entities and data processing entities, and the entity definition identifiers include database identifiers, data set identifiers, field identifiers and processing process identifiers; using the data lineage entity relationship model to parse the entity relationships of the data lineage relationship entities according to the Task configuration, and save the entity relationships to the entity relationship table; the entity lineage relationship includes database table entity relationships, input table and data processing entity relationships, data processing and output table relationships, and input table and output table entity relationships; binding the data lineage relationship entities and entity lineage relationships to the Task identifier and saving them to the task association table; using the data lineage relationship conversion model based on BFS, determine the retrieval center and find all data lineage relationship entities and entity lineage relationships from the entity relationship table through the data lineage relationship traversal algorithm.

[0119] In one embodiment, when the computer program is executed by a processor, it can also implement the steps or sub-steps added to the various embodiments of the above-mentioned multi-source heterogeneous data bloodline relationship construction method.

[0120] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, referred to as RDRAM) and interface dynamic random access memory (DRDRAM).

[0121] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for constructing kinship relationships of multi-source heterogeneous data, characterized in that: Including steps: Use the DAG drag-and-drop method to select input plug-ins, data processing plug-ins, and output plug-ins, and configure plug-in parameters; the input data source plug-in selected by the input plug-in includes a Kafka data source plug-in or a MySQL data source plug-in; the data processing process plug-in selected by the data processing plug-in includes an SQL statement plug-in, a data filtering plug-in, a rule calculation plug-in, a field conversion plug-in, or a date conversion plug-in; the output target source plug-in selected by the output plug-in includes a Hive data source plug-in, a DB2 data source plug-in, or a Phoenix data source plug-in; Performing plug-in relationship dynamic arrangement on the input plug-in, the data processing plug-in, and the output plug-in according to a defined data plug-in relationship dynamic arrangement model; The plug-in attribute information obtained after the dynamic arrangement of the plug-in relationship is saved to the Task configuration; the plug-in attribute information includes the input plug-in, the data processing plug-in, the output plug-in, the plug-in relationship, the plug-in data source, the plug-in data table and the plug-in field set; Utilize the data lineage entity combination sample model to parse the data lineage relationship entity according to the Task configuration, bind the data lineage relationship entity to the entity definition identifier and save it to the entity element table; the data lineage relationship entity includes the data source entity, the data table entity, the field entity and the data processing entity, and the entity definition identifier includes the database identifier, the data set identifier, the field identifier and the processing process identifier; Utilize the data entity lineage relationship model to parse the entity relationship of the data lineage relationship entity according to the Task configuration, and save the entity relationship into the entity relationship table; The entity blood relationship includes database table entity relationship, input table and data processing entity relationship, data processing and output table relationship, input table and output table entity relationship; Bind the data lineage entity and the entity lineage relationship to the Task task identifier and save them in the task association table; The data lineage relationship conversion model based on BFS is used to determine the retrieval center and all data lineage relationship entities and entity lineage relationships are found from the entity relationship table through the data lineage relationship traversal algorithm.

2. The method for constructing kinship relationships of multi-source heterogeneous data according to claim 1, characterized in that: Also includes the steps: Generate a data lineage graph based on all the data lineage relationship entities and the entity lineage relationships found; The data lineage map is output to the front end for WEB visualization display.

3. The method for constructing kinship relationships of multi-source heterogeneous data according to claim 1 or 2, characterized in that: The data lineage relationship traversal algorithm is a data lineage direct relationship traversal algorithm; The process of determining the retrieval center and searching all data lineage relationship entities and entity lineage relationships from the entity relationship table using a data lineage relationship traversal algorithm includes: Taking the data lineage relationship entity as the retrieval center, all the direct preceding and following node data lineage relationship entities and entity lineage relationships of the retrieval center are found from the entity relationship table.

4. The method for constructing kinship relationships of multi-source heterogeneous data according to claim 1 or 2, characterized in that: The data lineage relationship traversal algorithm is a data lineage predecessor relationship traversal algorithm; The process of determining the retrieval center and searching all data lineage relationship entities and entity lineage relationships from the entity relationship table using a data lineage relationship traversal algorithm includes: Taking the data lineage relationship entity as the retrieval center, searching the data lineage relationship entity and entity lineage relationship of the direct predecessor node of the retrieval center from the entity relationship table; A binary tree breadth-first traversal algorithm is used to search for the predecessor node layer by layer from top to bottom and from left to right, with the blood relationship entity of the direct predecessor node data as the center, until there is no predecessor node and the recursion is terminated; Save all data lineage entities and entity lineage relationships on the search link.

5. The method for constructing kinship relationships of multi-source heterogeneous data according to claim 1 or 2, characterized in that: The data lineage relationship traversal algorithm is a data lineage postposition relationship traversal algorithm; The process of determining the retrieval center and searching all data lineage relationship entities and entity lineage relationships from the entity relationship table using a data lineage relationship traversal algorithm includes: Taking the data lineage relationship entity as the retrieval center, searching the data lineage relationship entities and entity lineage relationships that are directly subsequent nodes of the retrieval center from the entity relationship table; A binary tree breadth-first traversal algorithm is used to search for subsequent nodes from top to bottom and from left to right, with the direct subsequent node data blood relationship entity as the center, until there is no subsequent node and the recursion is terminated; Save all data lineage entities and entity lineage relationships on the search link.

6. The method for constructing kinship relationships of multi-source heterogeneous data according to claim 1 or 2, characterized in that: The data lineage relationship traversal algorithm is a data lineage front-back relationship traversal algorithm; The process of determining the retrieval center and searching all data lineage relationship entities and entity lineage relationships from the entity relationship table using a data lineage relationship traversal algorithm includes: Taking the data lineage relationship entity as the retrieval center, searching the data lineage relationship entities and entity lineage relationships of the direct preceding and succeeding nodes of the retrieval center from the entity relationship table; A binary tree breadth-first traversal algorithm is used to search for the preceding node and the succeeding node data blood relationship entity from top to bottom and from left to right, until there are no preceding nodes and succeeding nodes left; Save all data lineage entities and entity lineage relationships on the search link.

7. The method for constructing kinship relationships of multi-source heterogeneous data according to claim 1, characterized in that: Also includes the steps: Creating a data lineage relationship structure storage table for a multi-source heterogeneous data processing system; the data lineage relationship structure storage table includes a data source table, a data set table, an entity definition table, an entity element table, an entity relationship table, and a task association table; Create multiple heterogeneous data source plug-ins and data processing process plug-ins; the heterogeneous data source plug-ins include Kafka data source plug-in, MySQL data source plug-in, Hive data source plug-in, DB2 data source plug-in and Phoenix data source plug-in; Configure the data source connection information of the heterogeneous data source plug-in through the WEB interface and save it in the data source table; Pull the table metadata of the data source type of the heterogeneous data source plug-in through the WEB interface and save it to the data set table; the table metadata includes table name, remarks, field name, field type, field length, precision and primary key; The data source type of the heterogeneous data source plug-in is parsed from the Task configuration and the entity definition identifier is created.

8. A multi-source heterogeneous data lineage relationship construction system, characterized by: include: The plug-in selection module is used to select input plug-ins, data processing plug-ins, and output plug-ins by dragging and dropping DAG, and configure plug-in parameters; The input data source plug-in selected by the input plug-in includes a Kafka data source plug-in or a MySQL data source plug-in; the data processing process plug-in selected by the data processing plug-in includes an SQL statement plug-in, a data filtering plug-in, a rule calculation plug-in, a field conversion plug-in or a date conversion plug-in; the output target source plug-in selected by the output plug-in includes a Hive data source plug-in, a DB2 data source plug-in or a Phoenix data source plug-in; a relationship arrangement module, configured to dynamically arrange the relationship between the input plug-in, the data processing plug-in, and the output plug-in according to a defined data plug-in relationship dynamic arrangement model; The task configuration module is used to save the plug-in attribute information obtained after the dynamic arrangement of the plug-in relationship to the Task task configuration; the plug-in attribute information includes the input plug-in, the data processing plug-in, the output plug-in, the plug-in relationship, the plug-in data source, the plug-in data table and the plug-in field set; An entity parsing module is used to parse data lineage relationship entities according to the Task configuration using the data lineage entity combination sample model, bind the data lineage relationship entities to entity definition identifiers and save them to the entity element table; the data lineage relationship entities include data source entities, data table entities, field entities and data processing entities, and the entity definition identifiers include database identifiers, data set identifiers, field identifiers and processing process identifiers; A relationship parsing module, configured to parse the entity relationships of the data lineage relationship entities according to the Task configuration using the data entity lineage relationship model, and save the entity relationships into an entity relationship table; The entity blood relationship includes database table entity relationship, input table and data processing entity relationship, data processing and output table relationship, input table and output table entity relationship; A blood relationship binding module is used to bind the data blood relationship entity and the entity blood relationship to a Task task identifier and save it in a task association table; The blood relationship search module is used to use the BFS-based data blood relationship conversion model to determine the retrieval center and find all data blood relationship entities and entity blood relationships from the entity relationship table through the data blood relationship traversal algorithm.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for constructing multi-source heterogeneous data bloodline relationships as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing multi-source heterogeneous data bloodline relationships according to any one of claims 1 to 7 are implemented.

Images