Data blood relationship full-link monitoring method and system, terminal and storage medium
By analyzing the information of the data source database and the target database, building a graph database relationship, and using the path search algorithm to analyze and visualize data blood relationships, solving the problems of high cost of data blood relationship construction and unclear relationships in the existing technology, achieving high efficiency of data relationships Monitor and visualize.
Patent Information
- Application Number
- CN202410953363.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-12
AI Technical Summary
The existing data blood relationship construction methods are costly and the data relationship is not concise enough, making it difficult to effectively monitor and visualize the complex relationship between data.
Through data extraction scripts, the source library and target library information is parsed, and the relationship between data sources, data tables and fields in the graph database is constructed. The path search algorithm is used to perform data blood relationship analysis, and the results are input into the visualization engine for display.
It realizes the use of graph database model to store data, simplifies the representation and query of complex relationships, tracks data transmission and processing processes, has reliable design principles and has wide application prospects.
Smart Images

Figure CN118939839A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data analysis, and specifically relates to a data lineage full-link monitoring method, system, terminal and storage medium. Background Art
[0002] Data lineage, also known as data pedigree, data origin, and data genealogy, refers to the natural relationship between data during its entire life cycle, from generation, processing, processing, fusion, circulation to final extinction. It records the link relationship generated by data, which is similar to human blood relationship, so it is called data lineage relationship.
[0003] Data lineage refers to the formation of various relationships between data throughout the entire life cycle of data, which runs through the entire data link and visualizes the flow of data. Whether it is structured data or unstructured data, there must be data lineage. Data lineage analysis sorts out the relationships between systems, tables, views, stored procedures, ETL, program codes, fields, etc., and uses graph databases for visualization. Simply put, it is to visualize how the data came from, what processes, stages and calculation logic it has gone through.
[0004] The existing data lineage construction uses the OpenLineage database, which is costly and the data relationship is not simple enough. Summary of the invention
[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a data lineage full-link monitoring method, system, terminal and storage medium to solve the above-mentioned technical problems.
[0006] In a first aspect, the present invention provides a data lineage full-link monitoring method, comprising: Analyze the source database information, target database information, field mapping information and dependencies between data tables through data extraction scripts; Based on the source database information, target database information, field mapping information and the dependencies between data tables, the relationships and properties between data sources, data tables and fields are constructed in the graph database, and a directed relationship is created with the source table as the starting point and the target table as the end point. Check the entities in the graph database and register data services for the graph database; Use the path search algorithm to perform data lineage analysis based on the graph database to obtain the lineage relationship between data elements; The blood relationship between the data elements is input into the visualization engine to visualize the blood relationship between the data elements.
[0007] In an optional implementation, the data extraction script is used to parse the source database information, target database information, field mapping information and dependencies between data tables, including: The ETL script is used to parse the source library information, target library information, field mapping information and dependencies between data tables in the data model; the source library information includes the database type, IP address, port, instance name, table name, and account number of the source library; the target library information includes the database type, IP address, port, instance name, table name, and account number of the target library.
[0008] In an optional implementation, based on the source database information, target database information, field mapping information and dependencies between data tables, the relationships and properties between data sources, data tables and fields are constructed in the graph database, and a directed relationship is created with the source table as the starting point and the target table as the end point, including: Build data models in graph databases based on pre-set data model entities, relationships, and attributes; The data source database information, target database information, field mapping information and dependencies between data tables are filled into the data model.
[0009] In an optional implementation, checking entities in a graph database and registering a data service for the graph database includes: Traverse the entities in the data model, update the attributes of the entity if it exists, and create the entity if it does not exist; Use the GraphQL Schema definition language to define GraphQL models; Put the GraphQL model into the GraphQL schema of a graph database, the graph database is Neo4j; Enable the GraphQL API of the graph database; Use a GraphQL client or tool to query the GraphQL API, store the GraphQL API information, and complete the data service registration.
[0010] In a second aspect, the present invention provides a data lineage full-link monitoring system, comprising: The data extraction module is used to parse the source database information, target database information, field mapping information and the dependencies between data tables through the data extraction script; The graph construction module is used to construct the relationship and attributes between data sources, data tables, and fields in the graph database based on the source database information, target database information, field mapping information, and dependencies between data tables, and to create a directed relationship with the source table as the starting point and the target table as the end point; The service registration module is used to check the entities in the graph database and register data services for the graph database; The blood relationship analysis module is used to perform data blood relationship analysis based on the graph database using a path search algorithm to obtain the blood relationship between data elements; The visualization module is used to input the blood relationship between data elements into the visualization engine to visualize the blood relationship between data elements.
[0011] In an optional implementation, the data extraction module includes: The data parsing unit is used to parse the source library information, target library information, field mapping information and dependencies between data tables in the data model using the ETL script; the source library information includes the database type, IP address, port, instance name, table name and account number of the source library; the target library information includes the database type, IP address, port, instance name, table name and account number of the target library.
[0012] In an optional embodiment, the map construction module includes: A model definition unit is used to build a data model in a graph database based on the entities, relationships, and attributes of a pre-set data model; The model building unit is used to fill the data source library information, target library information, field mapping information and dependency relationships between data tables into the data model.
[0013] In an optional implementation, the service registration module includes: The entity checking unit is used to traverse the entities in the data model, update the attributes of the entity if it exists, and create the entity if it does not exist; A standard definition unit for defining GraphQL models using the GraphQL Schema definition language; A schema configuration unit, used to put the GraphQL model into a GraphQL schema of a graph database, wherein the graph database is Neo4j; An interface enabling unit, used for enabling a GraphQL API of the graph database; The interface query unit is used to query the GraphQL API through a GraphQL client or tool, store the information of the GraphQL API, and complete the data service registration.
[0014] In a third aspect, a terminal is provided, including: processor, memory, wherein: The memory is used to store computer programs. The processor is used to call and run the computer program from the memory, so that the terminal executes the above-mentioned terminal method.
[0015] According to a fourth aspect, a computer storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the methods described in the above aspects.
[0016] The beneficial effects of the present invention are that the data lineage full-link monitoring method, system, terminal and storage medium provided by the present invention use a graph database model to store data, which is convenient for representing and querying complex relationships; use algorithms to perform data lineage analysis to track the transmission and processing of data in the system; and use the graph algorithm of the graph database to perform data analysis and visualization.
[0017] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.
[0020] Figure 2 is a schematic block diagram of a system according to an embodiment of the present invention.
[0021] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0022] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0024] The key terms appearing in the present invention are explained below.
[0025] Neo4j uses graph-related concepts to describe data models, saving data as nodes in the graph and the relationships between nodes. Data mainly consists of three parts: nodes. Nodes represent object instances. Each node has a unique ID to distinguish other nodes. Nodes have attributes; relationships. They are edges in the graph, connecting two nodes. In addition, the relationship here is directed and has attributes; attributes. Key-value pairs exist in nodes and relationships. Compared with other databases, Neo4j has the following obvious advantages: 1. Lightning read / write speed and unparalleled high performance; 2. Unstructured data storage method with great flexibility in database design; 3. It can adapt well to changes in demand and is suitable for agile development methods; 4. It is very easy to use and can be used in embedded, server mode, distributed mode and other ways; 5. Data models can be designed using simple block diagrams, which is convenient for modeling; 6. The structural characteristics of graph data can provide more and better algorithm designs; 7. Full support for ACID complete transaction management features; 8. Provide distributed high availability mode to support large-scale data growth; 9. The database is safe and reliable, can back up data in real time, and is very convenient to restore data; 10. The data structure of the graph intuitively and vividly represents the application scenarios of the real world.
[0026] G6 is a basic technical framework for relationship diagrams written in pure JavaScript. Developers can quickly perform secondary development on the relationship diagram view and edit view based on G6. It has the following features: 1. Easy-to-use graphics, interactive expansion and customization. 2. Exquisite, easy-to-use, and rich relationship diagrams. 3. Contains efficient and rich layout algorithms. 4. Contains basic relationship diagram editing behaviors.
[0027] The data lineage full-link monitoring method provided in the embodiment of the present invention is executed by a computer device, and accordingly, the data lineage full-link monitoring system runs in the computer device.
[0028] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject can be a data lineage full-link monitoring system. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.
[0029] like Figure 1 As shown, the method includes: Step 110, parsing the source database information, target database information, field mapping information and dependencies between data tables through a data extraction script; Step 120, based on the source database information, target database information, field mapping information and dependencies between data tables, the relationships and attributes between data sources, data tables and fields are constructed in the graph database, and a directed relationship is created with the source table as the starting point and the target table as the end point; Step 130, checking entities in the graph database and registering data services for the graph database; Step 140, using a path search algorithm to perform data lineage analysis based on a graph database to obtain lineage relationships between data elements; Step 150, input the blood relationship between the data elements into the visualization engine to visualize the blood relationship between the data elements.
[0030] To facilitate understanding of the present invention, the data lineage full-link monitoring method provided by the present invention is further described below based on the principle of the data lineage full-link monitoring method of the present invention and in combination with the process of performing full-link monitoring of data lineage in the embodiment.
[0031] Specifically, the data lineage full-link monitoring method includes: S1. Analyze the source database information, target database information, field mapping information and dependencies between data tables through data extraction scripts.
[0032] The ETL script is used to parse the source library information, target library information, field mapping information and dependencies between data tables in the data model; the source library information includes the database type, IP address, port, instance name, table name, and account number of the source library; the target library information includes the database type, IP address, port, instance name, table name, and account number of the target library.
[0033] Specifically, through the ETL script, parse the source database, target database and other related information (database type, IP address, port, instance name, table name, account number, etc.) and field mapping information, and parse the dependencies between tables in various SQL. The following steps are included: In the field of data warehouse and big data processing, ETL (Extract, Transform, Load) scripts are responsible for extracting data from source systems (such as relational databases, file systems, etc.), and after necessary transformations, loading it into target systems (such as data warehouses, data lakes, etc.). In order to complete this process efficiently and accurately, ETL scripts need to be able to parse and process a variety of complex information including database connection information, table structure, field mapping, and SQL dependencies. The following is a detailed expansion of this process: 1. Analyze the relevant information of the source and target libraries Database type: First, the ETL script needs to identify the types of source and target databases (such as MySQL, Oracle, SQLServer, PostgreSQL, etc.), because different types of databases may differ in connection, query syntax, and functions.
[0034] Connection information: includes the database IP address, port number, instance name (if applicable), database name, and the account and password used to connect to the database. This information is usually stored in configuration files or environment variables so that the ETL script can be dynamically loaded at runtime.
[0035] Table name and table structure: The ETL script needs to query the table names and their structures (such as field names, data types, primary keys, foreign keys, etc.) involved in the source and target libraries. This is usually achieved by executing SQL queries (such as DESCRIBE or INFORMATION_SCHEMA queries).
[0036] 2. Field mapping information Mapping rules: During the data migration from the source database to the target database, fields may need to be renamed, converted, or merged. The ETL script needs to parse and apply these mapping rules to ensure data consistency and accuracy.
[0037] Metadata management: Use metadata management tools or custom metadata tables to store and manage field mapping information, so that these mapping rules can be easily shared and reused among multiple ETL tasks.
[0038] 3. Analyze the dependencies between tables in SQL Dependency analysis: In a complex database environment, there are often complex dependencies between tables (such as foreign key constraints, view references, stored procedure calls, etc.). ETL scripts need to be able to parse these dependencies to determine the order of data extraction and loading to avoid data inconsistency or circular dependency issues.
[0039] Tools and technologies: Use the dependency analysis function in tools provided by the database management system (DBMS) (such as Oracle's Data Pump, SQL Server's Integration Services, etc.) or third-party ETL tools (such as Talend, Informatica, Apache NiFi, etc.) to automatically identify and resolve table dependencies in SQL.
[0040] Dynamic adjustment: In some cases, the ETL script may need to dynamically adjust the execution plan based on the actual data dependencies to ensure data integrity and consistency.
[0041] 4. Integration and implementation Script writing: Based on the information analyzed above, write ETL scripts, including the logic of data extraction, transformation, and loading.
[0042] Testing and Validation: Before deploying the ETL script to the production environment, perform adequate testing to ensure that the script can correctly handle various edge cases and exceptions.
[0043] S2. Based on the source database information, target database information, field mapping information and dependencies between data tables, the relationships and properties between data sources, data tables and fields are constructed in the graph database, and a directed relationship is created with the source table as the starting point and the target table as the end point.
[0044] Based on the entities, relationships and attributes of the pre-set data model, a data model is constructed in the graph database; the source library information, target library information, field mapping information and dependencies between data tables are filled into the data model.
[0045] Specifically, the following steps are included: 1. Define the node type in the graph database Data Source Node: represents the original source of data, such as relational database, file system, API, etc. Each data source node contains its unique identifier, type, IP address, port, authentication information and other attributes.
[0046] Table Node: represents a table or view in a data source. Each table node contains its name, data source, table structure (such as field list, data type), primary key, foreign key and other properties.
[0047] Field Node: represents a column or field in a data table. Each field node contains properties such as its name, data type, whether it is a primary key / foreign key, and the table it belongs to.
[0048] 2. Define the relationship type in the graph database Contains relationship: points from the data source node to the data table node, indicating which data tables a data source contains.
[0049] Field relationship (HasField): points from the data table node to the field node, indicating which fields a data table contains.
[0050] MapsTo: A special relationship between field nodes that indicates how source table fields are mapped to target table fields (may involve renaming, type conversion, etc.). This relationship may require the creation of additional mapping nodes between field nodes to represent complex mapping logic.
[0051] DependsOn: A directed relationship between data table nodes, indicating that one table (the dependent party) is logically dependent on the data of another table (the dependent party). It is used to represent foreign key relationships or the order of precedence in data processing flows.
[0052] 3. Build a network of relationships Initialize nodes: First, create corresponding data source nodes, data table nodes, and field nodes in the graph database based on the data source information, target library information, field mapping information, and data table structure.
[0053] Establish a containment relationship: For each data table, connect it to the corresponding data source through a "containment relationship".
[0054] Establish field relationships: Add field nodes to each data table node and connect them through "field relationships".
[0055] Establish a mapping relationship: According to the field mapping information, establish a "mapping relationship" between the field nodes of the source table and the field nodes of the target table. If the mapping logic is complex, you can create a mapping node to describe the conversion process in detail.
[0056] Establish dependencies: Establish "dependencies" between data table nodes based on dependencies between data tables (such as foreign key constraints, order in ETL process, etc.). These relationships should be directed, starting with the source table and ending with the target table.
[0057] S3. Check the entities in the graph database and register data services for the graph database.
[0058] Data resource registration - register data sources and data resources in sequence; registering data resources triggers the Neo4j entity check of the graph database. If the entity exists, update the attributes and complete the data table resources and other related information; if the entity does not exist, create the entity and create the data table resources and other related information.
[0059] Data service publishing - When converting data resources into data services, trigger the graph database Neo4j entity check, check the data resources and data services in turn, and create a directed relationship with the data resource as the starting point and the data service as the end point. If the data resource entity exists, update the attributes and improve the data table resource and other related information; if the data resource entity does not exist, create the entity and create the data table resource and other related information. If the data service entity exists, update the attributes and improve the data service and other related information; if the data service entity does not exist, create the entity and create the data service and other related information. And create a directed relationship with the data resource as the starting point and the data service as the end point. Create a scheduled task with the preset parameters when registering the service, and periodically poll and call the service to ensure the performance and availability of the monitoring data service.
[0060] Data application application - When applying for an application, the graph database Neo4j entity check is triggered, and the data service and data application are checked in turn, and a directed relationship is created with the data service as the starting point and the data application as the end point. If the data service entity exists, the attribute is updated and the relevant information of the data service is improved; if the data service entity does not exist, the entity is created and the relevant information of the data service is created. If the data application entity exists, the attribute is updated and the relevant information of the data application and other related information is improved; if the data application entity does not exist, the entity is created and the relevant information of the data application and other related information is created. A directed relationship is created with the data service as the starting point and the data application as the end point. Create a scheduled task to periodically check the validity of the data application application to ensure that the data application application has not expired.
[0061] Specifically, traverse the entities in the data model, update the attributes of the entity if the entity exists, and create the entity if the entity does not exist; use the GraphQL Schema definition language to define the GraphQL model; put the GraphQL model into the GraphQL mode of the graph database, which is Neo4j; enable the GraphQL API of the graph database; query the GraphQL API through a GraphQL client or tool, store the information of the GraphQL API, and complete the data service registration.
[0062] S4. Use the path search algorithm to perform data lineage analysis based on the graph database to obtain the lineage relationship between data elements.
[0063] The path search algorithm is used to recursively traverse the graph to find the path from the starting point to the end point, but it is not guaranteed to be the shortest path. In data lineage analysis, DFS can be used to find the complete path of data elements.
[0064] Data lineage analysis is usually conducted according to the level of data lineage, showing the data flow process in sequence. The goal of data lineage analysis is to accurately track the source of data, accurately restore the flow process, and accurately locate the destination of data.
[0065] Data lineage analysis is aimed at analyzing the data flow process, implementing algorithms to track the flow path of metadata, and determining the complete path from one data element to another. It also derives related information based on data elements, such as data collection log query, abnormal log tracking, data resource statistics, service call status, and application application information query.
[0066] S5. Input the blood relationship between the data elements into the visualization engine to visualize the blood relationship between the data elements.
[0067] After the lineage analysis is completed, it is necessary to rely on visualization technology to clearly and intuitively convey the analysis results to users to help customers conduct secondary analysis and specific applications. This module provides an intuitive visualization interface to display data lineage relationships. It uses the visualization engine G6 to achieve visual display of data lineage relationships and supports dynamic interactions, such as zooming in, zooming out, and highlighting. Use layout algorithms, such as force-directed layout and hierarchical layout, to optimize graph layout and improve readability. Design a friendly interactive interface to facilitate user operations, provide various types of data drill-down, and mark abnormalities in red. Support custom views to meet the needs of different users.
[0068] In some embodiments, the data lineage full-link monitoring system may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the data lineage full-link monitoring system may be stored in the memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) The function of monitoring the entire link of data lineage.
[0069] In this embodiment, the data lineage full-link monitoring system can be divided into multiple functional modules according to the functions it performs, such as Figure 2 As shown. The functional modules of the system 200 may include: a data extraction module 210, a graph construction module 220, a service registration module 230, a blood relationship analysis module 240 and a visualization module 250. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.
[0070] The data extraction module is used to parse the source database information, target database information, field mapping information and the dependencies between data tables through the data extraction script; The graph construction module is used to construct the relationship and attributes between data sources, data tables, and fields in the graph database based on the source database information, target database information, field mapping information, and dependencies between data tables, and to create a directed relationship with the source table as the starting point and the target table as the end point; The service registration module is used to check the entities in the graph database and register data services for the graph database; The blood relationship analysis module is used to perform data blood relationship analysis based on the graph database using a path search algorithm to obtain the blood relationship between data elements; The visualization module is used to input the blood relationship between data elements into the visualization engine to visualize the blood relationship between data elements.
[0071] Optionally, as an embodiment of the present invention, the data extraction module includes: The data parsing unit is used to parse the source library information, target library information, field mapping information and dependencies between data tables in the data model using the ETL script; the source library information includes the database type, IP address, port, instance name, table name and account number of the source library; the target library information includes the database type, IP address, port, instance name, table name and account number of the target library.
[0072] Optionally, as an embodiment of the present invention, the graph construction module includes: A model definition unit is used to build a data model in a graph database based on the entities, relationships, and attributes of a pre-set data model; The model building unit is used to fill the data source library information, target library information, field mapping information and dependency relationships between data tables into the data model.
[0073] Optionally, as an embodiment of the present invention, the service registration module includes: The entity checking unit is used to traverse the entities in the data model, update the attributes of the entity if it exists, and create the entity if it does not exist; A standard definition unit for defining GraphQL models using the GraphQL Schema definition language; A schema configuration unit, used to put the GraphQL model into a GraphQL schema of a graph database, wherein the graph database is Neo4j; An interface enabling unit, used for enabling a GraphQL API of the graph database; The interface query unit is used to query the GraphQL API through a GraphQL client or tool, store the information of the GraphQL API, and complete the data service registration.
[0074] Figure 3 A schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention, wherein the terminal 300 can be used to execute the data lineage full-link monitoring method provided in an embodiment of the present invention.
[0075] The terminal 300 may include: a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0076] The memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can perform some or all of the steps in the following method embodiments.
[0077] The processor 310 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 310 can only include a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.
[0078] The communication unit 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.
[0079] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0080] Therefore, the present invention uses a graph database model to store data, which is convenient for representing and querying complex relationships; uses algorithms to perform data lineage analysis and track the transmission and processing of data in the system; and uses the graph algorithm of the graph database to perform data analysis and visualization. The technical effects that can be achieved by this embodiment can be found in the description above and will not be repeated here.
[0081] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention.
[0082] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.
[0083] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.
[0084] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0085] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
[0086] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.
Claims
1. A data lineage full-link monitoring method, characterized in that: include: Analyze the source database information, target database information, field mapping information and dependencies between data tables through data extraction scripts; Based on the source database information, target database information, field mapping information and the dependencies between data tables, the relationships and properties between data sources, data tables and fields are constructed in the graph database, and a directed relationship is created with the source table as the starting point and the target table as the end point. Check the entities in the graph database and register data services for the graph database; Use the path search algorithm to perform data lineage analysis based on the graph database to obtain the lineage relationship between data elements; The blood relationship between the data elements is input into the visualization engine to visualize the blood relationship between the data elements.
2. The method according to claim 1, characterized in that The data extraction script is used to parse the source database information, target database information, field mapping information, and dependencies between data tables, including: The ETL script is used to parse the source library information, target library information, field mapping information and dependencies between data tables in the data model; the source library information includes the database type, IP address, port, instance name, table name, and account number of the source library; the target library information includes the database type, IP address, port, instance name, table name, and account number of the target library.
3. The method according to claim 1, characterized in that Based on the source database information, target database information, field mapping information and the dependencies between data tables, the relationship and attributes between data sources, data tables and fields are constructed in the graph database, and a directed relationship is created with the source table as the starting point and the target table as the end point, including: Build data models in graph databases based on pre-set data model entities, relationships, and attributes; The data source database information, target database information, field mapping information and dependencies between data tables are filled into the data model.
4. The method according to claim 3, characterized in that Check the entities in the graph database and register data services for the graph database, including: Traverse the entities in the data model, update the attributes of the entity if it exists, and create the entity if it does not exist; Use the GraphQL Schema definition language to define GraphQL models; Put the GraphQL model into the GraphQL schema of a graph database, the graph database is Neo4j; Enable the GraphQL API of the graph database; Use a GraphQL client or tool to query the GraphQL API, store the GraphQL API information, and complete the data service registration.
5. A data lineage full-link monitoring system, characterized in that: include: The data extraction module is used to parse the source database information, target database information, field mapping information and the dependencies between data tables through the data extraction script; The graph construction module is used to construct the relationship and attributes between data sources, data tables, and fields in the graph database based on the source database information, target database information, field mapping information, and dependencies between data tables, and to create a directed relationship with the source table as the starting point and the target table as the end point; The service registration module is used to check the entities in the graph database and register data services for the graph database; The blood relationship analysis module is used to perform data blood relationship analysis based on the graph database using a path search algorithm to obtain the blood relationship between data elements; The visualization module is used to input the blood relationship between data elements into the visualization engine to visualize the blood relationship between data elements.
6. The system according to claim 5, characterized in that The data extraction module comprises: The data parsing unit is used to parse the source library information, target library information, field mapping information and dependencies between data tables in the data model using the ETL script; the source library information includes the database type, IP address, port, instance name, table name and account number of the source library; the target library information includes the database type, IP address, port, instance name, table name and account number of the target library.
7. The system according to claim 5, characterized in that The graph construction module includes: A model definition unit is used to build a data model in a graph database based on the entities, relationships, and attributes of a pre-set data model; The model building unit is used to fill the data source library information, target library information, field mapping information and dependency relationships between data tables into the data model.
8. The system according to claim 7, characterized in that The service registration module includes: The entity checking unit is used to traverse the entities in the data model, update the attributes of the entity if it exists, and create the entity if it does not exist; A standard definition unit for defining GraphQL models using the GraphQL Schema definition language; A schema configuration unit, used to put the GraphQL model into a GraphQL schema of a graph database, wherein the graph database is Neo4j; An interface enabling unit, used for enabling a GraphQL API of the graph database; The interface query unit is used to query the GraphQL API through a GraphQL client or tool, store the information of the GraphQL API, and complete the data service registration.
9. A terminal, characterized in that: include: A memory device for storing a data lineage full-link monitoring program; A processor, used to implement the steps of the data lineage full-link monitoring method as described in any one of claims 1-4 when executing the data lineage full-link monitoring program.
10. A computer-readable storage medium storing a computer program, characterized in that: The readable storage medium stores a data lineage full-link monitoring program, which, when executed by a processor, implements the steps of the data lineage full-link monitoring method as described in any one of claims 1-4.
Citation Information
Cited By
Data consanguinity full-link traceability method and system based on multi-source heterogeneous metadata and pre-training large model
CN119917814A
Data blood relationship analysis method and system, terminal and medium
CN120045614A
Data blood relationship tracking method and device, computer equipment and storage medium
CN120256506A
Nifi data consanguinity map construction method and device, terminal and medium
CN120763363A
Multi-modal data lineage modeling method and device, electronic equipment and storage medium
CN121117265A
