Metadata blood relationship analysis method, device and equipment based on Neo4j and medium

Abstract

The invention discloses a Neo4j-based metadata blood relationship analysis method and device, equipment and a medium, and relates to the technical field of computers, and the method comprises the following steps: carrying out relationship graph construction on metadata of an obtained data source in a Neo4j library according to metadata entity information, entity attribute information and a metadata association relationship of a preset blood relationship graph, obtaining a metadata blood relationship graph; when a data analysis task is obtained, querying an association node and a target edge associated with a target node where target metadata is located in a metadata blood relationship graph of a Neo4j library through target metadata entity attribute information of a to-be-analyzed metadata blood relationship in the data analysis task and a preset query statement; and analyzing a transmission process and a conversion process of the target metadata corresponding to the target node through the association node and the target edge. The efficiency of metadata blood relationship analysis is improved, complex data relationship calculation is simplified, the increase of data scales and data source types can be coped with, and expansibility is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a metadata kinship analysis method, device, equipment and medium based on Neo4j. Background Art

[0002] With the rapid growth of data volume and the widespread popularity of data applications, the importance of metadata management has become increasingly prominent. Metadata lineage analysis can help enterprises understand the source, flow and evolution of data, thereby improving the efficiency of data governance and data quality management. However, existing metadata lineage analysis methods have problems such as complex calculations, low efficiency, and difficulty in scalability. The specific problems are as follows:

[0003] Calculation complexity: Traditional methods require huge amounts of calculation when dealing with complex data relationships, resulting in low analysis efficiency.

[0004] Inefficiency: Existing technologies often require a lot of computing resources and time when performing large-scale data lineage analysis.

[0005] Difficult to expand: Existing systems often require a lot of modifications and adjustments when faced with increases in data scale and data source types, and have poor scalability.

[0006] Therefore, a convenient, efficient and flexible metadata lineage collection and analysis method is urgently needed. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for analyzing metadata kinship based on Neo4j, which can automatically update metadata and kinship in the Neo4j library when data flows and data structures change. The specific scheme is as follows:

[0008] In a first aspect, the present application discloses a metadata kinship analysis method based on Neo4j, comprising:

[0009] In the Neo4j library, a relationship diagram is constructed for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset lineage relationship diagram to obtain a metadata lineage relationship diagram;

[0010] When a data analysis task is obtained, the target metadata entity attribute information of the metadata lineage relationship to be analyzed in the data analysis task and the preset query statement are used to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata lineage relationship diagram of the Neo4j library; wherein the edge represents the information flow path between the target node and other nodes;

[0011] The transfer process and the conversion process of the target metadata corresponding to the target node are analyzed through the associated node and the target edge.

[0012] Optionally, before constructing a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset blood relationship graph in the Neo4j library, the method further includes:

[0013] Determine the type information of each data source, and obtain the business process information of each data source;

[0014] Determine a metadata entity using the type information and the business process information to obtain corresponding metadata entity information;

[0015] Determine the attribute information required for each metadata entity according to business requirements to determine the corresponding entity attribute information;

[0016] A preset blood relationship graph is constructed using the metadata entity information and the entity attribute information required by the metadata entity.

[0017] Optionally, the metadata blood relationship analysis method based on Neo4j further includes:

[0018] The metadata association relationship between the metadata is defined in the preset blood relationship diagram, wherein the metadata association relationship includes an inclusion relationship, a data flow relationship, and a dependency relationship of the business process in the business process information on the metadata entity.

[0019] Optionally, before constructing a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset blood relationship graph in the Neo4j library, the method further includes:

[0020] A corresponding preset metadata collection tool is selected according to the type information of each data source, so as to collect metadata of the data source from the data source.

[0021] Optionally, the metadata blood relationship analysis method based on Neo4j further includes:

[0022] When a metadata update demand is detected, the current metadata is obtained as the metadata of the data source, and the step of constructing a relationship graph for the metadata of the obtained data source in the Neo4j library according to the metadata entity information, entity attribute information, and metadata association relationship of the preset lineage graph is jumped to obtain a metadata lineage graph; wherein the metadata update demand is an update demand generated when a new data source is monitored in real time, or a new metadata structure exists in the data source, or the metadata structure of the data source is changed.

[0023] Optionally, the metadata blood relationship analysis method based on Neo4j further includes:

[0024] When a historical metadata tracing requirement is detected, the historical metadata and the corresponding historical metadata lineage relationship graph are obtained from a preset storage location through the version control mechanism in the Neo4j library and based on the target version number in the historical metadata tracing requirement, and the historical metadata lineage relationship graph is used as a new metadata lineage relationship graph.

[0025] Optionally, when obtaining the data analysis task, querying the associated nodes and target edges associated with the target node where the target metadata is located in the metadata kinship graph of the Neo4j library through the target metadata entity attribute information of the metadata kinship to be analyzed in the data analysis task and a preset query statement, including:

[0026] When a data analysis task is obtained, a data source and a data source target of the metadata blood relationship to be analyzed are obtained from the data analysis task;

[0027] Constructing a query clause including information flow from the data source to the data source target as a preset query statement;

[0028] The preset query statement is executed by using the query execution tool of the Neo4j library to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata kinship graph of the Neo4j library.

[0029] In a second aspect, the present application discloses a metadata blood relationship analysis device based on Neo4j, comprising:

[0030] A graph construction module is used to construct a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset blood relationship graph in the Neo4j library to obtain a metadata blood relationship graph;

[0031] An information query module is used to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata kinship graph of the Neo4j library through the target metadata entity attribute information of the metadata kinship to be analyzed in the data analysis task and the preset query statement when obtaining the data analysis task; wherein the edge represents the information flow path between the target node and other nodes;

[0032] The information analysis module is used to analyze the transmission process and conversion process of the target metadata corresponding to the target node through the associated node and the target edge.

[0033] In a third aspect, the present application discloses an electronic device, comprising:

[0034] Memory, used to store computer programs;

[0035] The processor is used to execute the computer program to implement the steps of the aforementioned Neo4j-based metadata lineage relationship analysis method disclosed above.

[0036] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned Neo4j-based metadata kinship analysis method are implemented.

[0037] It can be seen that the present application discloses a metadata kinship analysis method based on Neo4j, including: constructing a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset kinship graph in the Neo4j library to obtain the metadata kinship graph; when obtaining a data analysis task, querying the associated nodes and target edges associated with the target node where the target metadata is located in the metadata kinship graph of the Neo4j library through the target metadata entity attribute information of the metadata kinship to be analyzed in the data analysis task and the preset query statement; wherein the edge represents the information flow path between the target node and other nodes; analyzing the transmission process and conversion process of the target metadata corresponding to the target node through the associated nodes and the target edges. It can be seen that the efficiency of metadata kinship analysis is greatly improved by using the graph computing capability of Neo4j. Complex data relationship calculations are simplified through relational modeling and query of the graph database. The system is flexible in design and can easily cope with the increase in data scale and data source types, and has good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A flowchart of a metadata blood relationship analysis method based on Neo4j disclosed in this application;

[0040] Figure 2 This is a schematic diagram of the structure of a metadata blood relationship analysis device based on Neo4j disclosed in this application;

[0041] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] With the rapid growth of data volume and the widespread popularity of data applications, the importance of metadata management has become increasingly prominent. Metadata lineage analysis can help enterprises understand the source, flow and evolution of data, thereby improving the efficiency of data governance and data quality management. However, existing metadata lineage analysis methods have problems such as complex calculations, low efficiency, and difficulty in scalability. The specific problems are as follows:

[0044] Calculation complexity: Traditional methods require huge amounts of calculation when dealing with complex data relationships, resulting in low analysis efficiency.

[0045] Inefficiency: Existing technologies often require a lot of computing resources and time when performing large-scale data lineage analysis.

[0046] Difficult to expand: Existing systems often require a lot of modifications and adjustments when faced with increases in data scale and data source types, and have poor scalability.

[0047] To this end, the present invention provides a metadata lineage relationship analysis solution based on Neo4j, which can automatically update metadata and lineage relationships in the Neo4j library when data flows and data structures change.

[0048] Reference Figure 1 As shown, the embodiment of the present invention discloses a metadata kinship analysis method based on Neo4j, comprising:

[0049] Step S11: construct a relationship graph for the metadata of the acquired data source in the Neo4j library according to the metadata entity information, entity attribute information, and metadata association relationship of the preset lineage relationship graph to obtain a metadata lineage relationship graph.

[0050] In this embodiment, before constructing the relationship graph of the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset blood relationship graph in the Neo4j library, it also includes: determining the type information of each data source and obtaining the business process information of each data source; determining the metadata entity using the type information and the business process information to obtain the corresponding metadata entity information; determining the attribute information required for each metadata entity according to business needs to determine the corresponding entity attribute information; and constructing the preset blood relationship graph using the metadata entity information and the entity attribute information required for the metadata entity. It can be understood that metadata is data that describes data. It provides information about a certain data to help people understand, use and manage the data. The role of metadata is similar to that of a card catalog in a library, which can quickly find the required books and understand the basic information of the books. Metadata can be divided into the following categories: Descriptive Metadata: Attributes that describe the content of the data. For example, the metadata of a book may include title, author, publication date, and abstract. Structural Metadata: Information that describes the organization and relationship of data. For example, the catalog and chapter information of a book. Administrative Metadata: Information that helps manage and process data. For example, the creation date, format, access rights, and copyright information of the data. Technical Metadata: Describes the technical information required for data generation and use. For example, image resolution, file format, data collection tools and parameters. Usage Metadata: Information that records data usage. For example, the number of accesses, citations, and user feedback of the data. Therefore, in order to better conduct data governance and data quality management, understand the source, flow, and evolution of the data, and build a lineage relationship diagram to obtain the corresponding data information from the lineage relationship diagram through data visualization. Specifically, before building the lineage relationship diagram, pre-build a preset lineage relationship diagram. Specifically, conduct a comprehensive demand analysis of the data source to be managed. Determine the type of data source, including relational databases, data warehouses, file systems, etc. Understand the data storage and processing links involved in the business process, and clarify which parts need metadata management, that is, obtain the business process information of each data source. For example, in an enterprise data management system, analyzing the business process may reveal that it is necessary to manage the table structure in the database, the data files in the file system, and the various steps in the data processing process. Have a deep understanding of the business domain and data usage scenarios. Communicate with business people, data analysts, and developers to understand their needs for and how they use data. This helps determine which metadata entities are critical and their role in business processes.For example, in the financial field, the tables and columns of transaction data may be important metadata entities because they are directly related to the recording and analysis of transactions. Further, starting from business needs, determine which attributes each metadata entity needs to describe its characteristics and relationships to determine the corresponding entity attribute information. Specifically, consider the focus of data in the business process, such as the source, purpose, timeliness, etc. of the data. For table entities, it may be necessary to define attributes such as name, type (such as fact table, dimension table), and database. For column entities, it may be necessary to define attributes such as name, data type, and whether it is a primary key. After determining the metadata entity information (table, column, database, file, process, etc.) and the corresponding entity attribute information (primary key, name, type, path, data type, etc.), the metadata entity information and entity attribute information are used to construct a preset blood relationship graph in the Neo4j (Neo4j graph database) library, wherein each metadata entity information is used as a node, and each node has a specific type and attribute. Next, the metadata association relationship between each metadata is defined in the preset blood relationship graph, wherein the metadata association relationship includes a containment relationship, a data flow relationship, and a dependency relationship between the business process in the business process information and the metadata entity. It can be understood that the inclusion relationship is determined, such as the database contains tables, and the table contains columns; the data flow relationship is defined, indicating that data flows from one table or file to another table or file; and the dependency relationship is determined, such as a process depends on a table or file. The lineage relationship between metadata is represented by defining the inclusion relationship, data flow relationship, and dependency relationship. These relationships are created in Neo4j through Cypher (a declarative graph query language) query statements, and the corresponding nodes are connected to obtain the preset lineage relationship graph. It can be seen that by clarifying the main entities that need to be modeled in metadata, such as tables, columns, databases, files, and processes, a clear object framework is provided for building a metadata lineage relationship graph. This makes the structure of the entire data system clearer and helps to understand the roles and status of different types of data storage and processing units in the system. The description information of the entity is further enriched by defining entity attribute information. These attributes can help accurately identify and distinguish different entities, providing key feature identifiers for subsequent data storage, query, and analysis.

[0051] In this embodiment, before the relationship graph is constructed, the corresponding preset metadata collection tool is selected according to the type information of each data source, so as to collect the metadata of the data source from the data source. It can be understood that the metadata is obtained from multiple data sources (relational databases, data warehouses, data lakes, etc.) in different ways. For relational databases, the API (Application Program Interface) or driver provided by the database can be used to establish a connection. By executing specific query statements, such as querying the system table or view of the database, metadata information about tables, columns, indexes, etc. can be obtained. For example, in Java, JDBC (Java Database Connectivity) can be used to connect to the database, and then SQL (Structured Query Language) queries can be executed to obtain metadata. Common queries include obtaining table structure information (such as column name, data type, length, etc.), index information, constraints, etc. For file data sources, such as CSV (Comma-Separated Values) files, Excel (table) files, etc., corresponding file processing libraries can be used for reading. By parsing the structure and content of the file, metadata information about the file, such as file name, file path, file format, column name, etc., can be extracted. For example, the Apache POI library of Java can read the metadata of Excel files, including worksheet names, column names, data types, etc. If you use ETL (ExtractTransform Load) tools for data integration, you can use the functions provided by ETL tools to obtain metadata. ETL tools can usually connect to different data sources, extract metadata, and perform transformation and loading operations. For example, Informatica Power Center can extract metadata from various data sources and provide a visual interface to manage and analyze metadata. Data warehouse management tools usually provide rich metadata management functions. You can connect to the data warehouse to obtain metadata information such as tables, dimensions, and measures in the data warehouse. For example, Oracle Warehouse Builder can help users manage the metadata of the data warehouse, including data source definitions, transformation rules, data quality rules, etc. Data catalog tools can help users organize and manage metadata and provide metadata search, browsing, and sharing functions. These tools can usually connect to different data sources, collect metadata, and display it to users. For example, Alation Data Catalog can automatically discover metadata in data sources and provide a user-friendly interface to browse and search metadata.

[0052] The ways to obtain metadata information through the connection with the data source include directly connecting to the data source, using middleware or tools, and using automated metadata collection tools. These methods can be selected and combined according to specific needs and data source types to achieve efficient metadata collection and management. The metadata is stored in the Neo4j library through the Java Driver of the Neo4j library, so that nodes can be created and node properties can be set based on the preset blood relationship diagram.

[0053] In this embodiment, after defining a preset blood relationship graph in the Neo4j library, the metadata obtained from the data source through a Java program is used to construct a relationship graph. Specifically, the metadata entity of the metadata is determined based on the metadata and the metadata entity information in the defined preset blood relationship graph, and then the corresponding entity attribute is determined from the attribute information of the acquired metadata based on the defined entity attribute information in the preset blood relationship graph to obtain the metadata blood relationship graph.

[0054] In this embodiment, in order to facilitate the visual query data setting from the metadata kinship graph, a Cypher query statement is used to query the node through the ID (Identity document, unique code) attribute. Then the query can be executed in Java to obtain the query results and process them, such as printing the name and other attributes of the node. In addition, in order to optimize the query performance through the ID attribute, an index is created for the ID attribute in the Neo4j library.

[0055] In this embodiment, when a metadata update requirement is detected, the current metadata is obtained as the metadata of the data source, and the step of constructing a relationship graph of the metadata of the obtained data source in the Neo4j library according to the metadata entity information, entity attribute information, and metadata association relationship of the preset lineage graph is jumped to obtain the step of obtaining the metadata lineage graph; wherein the metadata update requirement is an update requirement generated when a new data source is detected in real time, or a new metadata structure exists in the data source, or the metadata structure of the data source is changed. It can be understood that when the metadata changes, the MERGE statement is used to update or create the node, and the attribute value of the node is set according to the specific situation.

[0056] In this embodiment, when the historical metadata tracing requirement is detected, the historical metadata and the corresponding historical metadata lineage relationship diagram are obtained from the preset storage location through the version control mechanism in the Neo4j library and based on the target version number in the historical metadata tracing requirement, and the historical metadata lineage relationship diagram is used as the new metadata lineage relationship diagram. It is understandable that, as time goes by, metadata may change for various reasons, such as data structure adjustment, business demand change, data processing flow optimization, etc. Through version management, the change history of metadata can be recorded so that the metadata status at a specific time point can be viewed when needed. Therefore, the introduction of the version control mechanism in the Neo4j library means that each time the metadata is updated, the system will not directly overwrite the old metadata, but retain the old version. In this way, a historical record of metadata can be established to facilitate subsequent tracing and analysis. Version management can be performed using timestamps or version numbers. Timestamps can record the specific time of metadata updates, while version numbers can be automatically generated by the system or manually specified by the user to identify different metadata versions. When metadata is updated, the system stores the old version of metadata in a specific location or node, and creates a new version node to store the updated metadata. In this way, the historical continuity of metadata can be maintained. For the relationship between metadata, corresponding version management is also required. For example, if the structure of a table changes, the metadata such as columns and indexes related to it and the relationship between them also need to be versioned. Cypher queries can query the metadata and lineage relationships of a specific version based on the version number or timestamp. This enables users to easily obtain metadata information and lineage relationship diagrams at a specific point in time when needed. It can be seen that version management can support data auditing and troubleshooting. During the data audit process, historical versions of metadata can be viewed to determine whether the source and processing of the data meet the specifications. When data problems occur, the time point and cause of the problem can be found by tracing back the historical metadata, so as to more effectively troubleshoot and solve the problem.

[0057] Step S12: When a data analysis task is obtained, the target metadata entity attribute information of the metadata lineage relationship to be analyzed in the data analysis task and the preset query statement are used to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata lineage relationship diagram of the Neo4j library; wherein the edge represents the information flow path between the target node and other nodes.

[0058] In this embodiment, when a data analysis task is obtained, the data source and data source target of the metadata lineage relationship to be analyzed are obtained from the data analysis task; an information flow query clause containing the information flow from the data source to the data source target is constructed as a preset query statement; and the query execution tool of the Neo4j library is used to execute the preset query statement to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata lineage relationship graph of the Neo4j library. It can be understood that, first, after obtaining the data analysis task, the data source and data target in the task must be analyzed first, that is, the source and end point of the data analysis. This can be a specific metadata entity such as a table, column, file or process. For example, suppose you want to find the lineage path from Table A to Table Z, Table A is the source, and Table Z is the target. Use the MATCH clause to describe the nodes and relationships to be queried. For example, the statement to query the lineage path from one table A to another table Z is expressed as follows: MATCH (source:Table {name: 'TableA'})-[:DATA_FLOW*]->(target:Table {name: 'TableZ'}). Among them, source and target represent the source table and target table respectively, and the DATA_FLOW relationship represents the flow of data. * represents a path of any length, that is, it can pass through multiple intermediate nodes. Use the RETURN clause to specify the results to be returned. Usually, the nodes and relationships on the path are returned so that the specific flow of data can be viewed. The RETURN clause is expressed as follows: RETURN nodes(path) AS nodes, relationships(path) AS relationships. This will return a list of nodes and a list of relationships on the path. Use Neo4j's query execution tool (such as Neo4j Browser or programmatically use Neo4j's driver) to execute the constructed Cypher query statement to obtain the node list that represents the passed associated nodes and the relationship list that represents the target edge of the existing blood relationship.

[0059] Step S13: Analyze the transfer process and conversion process of the target metadata corresponding to the target node through the associated node and the target edge.

[0060] In this embodiment, by analyzing all associated nodes, all metadata entities involved in the path will be displayed, including source tables, intermediate tables, and target tables, as well as possible columns, databases, and other entities. By analyzing the target edge, the specific relationship between these entities will be displayed, that is, how the data flows from one entity to another. In this way, by analyzing these results, the specific path of the data from the source to the target can be clearly seen, including the intermediate nodes and relationship types passed through. By analyzing the usage and heat index of metadata, analyzing the number of relationships and nodes associated with nodes in the relationship graph, the popularity and frequency of use of metadata can be understood. This analysis can help understand which metadata are widely concerned and used, and which metadata may need further optimization or updating. It can be seen that the use of Cypher queries can effectively analyze metadata lineage relationships, find out the path of data from the source to the target, and provide strong support for data governance, problem troubleshooting, and business decisions.

[0061] It can be seen that the present application discloses a metadata kinship analysis method based on Neo4j, including: constructing a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset kinship graph in the Neo4j library to obtain the metadata kinship graph; when obtaining a data analysis task, querying the associated nodes and target edges associated with the target node where the target metadata is located in the metadata kinship graph of the Neo4j library through the target metadata entity attribute information of the metadata kinship to be analyzed in the data analysis task and the preset query statement; wherein the edge represents the information flow path between the target node and other nodes; analyzing the transmission process and conversion process of the target metadata corresponding to the target node through the associated nodes and the target edges. It can be seen that the efficiency of metadata kinship analysis is greatly improved by using the graph computing capability of Neo4j. Complex data relationship calculations are simplified through relational modeling and query of the graph database. The system is flexible in design and can easily cope with the increase in data scale and data source types, and has good scalability.

[0062] Reference Figure 2 As shown, the present invention also discloses a metadata blood relationship analysis device based on Neo4j, including:

[0063] A graph construction module 11 is used to construct a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset blood relationship graph in the Neo4j library to obtain a metadata blood relationship graph;

[0064] The information query module 12 is used for, when obtaining a data analysis task, querying the associated nodes and target edges associated with the target node where the target metadata is located in the metadata kinship graph of the Neo4j library through the target metadata entity attribute information of the metadata kinship to be analyzed in the data analysis task and a preset query statement; wherein the edge represents the information flow path between the target node and other nodes;

[0065] The information analysis module 13 is used to analyze the transfer process and conversion process of the target metadata corresponding to the target node through the associated node and the target edge.

[0066] It can be seen that the present application discloses constructing a relationship graph of the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset lineage relationship graph in the Neo4j library to obtain the metadata lineage relationship graph; when a data analysis task is obtained, the target metadata entity attribute information of the metadata lineage relationship to be analyzed in the data analysis task and the preset query statement are used to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata lineage relationship graph of the Neo4j library; wherein the edge represents the information flow path between the target node and other nodes; the transmission process and conversion process of the target metadata corresponding to the target node are analyzed through the associated nodes and the target edges. It can be seen that the efficiency of metadata lineage analysis is greatly improved by using the graph computing capability of Neo4j. Complex data relationship calculations are simplified through relational modeling and query of the graph database. The system is flexible in design and can easily cope with the increase in data scale and data source types, and has good scalability.

[0067] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0068] Figure 3 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the metadata kinship analysis method based on Neo4j disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0069] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0070] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0071] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0072] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the metadata kinship analysis method based on Neo4j executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data transmitted from an external device received by the electronic device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0073] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned metadata kinship analysis method based on Neo4j is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0074] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0075] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented directly with hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory RAM (Random Access Memory), memory, read-only memory ROM (Read Only Memory), electrically programmable EPROM (Electrically Programmable Read Only Memory), electrically erasable programmable EEPROM (ElectricErasable Programmable Read Only Memory), register, hard disk, removable disk, CD-ROM (CompactDisc-Read Only Memory), or any other form of storage medium known in the technical field.

[0076] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0077] The scheme provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A metadata kinship analysis method based on Neo4j, characterized in that: include: In the Neo4j library, a relationship diagram is constructed for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset lineage relationship diagram to obtain a metadata lineage relationship diagram; When a data analysis task is obtained, the target metadata entity attribute information of the metadata lineage relationship to be analyzed in the data analysis task and the preset query statement are used to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata lineage relationship diagram of the Neo4j library; wherein the edge represents the information flow path between the target node and other nodes; The transfer process and the conversion process of the target metadata corresponding to the target node are analyzed through the associated node and the target edge.

2. The metadata kinship analysis method based on Neo4j according to claim 1, characterized in that: Before constructing a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset blood relationship graph in the Neo4j library, the method further includes: Determine the type information of each data source, and obtain the business process information of each data source; Determine a metadata entity using the type information and the business process information to obtain corresponding metadata entity information; Determine the attribute information required for each metadata entity according to business requirements to determine the corresponding entity attribute information; A preset blood relationship graph is constructed using the metadata entity information and the entity attribute information required by the metadata entity.

3. The method for analyzing metadata blood relationship based on Neo4j according to claim 2, characterized in that: Also includes: The metadata association relationship between the metadata is defined in the preset blood relationship diagram, wherein the metadata association relationship includes an inclusion relationship, a data flow relationship, and a dependency relationship of the business process in the business process information on the metadata entity.

4. The method for analyzing metadata blood relationship based on Neo4j according to claim 1, characterized in that: Before constructing a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset blood relationship graph in the Neo4j library, the method further includes: A corresponding preset metadata collection tool is selected according to the type information of each data source, so as to collect metadata of the data source from the data source.

5. The method for analyzing metadata blood relationship based on Neo4j according to claim 1, characterized in that: Also includes: When a metadata update demand is detected, the current metadata is obtained as the metadata of the data source, and the step of constructing a relationship graph for the metadata of the obtained data source in the Neo4j library according to the metadata entity information, entity attribute information, and metadata association relationship of the preset lineage graph is jumped to obtain a metadata lineage graph; wherein the metadata update demand is an update demand generated when a new data source is monitored in real time, or a new metadata structure exists in the data source, or the metadata structure of the data source is changed.

6. The method for analyzing metadata blood relationship based on Neo4j according to claim 1, characterized in that: Also includes: When a historical metadata tracing requirement is detected, the historical metadata and the corresponding historical metadata lineage relationship graph are obtained from a preset storage location through the version control mechanism in the Neo4j library and based on the target version number in the historical metadata tracing requirement, and the historical metadata lineage relationship graph is used as a new metadata lineage relationship graph.

7. The method for analyzing metadata blood relationship based on Neo4j according to any one of claims 1 to 6, characterized in that: When the data analysis task is obtained, the target metadata entity attribute information of the metadata lineage relationship to be analyzed in the data analysis task and the preset query statement are used to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata lineage relationship graph of the Neo4j library, including: When a data analysis task is obtained, a data source and a data source target of the metadata blood relationship to be analyzed are obtained from the data analysis task; Constructing a query clause including information flow from the data source to the data source target as a preset query statement; The preset query statement is executed by using the query execution tool of the Neo4j library to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata kinship graph of the Neo4j library.

8. A metadata blood relationship analysis device based on Neo4j, characterized in that: include: A graph construction module is used to construct a relationship graph for the metadata of the acquired data source according to the metadata entity information, entity attribute information, and metadata association relationship of the preset blood relationship graph in the Neo4j library to obtain a metadata blood relationship graph; An information query module is used to query the associated nodes and target edges associated with the target node where the target metadata is located in the metadata kinship graph of the Neo4j library through the target metadata entity attribute information of the metadata kinship to be analyzed in the data analysis task and the preset query statement when obtaining the data analysis task; wherein the edge represents the information flow path between the target node and other nodes; The information analysis module is used to analyze the transmission process and conversion process of the target metadata corresponding to the target node through the associated node and the target edge.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, used to execute the computer program to implement the steps of the metadata blood relationship analysis method based on Neo4j as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the metadata blood relationship analysis method based on Neo4j are implemented as described in any one of claims 1 to 7.

Images