A distributed data bloodline construction and display method

Abstract

The application discloses a kind of distributed data blood relationship construction and show method, belong to visual data resource analysis show technical field.The method includes the following steps: step SS1: data blood relationship construction step, including: respectively through distributed metadata acquisition, data processing blood relationship acquisition, data access middleware object resolution and data storage access monitoring generate blood relationship data;Step SS2: the blood relationship data is analyzed and integrated by data analysis processing module to data blood relationship, generate data blood relationship;Step SS3: the data blood relationship is stored in graph database, for user query and visual display.The application is in the process of data platform, by metadata acquisition realizes automatic acquisition analysis data and artificial collection realizes data blood relationship construction, simultaneously provides a kind of relationship exploration technology, discovers potential correlation between data resources Existence, support for the flow management of data.

Description

TECHNICAL FIELD

[0001] The present application relates to a distributed data blood relationship construction and display method, belonging to the technical field of visual data resource analysis and display. BACKGROUND

[0002] Any data, from generation, ETL processing, processing, fusion, circulation, to final extinction, data will naturally form a relationship. Analogous to human interpersonal relationships, a similar relationship is used to express the relationship between data, called data blood relationship. Data blood relationship is a concept in data governance, which finds the relationship between related data in the data traceability process, and it is a logical concept. Blood relationship analysis is part of data governance, and blood relationship analysis is a means to ensure data fusion, and through blood relationship analysis, data fusion processing is traceable. The data blood relationship of big data refers to the link of data generation, which records how the data comes and what processes and stages it has gone through.

[0003] The general blood relationship analysis method is to analyze blood relationship of different granularity, develop different data blood relationship analysis schemes, and display data flow in a graphical way to help users understand complex blood relationship and realize data blood relationship collection, data blood relationship analysis and data blood relationship display.

[0004] The disadvantages of the prior art are: (1) In the current blood relationship analysis technology, the metadata collection step mainly faces data warehouses, and collects the metadata of the tables in the data warehouse through the "API direct connection method". With the rapid development of business, the demand for data operation and cost management is becoming more and more intense, and the collection of metadata needs to cover the whole life cycle of data, including databases, offline computing services, online computing services, data center components, computing tasks, etc.; (2) The current blood relationship analysis mainly analyzes the data in the data center, but in a large distributed system, the data is distributed in each node, and is limited by network and permission factors, so it is difficult to physically converge data and analyze blood relationship. The present application proposes a distributed blood relationship analysis method to solve this problem. SUMMARY

[0005] The purpose of the present application is to overcome the technical defects of the prior art, solve the above technical problems, and propose a distributed data blood relationship construction and display method.

[0006] The present application specifically adopts the following technical solutions: A distributed data blood relationship construction and display method, comprising the following steps:

[0007] Step SS1: data blood relationship construction step, comprising: generating blood relationship data by distributed metadata collection, data processing blood relationship acquisition, data access middleware object analysis and data storage access monitoring, respectively;

[0008] Step SS2: The blood relationship data is parsed and integrated by a data parsing processing module to generate data blood relationship;

[0009] Step SS3: The data blood relationship is stored in a graph database for user query and visual presentation.

[0010] As a preferred embodiment, the distributed metadata collection in step SS1 includes: each node acquires metadata in the data life cycle as a collection object to create a data source, the collection object including a database, an offline computing service, an online computing service, a data middle platform component, and a computing task; a collection task is configured for the data source, and then metadata collection is performed, the collected metadata is labeled, and then the metadata is written into a database and cross-node metadata aggregation and deduplication fusion are performed.

[0011] As a preferred embodiment, the distributed metadata collection in step SS1 specifically further includes: for the database as the collection object, a collection method of database access is used, and the collection content is table name, note, field list, primary key, foreign key, table size, row number, file number, partition number, and upstream / downstream dependency relationship of table / field.

[0012] As a preferred embodiment, the distributed metadata collection in step SS1 specifically further includes: for the offline computing service as the collection object, a collection method of calling a computing service interface is used, and the collection content is Hive / RDS table metadata, the table metadata including file status, file number, file size, and trend data of data update time; for the online computing service as the collection object, a collection method of accessing service landing work order data to acquire basic metadata information of a computing theme is used, and the collection content is metadata of Flume / Hbase / Kafka components.

[0013] As a preferred embodiment, the distributed metadata collection in step SS1 specifically further includes: for the data middle platform component as the collection object, a collection method of synchronizing component data to a database and offline extracting metadata is used, and the collection content is blood data of BI report system, index library, and OneService service.

[0014] As a preferred embodiment, the distributed metadata collection in step SS1 specifically further includes: for the computing task as the collection object, a collection method of parsing task input / output dependency configuration or parsing blood relationship of tables / fields in a computing script is used, and the collection content is name, responsible person, deadline alarm time, script, and task configuration information of offline / real-time computing task.

[0015] As a preferred embodiment, the writing metadata into the database and the cross-node metadata aggregation and deduplication fusion specifically comprises: a centralized metadata aggregation and fusion mode is adopted, that is, a central node is taken as a master node, and other data nodes are taken as slave nodes, and after each node completes metadata collection, the slave nodes aggregate metadata to the master node through periodic metadata synchronization tasks.

[0016] As a preferred embodiment, the data processing blood relationship acquisition in the step SS1 comprises: firstly, data extraction, the dispersed data generated by each business system and each type of data source is comprehensively identified, and then the required data source is set and defined, the data source capable of operation is selected, and the definition of incremental extraction is determined; then, data conversion, through conversion measures, various data are converted from business models into analysis models, and basic tasks of selection, separation / merging, transformation and summarization are provided; after that, data loading, the data after conversion is loaded into the database through direct loading or database connection method; finally, the field-level blood relationship of the source data is extracted by extracting the data processing process information of the source data in the data conversion process, and the field-level blood relationship of the data after conversion and the source data is stored in the database.

[0017] As a preferred embodiment, the data blood relationship is owner-database-table-field.

[0018] As a preferred embodiment, the data blood relationship analysis in the step SS2 comprises: rich information of input tables and output tables is acquired through a HiveHook plug-in, is asynchronously sent to Kafka, is written to a graph database after analysis and processing, provides metadata system display and REST API service, and lands into a Hive relationship table, so that a user can query and visually display.

[0019] The present application has the following beneficial effects: (1) the data middle platform data blood relationship construction method is proposed, comprising four processes of metadata collection, data processing blood relationship acquisition, data access middleware object analysis and data storage access monitoring, the four processes are realized through a unified data analysis processing module, data analysis and blood relationship integration are realized, and finally stored in a graph database.

[0020] (2) The application proposes a metadata collection method for the whole life cycle of data, which collects metadata of databases, offline computing services, online computing services, middle components, computing tasks, organizes the metadata, and then writes the metadata into a database. Further, the application lists different metadata collection objects, collection contents, and collection methods.

[0021] (3) The application proposes a data processing bloodline acquisition method. The data processing first extracts relevant data in a data source, then implements further conversion according to the determined conversion requirements, and then loads these more standardized data into a data warehouse. In this process, the data processing bloodline acquisition collects the modification information of the data processing process to the data, thereby generating the blood relationship between new and old data.

[0022] (4) The application proposes a bloodline analysis and display method, including a hierarchical structure of structured data blood relationship stored in a database; a bloodline analysis process, which acquires rich information such as input tables and output tables through a HiveHook plug-in, asynchronously sends to Kafka, and after analysis and processing, writes data to a graph database, provides metadata system display and RESTAPI service, and lands into a Hive relationship table for user query and visual display; a blood relationship visualization display method, which displays rules and flow distribution at different positions on the graph, realizes tracking data traceability, data value evaluation, and data quality evaluation capabilities. (5) The application acquires data processing bloodline through dynamic and static operation methods, and uses different technical means for processing. Through distributed metadata collection, the static physical association of metadata and original data in the whole life cycle of data processing is acquired, and the dynamic logical association of data bloodline is acquired through data processing tasks, data storage access monitoring, data access middleware object analysis, etc. Data bloodline display can clearly reflect the data flow relationship path between table, file, field, task, and other entity objects, can intuitively display the data flow link of the whole system to the user, support impact analysis and data cold and hot degree analysis, and assist users in understanding complex blood relationship. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a topological schematic diagram of a preferred embodiment of a distributed data bloodline construction and display method of the application;

[0024] Figure 2 is a topological schematic diagram of distributed metadata collection of the application;

[0025] Figure 3 is a topological schematic diagram of data processing bloodline acquisition of the application;

[0026] Figure 4 is a hierarchical schematic diagram of data blood relationship of the application;

[0027] Figure 5 is a bloodline analysis process schematic diagram of the present application;

[0028] Figure 6 is a data bloodline visualization schematic diagram of the present application. DETAILED DESCRIPTION

[0029] The present application will be further described below in conjunction with the accompanying drawings. The following examples are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0030] Embodiment 1: as shown in the figure, the present application proposes a distributed data bloodline construction and display method, in the process of data platform running, through metadata acquisition, automatic acquisition and analysis of data and collection, data bloodline relationship construction is realized, at the same time, a relationship exploration technology is provided to discover the potential correlation between data resources, including the following steps: Figure 1

[0031] Step SS1: data bloodline construction step, including: respectively through distributed metadata acquisition, data processing bloodline acquisition, data access middleware object analysis and data storage access monitoring to generate bloodline relationship data;

[0032] Step SS2: the bloodline relationship data is analyzed and integrated by a data analysis processing module to generate data bloodline relationship;

[0033] Step SS3: the data bloodline relationship is stored into a graph database for user query and visual display.

[0034] Figure 1 The figure shows four processes for obtaining data bloodline used by the present application: distributed metadata acquisition, data processing bloodline acquisition, data access middleware object analysis, and data storage access monitoring. The four processes realize data analysis and bloodline relationship integration through a unified data analysis processing module, and are finally stored into a graph database.

[0035] As shown in the figure, the present application proposes a distributed data bloodline construction and display method, in the process of data platform running, through metadata acquisition, automatic acquisition and analysis of data and collection, data bloodline relationship construction is realized, at the same time, a relationship exploration technology is provided to discover the potential correlation between data resources, including the following steps: Figure 2 ​As shown, as a preferred embodiment, the distributed metadata collection in step SS1 includes: each node acquires metadata in the data life cycle as a collection object to create a data source, the collection object including a database, an offline computing service, an online computing service, a data middle platform component, and a computing task; configuring a collection task for the data source, then performing metadata collection, tagging the collected metadata, and then writing the metadata into a database and performing cross-node metadata aggregation and deduplication fusion. Different sources of metadata acquisition methods are also different. The data dictionary of structured data and the metadata information of unstructured data are collected. After the metadata collection is completed, it is stored in the database. Metadata is data used to describe data. In addition to those business data directly read and written by business logic, all other information / data required to maintain the operation of the entire system can be called metadata. For example, database Schema, Table, Column information, task blood relationship, user and script / task permission mapping relationship information, etc.

[0036] As a preferred embodiment, the distributed metadata collection in step SS1 further includes: for the database as a collection object, the collection method is through database access, and the collection content is table name, note, field list, primary key, foreign key, table size, row number, file number, partition number, and table / field upstream and downstream dependency relationship.

[0037] As a preferred embodiment, the distributed metadata collection in step SS1 further includes: for the offline computing service as a collection object, the collection method is through calling the computing service interface, and the collection content is Hive / RDS table metadata, including file status, file number, file size, and data update time trend data; for the online computing service as a collection object, the collection method is through accessing the service landing work order data to acquire the basic metadata information of the computing theme, and the collection content is the metadata of Flume / Hbase / Kafka component.

[0038] As a preferred embodiment, the distributed metadata collection in step SS1 further includes: for the data middle platform component as a collection object, the collection method is through synchronizing the component data to the database and offline extracting metadata, and the collection content is the blood data of BI report system, index library, and OneService service.

[0039] As a preferred embodiment, the distributed metadata collection in step SS1 further includes: taking the computing task as the collection object, and collecting the content by parsing the task input / output dependency configuration or parsing the lineage of tables / fields in the computing script, including the name of the offline / real-time computing task, the person in charge, the deadline alarm time, the script and task configuration information.

[0040] This invention describes the metadata collection content and collection methods for various metadata collection objects, as shown in Table 1.

[0041] Table 1 Data Collection Content and Methods

[0042]

[0043] As a preferred embodiment, the step of writing metadata into the database and performing cross-node metadata aggregation and deduplication fusion specifically includes: adopting a centralized metadata aggregation and fusion mode, that is, the middleware node acts as the master node and other data nodes act as slave nodes. After each node completes metadata collection, the slave nodes aggregate the metadata to the master node through periodic metadata synchronization tasks. After receiving the metadata, the master node first stores the original data in the database as the original data, and then performs deduplication and fusion operations on the metadata through metadata tags to obtain the metadata set of global data.

[0044] As a preferred embodiment, data processing involves extracting relevant data from the data source according to unified specifications, then performing further transformations according to predetermined requirements, and finally loading this standardized data into the data warehouse. Data lineage retrieval in data processing involves collecting information on data modifications made during the processing to establish the lineage relationship between old and new data. For example... Figure 3 As shown, the data processing lineage acquisition in step SS1 includes: First, data extraction, which involves comprehensively identifying the scattered data generated by various business systems and data sources, then setting and defining the required data sources, selecting data sources that can be operated on, and determining the definition of incremental extraction; next, data transformation, which transforms various types of data from business models into analytical models, providing basic tasks such as selection, separation / merging, transformation, and summarization; then, data loading, which loads the transformed data into the database through direct loading or database connection methods; finally, extracting the data processing information of the source data during the data transformation process to obtain the field-level lineage relationship of the source data, and storing the field-level lineage relationship between the transformed data and the source data in the target database.

[0045] As a preferred embodiment, such as Figure 4As shown, a hierarchy of structured data blood relationship stored in a database is described. There are subtle differences in the hierarchy of blood relationship for different types of data. Data flows between different owners, fuses, forms a relationship between owners through data, which is a kind of data blood relationship. The data blood relationship is owner-database-table-field.

[0046] As a preferred embodiment, as shown in Figure 5 As shown, the data blood relationship analysis in the step SS2 includes obtaining rich information of input table and output table through HiveHook plug-in, sending to Kafka asynchronously, writing data to graph database after parsing processing, providing metadata system display and REST API service, landing into Hive relationship table, and landing into Hive relationship table for user query and visual display.

[0047] As shown in Figure 6 As shown, the content to be displayed in the blood relationship visualization is described. The data blood visualization displays rules and flow distribution at different positions on the graph, thereby playing the following roles:

[0048] 1) Trace data source: when data is abnormal, help to trace the cause of the abnormality, and also help to trace the source of the data and the data processing process;

[0049] 2) Data value evaluation: to evaluate the data value, it is necessary to have a basis, and the data blood relationship can provide a basis for data value evaluation from the aspects of data audience, data update magnitude, data update frequency, etc.

[0050] 3) Data quality evaluation: on the data blood relationship graph, the standard list of data cleaning can be easily seen, reflecting the requirements for data quality.

[0051] The present application supports scientific and reasonable integration and integration of scattered and non-standardized, low-availability data, and forms unified, high-quality and high-credibility data assets through the construction of data warehouse. Through data blood collection and analysis, the present application can efficiently obtain complex blood relationship data and intuitively provide visual services for users on the blood relationship map, thereby greatly improving the efficiency and experience of user analysis and decision-making.

[0052] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0053] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0054] Finally, it should be noted that the above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalent alternatives without departing from the spirit and scope of the present application. Any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A method for constructing and displaying distributed data lineage, characterized in that, Includes the following steps: Step SS1: Data lineage construction step, including: generating lineage data through distributed metadata collection, data processing for lineage acquisition, data access middleware object parsing, and data storage access monitoring; the distributed metadata collection in step SS1 includes: each node acquiring metadata in the data lifecycle as collection objects to create data sources, the collection objects including databases, offline computing services, online computing services, data middleware components, and computing tasks; configuring collection tasks for the data sources, then executing metadata collection, tagging the collected metadata, then writing the metadata into the database and performing cross-node metadata aggregation and deduplication fusion; the step of writing metadata into the database and performing cross-node metadata aggregation and deduplication fusion specifically includes: adopting a centralized metadata aggregation and fusion mode, that is, the middleware node is the master node, and other data nodes are slave nodes. After each node completes metadata collection, the slave nodes aggregate the metadata to the master node through periodic metadata synchronization tasks; after receiving the metadata, the master node first stores the raw data in the database as raw data, and then performs deduplication and fusion operations on the metadata through metadata tags to obtain the metadata set of global data; Step SS2: The bloodline data is parsed and integrated using the data parsing and processing module to generate the bloodline data. Step SS3: Store the data lineage relationships in a graph database for users to query and visualize.

2. The method for constructing and displaying distributed data lineage according to claim 1, characterized in that, The distributed metadata collection in step SS1 further includes: using the database as the collection object, a collection method that accesses the database to collect information such as table name, remarks, field list, primary key, foreign key, table size, number of rows, number of files, number of partitions, and upstream and downstream dependencies of tables / fields.

3. The method for constructing and displaying distributed data lineage according to claim 1, characterized in that, The distributed metadata collection in step SS1 further includes: for offline computing services as the collection object, a collection method is used to call the computing service interface to collect Hive / RDS table metadata, which includes trend data on file status, number of files, file size, and data update time; for online computing services as the collection object, a collection method is used to access the work order data persisted to the service disk to obtain the basic metadata information of the computing topic, and the collected content is the metadata of Flume / Hbase / Kafka components.

4. The method for constructing and displaying distributed data lineage according to claim 1, characterized in that, The distributed metadata collection in step SS1 further includes: a collection method that takes the data platform components as the collection objects, synchronizes the component data to the database, and extracts metadata offline. The collection content includes the lineage data of the BI reporting system, indicator library, and OneService service.

5. The method for constructing and displaying distributed data lineage according to claim 1, characterized in that, The distributed metadata collection in step SS1 further includes: taking the computing task as the collection object, and collecting the content by parsing the task input / output dependency configuration or parsing the lineage of tables / fields in the computing script. The collected content includes the name of the offline / real-time computing task, the person in charge, the deadline alarm time, the script and task configuration information.

6. The method for constructing and displaying distributed data lineage according to claim 1, characterized in that, The data processing lineage acquisition in step SS1 includes: First, data extraction, which involves comprehensively identifying the scattered data generated by various business systems and data sources, then setting and defining the required data sources, selecting data sources that can be operated on, and determining the definition of incremental extraction; next, data transformation, which transforms various types of data from business models into analytical models, providing basic tasks such as selection, separation / merging, transformation, and summarization; then, data loading, which loads the transformed data into the database through direct loading or database connection methods; finally, extracting the data processing information of the source data during the data transformation process to obtain the field-level lineage relationship of the source data, and storing the field-level lineage relationship between the transformed data and the source data in the database.

7. The method for constructing and displaying distributed data lineage according to claim 1, characterized in that, The data lineage is owner-database-table-field.

8. The method for constructing and displaying distributed data lineage according to claim 1, characterized in that, The data lineage parsing in step SS2 includes obtaining rich information from the input and output tables through the HiveHook plugin, sending it asynchronously to Kafka, parsing it, and then writing the data to a graph database. This provides metadata system display and REST API services, and the data is stored in a Hive relationship table for users to query and visualize.

Images