Data blood relationship analysis method, device, equipment and system and readable storage medium

An analysis method and data system technology, applied in the field of data analysis, can solve problems such as unfriendly scalability and inability to adapt to different types of data systems, and achieve the effect of improving scalability and reducing complexity

Active Publication Date: 2019-04-05
WEBANK (CHINA)
View PDF4 Cites 38 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The main purpose of the present invention is to provide a data lineage analysis method, device, equipment, system and readable storage medium...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data blood relationship analysis method, device, equipment and system and readable storage medium
  • Data blood relationship analysis method, device, equipment and system and readable storage medium
  • Data blood relationship analysis method, device, equipment and system and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0036] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0037] Such as figure 1 as shown, figure 1 It is a schematic structural diagram of the hardware operating environment involved in the solution of the embodiment of the present invention.

[0038] It should be noted, figure 1 That is, it is a structural schematic diagram of the hardware operating environment of the data lineage analysis device. The data lineage analysis device in this embodiment of the present invention may be a terminal device such as a PC or a portable computer.

[0039] Such as figure 1 As shown, the data lineage analysis device may include: a processor 1001 , such as a CPU, a network interface 1004 , a memory 1005 , and a communication bus 1002 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. Optionally, the network interface 1004...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data blood relationship analysis method, device, equipment and system and a readable storage medium, and the method comprises the steps: obtaining blood relationship data corresponding to an SQL statement through Hook when the data system executes the SQL statement; Determining a data table type of a data table where the blood relationship data is located through a streaming analysis system, and constructing a directed graph corresponding to the blood relationship data according to the data table type and the blood relationship data; And storing the directed graph inan HBase, and storing the blood relationship data in an HDFS. According to the invention, blood relationship data of different types of data systems are acquired through Hook; the data blood relationship analysis method is adaptive to different types of data systems; the directed graph corresponding to the blood relationship is obtained by analyzing the data table type and the association information of the data table where the blood relationship data is located, it is avoided that the blood relationship of the data is obtained through SQL script analysis, the complexity of analyzing the bloodrelationship of the data is reduced, and the expansibility of the data blood relationship analysis method is improved.

Description

technical field [0001] The present invention relates to the technical field of data analysis, in particular to a data lineage analysis method, device, equipment, system and readable storage medium. Background technique [0002] Data lineage analysis is the core function of metadata management and data governance tools. By establishing lineage relationships between data, it is possible to analyze whether upstream data changes affect downstream associated data; if technical metadata and business metadata are established on metadata Through blood relationship, you can analyze the data flow between different business products, and analyze the business relationship between different products; through the analysis of data blood relationship, you can better understand and use data. At present, MetaOne of Dianthus already supports data lineage analysis. MetaOne constructs lineage links of data by analyzing SQL (Structured Query Language, Structured Query Language) scripts, and decom...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/182G06F16/242
Inventor 周可邸帅汪亚男兰冲
Owner WEBANK (CHINA)
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap