Unlock instant, AI-driven research and patent intelligence for your innovation.

Data blood relationship analysis method, terminal equipment and storage medium

An analysis method and data technology, applied in the field of big data analysis, can solve problems such as the inability to meet the requirements of cross-level data lineage analysis, and achieve the effect of finer granularity

Pending Publication Date: 2022-04-08
XIAMEN MEIYA PICO INFORMATION
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Traditional data lineage analysis considers the three dimensions of table level, data item level, and data row level independently for analysis, and does not combine the three dimensions. It cannot meet the requirements of cross-level data lineage analysis and has certain advantages for data traceability. limitations

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data blood relationship analysis method, terminal equipment and storage medium
  • Data blood relationship analysis method, terminal equipment and storage medium
  • Data blood relationship analysis method, terminal equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0030] The embodiment of the present invention provides a data lineage analysis method, such as figure 1 As shown, the method includes the following steps:

[0031] S1: Extract all data tables that pass through the data transfer process, and set unique identifiers in each data table based on the three analysis dimensions of data table level, data item level, and data row level.

[0032] In the flow of data transfer in this embodiment, four data tables are sequentially passed through according to the data processing method, which are source table, A resource table, B resource table and C resource table. in:

[0033] The source table is the source data that needs to be accessed. After data detection of the source table, the A resource table is generated according to the obtained detection log. In this embodiment, data probing includes probing the attribute, format and storage location of the data, and the content obtained through probing generates a corresponding probing log a...

Embodiment 2

[0051] The present invention also provides a data lineage analysis terminal device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and the present invention is realized when the processor executes the computer program Steps in the above method embodiment of Embodiment 1.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a data consanguinity analysis method, terminal equipment and a storage medium, and the method comprises the steps: S1, extracting all data tables passed in a data circulation process, and respectively setting a unique identifier in each data table based on three analysis dimensions of a data table level, a data item level and a data row level; s2, according to the upstream and downstream unique identifier of each piece of data in the flow process of the data, constructing a mapping table representing the flow process of the data; and S3, according to the analysis dimension corresponding to the to-be-analyzed data, searching the corresponding upstream and downstream unique identifiers of the to-be-analyzed data under the analysis dimension from the mapping table, and constructing a directed graph representing the data flow process of the to-be-analyzed data under the analysis dimension based on the extracted upstream and downstream unique identifiers. According to the method, data traceability in three dimensions of the data table, the data item and the data row is realized, the granularity of data traceability is refined, the data processing flow can be tracked, and convenience is provided for quickly positioning problem nodes and querying changed data points.

Description

technical field [0001] The present invention relates to the field of big data analysis, in particular to a data lineage analysis method, a terminal device and a storage medium. Background technique [0002] With the development of big data, the total amount of aggregated data resources is increasing day by day. The quality of data from different sources is uneven, and the impact on the results of analysis and processing is also different. When the data is abnormal, it is necessary to trace the cause of the abnormality and control the risk at an appropriate level. The blood relationship of the data reflects the ins and outs of the data, which can help us track the source of the data and track the data processing process. Therefore, how to quickly and effectively realize data blood relationship analysis is particularly important. [0003] Traditional data lineage analysis considers the three dimensions of table level, data item level, and data row level independently, and d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2455G06F16/2458G06F21/60G06F16/22
Inventor 蔡晓梅黄荣昌吴文吴鸿伟鄢小征
Owner XIAMEN MEIYA PICO INFORMATION