Method and device for determining data consanguinity based on structural data

Abstract

The present invention provides a method and device for determining data consanguinity based on structural data. The method comprises: parsing a selection statement to obtain a source abstract syntax tree, and organizing the table information and the field information obtained by traversing the source abstract syntax tree into a source list layer by layer; obtaining the target abstract syntax treeby parsing the insertion statement, and organizing the table information and the field information obtained by traversing the target abstract syntax tree into the target list layer by layer; traversing the source manifest to obtain the source table information and traversing the target manifest to obtain the target table information to obtain the table granularity data kinship relationship; extracting the target field information for the target table from the target manifest, starting from the first layer of the source manifest, finding the source field in the source table with the same name as the target field information of the target table layer by layer, and when the source table to which the source field belongs no longer comes from the subquery, determining the corresponding source field to be the source field with blood relationship corresponding to the target field information. The scheme can parse the field-level granularity of data consanguinity based on structured data.

Description

technical field

[0001] The present disclosure relates to the technical field of data processing, and in particular to a data lineage determination method, device, electronic device and computer-readable storage medium based on structural data. Background technique

[0002] There is currently no unified definition of data lineage, which can be roughly understood as the link generated by data. Data lineage describes which tables a table depends on, and how the fields in the table are generated, and further even describes which fields these fields depend on in other tables. The upstream and downstream dependencies of data production can be known through data blood relationship. Data lineage is mainly used in the field of big data. As background knowledge, let's first understand the entire production process of big data. The overall production process of big data is generally divided into four layers: data source, production, warehouse, and data application. The data source is...

Examples

Embodiment Construction

[0078] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily realize them. Also, for clarity, parts not related to describing the exemplary embodiments are omitted in the drawings.

[0079] In the present disclosure, it should be understood that terms such as "comprising" or "having" are intended to indicate the presence of features, numbers, steps, acts, components, parts or combinations thereof disclosed in the specification, and are not intended to exclude one or a plurality of other features, numbers, steps, acts, parts, parts or combinations thereof exist or are added.

[0080] In addition, it should be noted that, in the case of no conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings...

technical field

[0001] The present disclosure relates to the technical field of data processing, and in particular to a data lineage determination method, device, electronic device and computer-readable storage medium based on structural data. Background technique

[0002] There is currently no unified definition of data lineage, which can be roughly understood as the link generated by data. Data lineage describes which tables a table depends on, and how the fields in the table are generated, and further even describes which fields these fields depend on in other tables. The upstream and downstream dependencies of data production can be known through data blood relationship. Data lineage is mainly used in the field of big data. As background knowledge, let's first understand the entire production process of big data. The overall production process of big data is generally divided into four layers: data source, production, warehouse, and data application. The data source is...

Examples

Embodiment Construction

[0078] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily realize them. Also, for clarity, parts not related to describing the exemplary embodiments are omitted in the drawings.

[0079] In the present disclosure, it should be understood that terms such as "comprising" or "having" are intended to indicate the presence of features, numbers, steps, acts, components, parts or combinations thereof disclosed in the specification, and are not intended to exclude one or a plurality of other features, numbers, steps, acts, parts, parts or combinations thereof exist or are added.

[0080] In addition, it should be noted that, in the case of no conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings...

Images