Method and device for parsing column-level kinship of data sets

Abstract

The purpose of the various embodiments of the present disclosure is to provide a method, device, computer program product and computer program storage medium for parsing the column-level lineage relationship of a data set. Obtain data analysis code; obtain column metadata of the initial data set; parse the data analysis code to obtain the lineage chain between upstream and downstream variables; according to the lineage chain, parse the assignment expression of each variable step by step until the return expression is parsed; based on the parsed result of the return expression, generate the column metadata of the output data set and the lineage traceability information of each column therein. The various embodiments of the present disclosure can establish a correspondence between the columns of the initial data set and the columns of the output data set, thereby realizing column-level lineage tracking and association. Furthermore, it can simplify complex tasks such as locating downstream problems and column-level data permission access control, thereby improving the efficiency of data management and problem troubleshooting.

Images