A method and system for constructing and displaying end-to-end report lineage in the entire data chain

Abstract

The present invention relates to the field of report lineage technology, and discloses a data full-link end-to-end report lineage construction and display method and system, which collect multiple types of data tables; segment the cells of the data table to obtain a plurality of cells; divide the cells into a plurality of categories according to the data table type to which the cells belong, the data table name to which the cells belong, and the parent cell to which the cells belong; construct a lineage graph; identify the lineage path from each report cell to the source table cell in the lineage graph, and classify the lineage paths according to the types of report cells; identify the characteristic lineage paths and non-characteristic lineage paths of each type of lineage path, and review the non-characteristic lineage paths of the same type according to the characteristic lineage paths of the same type of lineage paths to obtain the deviation area of ​​the non-characteristic lineage paths; feed back the deviation area to the human to obtain a corrected lineage graph; and construct a characteristic lineage display platform according to the corrected lineage graph.

Description

Technical Field

[0001] The present invention relates to the field of report lineage, and more specifically, to a method and system for constructing and displaying a report lineage with a full data link and end-to-end. Background Art

[0002] A report is a tool that collects and processes the data stored in the current data source and dynamically presents the data in the data format desired by the user. By generating data reports, cumbersome and complex data can be collected and processed according to unified rules to provide regular and readable data.

[0003] The current report lineage system can only display the static dependency relationship between data and cannot deeply trace the source data flow. When errors occur in the lineage of a report, it is difficult to dynamically review and repair the lineage of the report. Summary of the invention

[0004] The present invention provides a method and system for constructing and displaying a report lineage of a full data link end-to-end, which solves the technical problems raised in the background technology.

[0005] In a first aspect, a method for constructing and displaying a report lineage of a full data link end-to-end includes:

[0006] Step 1: Collect multiple types of data tables, including source tables, intermediate tables, and reports;

[0007] Step 2, split the cells of the data table to obtain a plurality of cells; divide the cells into a plurality of categories according to the data table type to which the cells belong, the data table name to which the cells belong, and the parent cell to which the cells belong;

[0008] Step 3: Collect SQL statements in the data table extraction, conversion and loading process, and parse the SQL statements to obtain the dependency relationship between cells and construct mapping functions between cells with dependency relationships;

[0009] Step 4, construct a blood relationship graph, map each cell to a node of the blood relationship graph, map the dependency relationship of the cell to a directed edge of the blood relationship graph, and use the mapping function between the cells with dependency relationship as the weight of the directed edge;

[0010] The feature vector of a cell is constructed by the fields in the cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns of the data table to which it belongs;

[0011] Step 5, identifying the lineage path from each report cell to the source table cell in the lineage graph, and classifying the lineage path according to the type of the report cell;

[0012] Step 6, identifying characteristic bloodline paths and non-characteristic bloodline paths of each type of bloodline path, and reviewing the same type of non-characteristic bloodline paths based on the characteristic bloodline paths of the same type of bloodline paths to obtain the deviation area of ​​the non-characteristic bloodline paths;

[0013] Step 7, feed back the deviation area to the human to obtain a corrected blood relationship map; build a feature blood relationship display platform based on the corrected blood relationship map.

[0014] Furthermore, the cells of the data table are segmented, including:

[0015] Translate the table names of the data tables, unify the language, and obtain the same-language data tables;

[0016] Semantically match the table names of the same-text data tables to determine the data tables with the same semantics;

[0017] Classify data tables with the same semantics according to the types of data tables to obtain several homologous data tables;

[0018] Identify the field information in each cell in the homologous data table, filter the field information by the preset filtering conditions, and determine the parent cell in the homologous data table;

[0019] Classify the remaining cells in the source data table according to the parent cell in the source data table.

[0020] Furthermore, the SQL statement is parsed, including:

[0021] Read the text in the SQL statement one by one and identify the keywords in the SQL statement;

[0022] The data table type, the data table name, the parent cell in the data table, the number of rows and columns in the data table, and the fields in the cells of the data table are extracted as feature words;

[0023] Decompose the keywords and feature words obtained from the text in the SQL statement according to the rules of the standard syntax tree, and generate the corresponding abstract syntax tree;

[0024] Based on the abstract syntax tree, the dependency relationship and mapping function between cells are obtained.

[0025] Furthermore, directed edges and edge weights include:

[0026] A directed edge consists of one or more starting points and an end point;

[0027] Each starting point of a directed edge is connected to a dependent cell node, and the end point of a directed edge is connected to a dependent cell node;

[0028] The weight of the edge indicates that the field in the dependent cell is taken as input. Through the mapping function corresponding to the weight of the directed edge, the input field is calculated based on the mapping function to obtain the field in the dependent cell.

[0029] Furthermore, the characteristic bloodline paths and non-characteristic bloodline paths of each type of bloodline path are identified, including:

[0030] For each report cell, the bloodline path of each report cell is obtained by reversely identifying the directed edges;

[0031] Take the report cell node in each bloodline path as the root node and obtain the degree of the remaining non-report cell nodes;

[0032] Determine whether the lineage path of the report cell is a characteristic lineage path or a non-characteristic lineage path;

[0033] If the parent cell, data table type and data table name of the cells mapped by the nodes of each level of the lineage path in which the same type of report cells exceed the preset threshold ratio are the same, and the directed edges and edge weights between adjacent levels are the same, then the lineage path is judged to be a characteristic lineage path; otherwise, the lineage path is judged to be a non-characteristic lineage path.

[0034] Furthermore, the same type of non-characteristic bloodline paths are reviewed based on the characteristic bloodline paths of the same type of bloodline paths, including:

[0035] For the fields in the report cell, the directed edges and edge weights of the characteristic lineage paths of the same type of report cells are reversed hierarchically to obtain the fitted lineage paths;

[0036] The fitted lineage path is compared with the lineage path of the report cell to determine whether the field in the cell where the node is mapped in the lineage path of the report cell is abnormal or the cell is missing.

[0037] Furthermore, the deviation area is fed back to the human to obtain a corrected blood relationship map, including:

[0038] Feedback the abnormal field or missing cell in the node mapping in the lineage path of the report cell to the manual, and obtain the corrected lineage path by manually modifying the non-feature lineage path;

[0039] The corrected bloodline path is used to replace the corresponding non-characteristic bloodline path in the bloodline graph to obtain a corrected bloodline graph.

[0040] Furthermore, a characteristic blood relationship display platform is constructed, including:

[0041] Visual display platform, including front-end interface and back-end services;

[0042] In the front-end interface, load the report and display the report content;

[0043] In the backend service, based on the corrected bloodline graph, bloodline relationship query and data interaction interface are provided;

[0044] In the front-end interface, bind a click event listener to each non-parent cell in the report. When the user clicks a non-parent cell in the report, the event processing function is triggered:

[0045] Get the fields in the clicked cell, its parent cell, its data table type, its table name, and the number of rows and columns in the data table;

[0046] The front-end sends a request to the back-end service, carrying the fields in the clicked cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns in the data table to which it belongs;

[0047] In the corrected lineage graph, the backend service queries the lineage link based on the clicked cell and finds the lineage path from the cell to the source table cell;

[0048] For each bloodline path, extract the nodes, directed edges, node feature vectors and edge weights;

[0049] Serialize the queried bloodline path data into a data structure in JSON format;

[0050] The backend service sends the data structure in JSON format back to the frontend interface;

[0051] The front-end interface receives the bloodline path, parses it and builds a visual graph;

[0052] Use the visualization library to draw a lineage diagram to obtain the report lineage tree of the report cell.

[0053] In a second aspect, a data full-link end-to-end report lineage construction and display system is applied to the data full-link end-to-end report lineage construction and display method, including:

[0054] Data collection module collects multiple types of data tables, including source tables, intermediate tables and reports;

[0055] The data processing module divides the cells of the data table to obtain a plurality of cells; the cells are divided into a plurality of categories according to the data table type to which the cells belong, the data table name to which the cells belong and the parent cell to which the cells belong;

[0056] The data analysis module collects SQL statements in the data table extraction, conversion and loading process, and parses the SQL statements to obtain the dependencies between cells and build mapping functions between cells with dependencies;

[0057] The bloodline graph module constructs a bloodline graph, maps each cell to a node of the bloodline graph, maps the dependency relationship of the cell to a directed edge of the bloodline graph, and uses the mapping function between cells with dependency relationships as the weight of the directed edge;

[0058] The feature vector of a cell is constructed by the fields in the cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns of the data table to which it belongs;

[0059] The lineage path recognition module recognizes the lineage path from each report cell to the source table cell in the lineage graph, and classifies the lineage path according to the type of report cell;

[0060] The bloodline path analysis module identifies the characteristic bloodline paths and non-characteristic bloodline paths of each type of bloodline path, and reviews the non-characteristic bloodline paths of the same type based on the characteristic bloodline paths of the same type of bloodline paths to obtain the deviation area of ​​the non-characteristic bloodline paths;

[0061] The display platform construction module feeds back the deviation area to the human to obtain the corrected blood relationship map; and constructs the feature blood relationship display platform based on the corrected blood relationship map.

[0062] According to a third aspect, a computer device includes:

[0063] one or more processors;

[0064] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method for constructing and displaying a report lineage for the entire data link end-to-end.

[0065] The beneficial effect of the present invention is that by dividing the lineage path from the report cell to the source table into a characteristic path and a non-characteristic path, the deviation or anomaly in the data is automatically detected and identified, helping users to quickly locate the problem path. At the same time, the system provides a dynamic correction mechanism, and users can correct the non-characteristic path through manual feedback, generate a corrected lineage diagram, and intuitively view the data dependency on the characteristic lineage display platform to ensure the accuracy and transparency of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of a method for constructing and displaying a report lineage of a full data link end-to-end according to the present invention;

[0067] Figure 2 It is a module diagram of a data full-link end-to-end report lineage construction and display system of the present invention. DETAILED DESCRIPTION

[0068] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.

[0069] like Figure 1-2 As shown, a method for constructing and displaying a report lineage of a full data link end-to-end includes:

[0070] Step 1: Collect multiple types of data tables, including source tables, intermediate tables, and reports;

[0071] Step 2, split the cells of the data table to obtain a plurality of cells; divide the cells into a plurality of categories according to the data table type to which the cells belong, the data table name to which the cells belong, and the parent cell to which the cells belong;

[0072] Step 3: Collect SQL statements in the data table extraction, conversion and loading process, and parse the SQL statements to obtain the dependency relationship between cells and construct mapping functions between cells with dependency relationships;

[0073] Step 4, construct a blood relationship graph, map each cell to a node of the blood relationship graph, map the dependency relationship of the cell to a directed edge of the blood relationship graph, and use the mapping function between the cells with dependency relationship as the weight of the directed edge;

[0074] The feature vector of a cell is constructed by the fields in the cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns of the data table to which it belongs;

[0075] Step 5, identifying the lineage path from each report cell to the source table cell in the lineage graph, and classifying the lineage path according to the type of the report cell;

[0076] Step 6, identifying characteristic bloodline paths and non-characteristic bloodline paths of each type of bloodline path, and reviewing the same type of non-characteristic bloodline paths based on the characteristic bloodline paths of the same type of bloodline paths to obtain the deviation area of ​​the non-characteristic bloodline paths;

[0077] Step 7, feed back the deviation area to the human to obtain a corrected blood relationship map; build a feature blood relationship display platform based on the corrected blood relationship map.

[0078] In one embodiment of the present invention, segmenting cells of a data table includes:

[0079] Translate the table names of the data tables, unify the language, and obtain the same-language data tables;

[0080] Semantically match the table names of the same-text data tables to determine the data tables with the same semantics;

[0081] Classify data tables with the same semantics according to the types of data tables to obtain several homologous data tables;

[0082] Identify the field information in each cell in the homologous data table, filter the field information by the preset filtering conditions, and determine the parent cell in the homologous data table;

[0083] Classify the remaining cells in the source data table according to the parent cell in the source data table.

[0084] In one embodiment of the present invention, the following steps are included:

[0085] Step 201, translating the table names of the data tables by using translation software, converting the table names of all data tables into Chinese, and obtaining the same-language data tables;

[0086] Step 202: Map the table names of the same text data tables into word vectors using a word vector tool, and calculate the word vector similarity between the table names of the data tables and the table names of other data tables. The word vector similarity can be calculated using Euclidean distance or cosine similarity to obtain data tables with the same semantics.

[0087] Step 203: classify data tables with the same semantics according to the type of the data tables.

[0088] Through steps 201 to 203 , the data table of a third party can be compatible, thereby enhancing the robustness of the system.

[0089] In one embodiment of the present invention, the following steps are included:

[0090] Step 204, setting keywords to identify field information in each cell in the data table, if keywords in the field information are detected, then the cell is determined to be a parent cell;

[0091] Step 205, dividing the data table according to the frame lines in the table to obtain a number of non-parent cells;

[0092] Step 206, determining whether the non-parent cell is from the same source data table, and determining whether the parent cells of the non-parent cell are the same, if they are, classifying the non-parent cell into one category.

[0093] In one embodiment of the present invention, parsing the SQL statement includes:

[0094] Read the text in the SQL statement one by one and identify the keywords in the SQL statement;

[0095] The data table type, the data table name, the parent cell in the data table, the number of rows and columns in the data table, and the fields in the cells of the data table are extracted as feature words;

[0096] Decompose the keywords and feature words obtained from the text in the SQL statement according to the rules of the standard syntax tree, and generate the corresponding abstract syntax tree;

[0097] Based on the abstract syntax tree, the dependency relationship and mapping function between cells are obtained.

[0098] In one embodiment of the present invention, the following steps are included:

[0099] Step 301, reading the text in the SQL statement character by character, and identifying keywords based on preset SQL language rules, the keywords include but are not limited to SQL operators such as SELECT, FROM, WHERE, JOIN and their related control words;

[0100] Step 302, extract relevant information in the data table referenced by the SQL statement as feature words; the feature words include: the type of the data table (source table, intermediate table or report), the table name of the data table, the parent cell of each cell in the data table, the number of rows and columns in the data table, and the fields in each cell; the extraction of feature words provides basic data for the subsequent dependency relationship construction;

[0101] Step 303, combining the parsed keywords and feature words according to the rules of the standard syntax tree, gradually decomposing the SQL statement, and generating an abstract syntax tree corresponding to the SQL statement; the abstract syntax tree graphically represents the logical structure of the SQL statement, reflecting the hierarchical relationship between various operations;

[0102] Step 304, based on the generated abstract syntax tree, further analyze the dependencies between data tables to determine the data transfer and conversion process between cells; by identifying the starting point and end point of the data flow, generate mapping functions between cells to describe the mapping rules of data from source table cells to report cells; these mapping functions can include basic field mappings and complex conversion functions, such as aggregation operations or mathematical operations.

[0103] In one embodiment of the present invention, directed edges and edge weights include:

[0104] A directed edge consists of one or more starting points and an end point;

[0105] Each starting point of a directed edge is connected to a dependent cell node, and the end point of a directed edge is connected to a dependent cell node;

[0106] The weight of the edge indicates that the field in the dependent cell is taken as input. Through the mapping function corresponding to the weight of the directed edge, the input field is calculated based on the mapping function to obtain the field in the dependent cell.

[0107] For example, including nodes A, B, C, D and E; based on the dependency relationship and summation function obtained from the parsed SQL data, it is obtained that nodes B, C, D and E are dependent on A, by establishing a directed edge with four starting points and one end point between nodes A, B, C, D and E, and assigning a weighted summation function to this directed edge as the weight of the edge, the field value in the cell mapped by node A is obtained by adding the field values ​​in the cells mapped by nodes B, C, D and E.

[0108] In one embodiment of the present invention, identifying characteristic bloodline paths and non-characteristic bloodline paths of each type of bloodline path includes:

[0109] For each report cell, the bloodline path of each report cell is obtained by reversely identifying the directed edges;

[0110] Take the report cell node in each bloodline path as the root node and obtain the degree of the remaining non-report cell nodes;

[0111] Determine whether the lineage path of the report cell is a characteristic lineage path or a non-characteristic lineage path;

[0112] If the parent cell, data table type and data table name of the cells mapped by the nodes of each level of the lineage path in which the same type of report cells exceed the preset threshold ratio are the same, and the directed edges and edge weights between adjacent levels are the same, then the lineage path is judged to be a characteristic lineage path; otherwise, the lineage path is judged to be a non-characteristic lineage path.

[0113] In one embodiment of the present invention, it is assumed that there is a report system that generates reports from multiple data source tables and intermediate tables; now, we need to analyze a cell named total sales in the report and identify its lineage path; the cell depends on multiple source tables and intermediate tables;

[0114] Step 401, reverse identification of bloodline path;

[0115] Assume that the data of the sales total cell is generated step by step from multiple tables; by tracing the lineage path in reverse, the system finds that the data of this cell depends on the following tables:

[0116] Source table 1 (sales table): contains the original records of all sales;

[0117] Source table 2 (product table): contains product-related information;

[0118] The middle table (sales summary table): contains the total sales amount summarized by product category;

[0119] By tracing the directed edges in reverse, it is identified that the data of the total sales cell is aggregated from source table 1 and source table 2 through the intermediate table to the final report cell.

[0120] Step 402, calculating the degree of non-report cell nodes;

[0121] The system takes the total sales cell node as the root node and then calculates the node degrees in the remaining lineage paths. Assumptions:

[0122] Node A (sales data) in source table 1 has degree 2, which means it is connected to both the intermediate table and the report.

[0123] Node B (Sales Summary) in the middle table has degree 1, meaning it is connected only to the report cell.

[0124] Node C (Product Information) in Source Table 2 has degree 1, which means it is only connected to the intermediate table.

[0125] At this point, the system has a clear understanding of the connection between each node and the report.

[0126] Step 403, determining whether it is a characteristic bloodline path;

[0127] Next, the system compares the cells of the same type based on their lineage paths;

[0128] The system compares the following factors:

[0129] Parent grid information: whether nodes at the same level come from the same parent grid;

[0130] Data table type: whether the types of the source table and intermediate table are consistent;

[0131] Data table name: whether each table in the path is a data table of the same source;

[0132] Directed edges and edge weights: whether the weights of edges between adjacent nodes are consistent;

[0133] If more than the preset threshold ratio of bloodline paths in the same report cells meet the above conditions (for example, 75% of the paths are the same), then the path is identified as a characteristic bloodline path. Otherwise, it will be identified as a non-characteristic bloodline path;

[0134] After comparison, the system found that in the lineage path of the total sales cell, the table name, table type, and parent cell information of the source table and the intermediate table are exactly the same as those of the other 75% of similar report cells.

[0135] The directed edges and edge weights (such as data aggregation functions, etc.) between adjacent levels are also consistent.

[0136] Therefore, the system determines that the bloodline path of the cell is a characteristic bloodline path.

[0137] Examples of non-characteristic bloodline paths:

[0138] If the lineage path of another cell (such as sales discount) cannot be completely identified through reverse identification, the system will determine it as a non-characteristic lineage path.

[0139] In one embodiment of the present invention, the non-characteristic bloodline paths of the same type are reviewed according to the characteristic bloodline paths of the same type, including:

[0140] For the fields in the report cell, the directed edges and edge weights of the characteristic lineage paths of the same type of report cells are reversed hierarchically to obtain the fitted lineage paths;

[0141] The fitted lineage path is compared with the lineage path of the report cell to determine whether the field in the cell where the node is mapped in the lineage path of the report cell is abnormal or the cell is missing.

[0142] For example, we have a reporting system in which multiple cells display sales-related data information, including total sales. The fields of these cells are extracted and calculated from multiple source tables and intermediate tables. Now, we need to review the total sales cell to determine whether there are any anomalies or missing fields in its lineage path, and verify it based on the same type of feature lineage paths.

[0143] Step 501: Reverse fitting the bloodline path through the characteristic bloodline path;

[0144] Based on the feature lineage path that the system has identified for the same type of report cells; the feature lineage path starts from the report, is aggregated through the intermediate table, and reaches the source table cell; the feature lineage path includes the following dependencies:

[0145] Source Table 3 (Sales Record Table): records the detailed information of each sale, including the sales amount;

[0146] Source table 4 (product information table): contains product discount information;

[0147] Middle Table 3 (Sales Summary Table): summarizes the total sales and discounts for each product category;

[0148] Middle Table 4 (Sales Analysis Table): Sales data analyzed by region and time period;

[0149] Based on the characteristic lineage path, the system reverses through each directed edge and the edge weight to obtain the fitted lineage path of the total sales cell:

[0150] Step 502: Comparison of the fitted bloodline path and the bloodline path;

[0151] Next, the system compares the fitted lineage path of the total sales cell with the lineage path nodes of the total sales cell level by level;

[0152] By comparing layer by layer, the system detected two possible problems:

[0153] Field anomaly: In the fitted lineage path, the system expects the field in the product information table to be 5%, but in the lineage path of the total sales, the referenced field is 3%; therefore, the system identifies the node field as an abnormal field;

[0154] Missing fields: In fitting the lineage path, all sales data should be extracted from the sales record table and then summarized to generate the total sales amount. However, in the lineage path, the system finds that some records in the sales summary table do not contain all the sales data and there are missing fields.

[0155] Audit results: By comparing the fitted bloodline path and the bloodline path, the system concluded:

[0156] There are field anomalies in the lineage path of the total sales cell;

[0157] At the same time, the system detected that some fields were missing, resulting in incomplete aggregation of sales data.

[0158] In one embodiment of the present invention, the deviation area is fed back to the human to obtain a corrected blood relationship map, including:

[0159] Feedback the abnormal field or missing cell in the node mapping in the lineage path of the report cell to the manual, and obtain the corrected lineage path by manually modifying the non-feature lineage path;

[0160] The corrected bloodline path is used to replace the corresponding non-characteristic bloodline path in the bloodline graph to obtain a corrected bloodline graph.

[0161] For example, we have a report system. The total sales amount in a certain cell in the report shows the aggregated sales data. The lineage path of this cell is obtained by summarizing multiple source tables and intermediate tables. However, the system detects that there are field anomalies and field missing problems in the lineage path of this cell.

[0162] Step 601: Identify deviations and provide feedback to human;

[0163] During the automatic review process, the system found two major deviations in the lineage path of the sales total cell: field anomalies and field missing;

[0164] The system will feedback the deviation area to the manual review interface, prompting the user to manually correct the lineage path of the total sales cell; the deviation area feedback includes:

[0165] The location of the cell field anomaly and the field to which the cell fits;

[0166] Missing fields or unsummarized data table information prompts manual repair of missing data issues;

[0167] Step 602: manually correcting the non-characteristic bloodline path;

[0168] In the manual review interface, non-characteristic bloodline paths are manually corrected based on feedback from the deviation area;

[0169] Step 603: Generate a corrected bloodline path;

[0170] Through manual correction, the system obtains the corrected lineage path and marks the corrected path with an audit stamp, which includes the auditor and the audit time. The system will no longer audit the lineage path marked with the audit stamp.

[0171] Step 604: Update the bloodline graph

[0172] After the correction is completed, the system will replace the original non-characteristic bloodline path with the corrected bloodline path and update the bloodline graph.

[0173] In one embodiment of the present invention, a characteristic blood relationship display platform is constructed, including:

[0174] Visual display platform, including front-end interface and back-end services;

[0175] In the front-end interface, load the report and display the report content;

[0176] In the backend service, based on the corrected bloodline graph, bloodline relationship query and data interaction interface are provided;

[0177] In the front-end interface, bind a click event listener to each non-parent cell in the report. When the user clicks a non-parent cell in the report, the event processing function is triggered:

[0178] Get the fields in the clicked cell, its parent cell, its data table type, its table name, and the number of rows and columns in the data table;

[0179] The front-end sends a request to the back-end service, carrying the fields in the clicked cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns in the data table to which it belongs;

[0180] In the corrected lineage graph, the backend service queries the lineage link based on the clicked cell and finds the lineage path from the cell to the source table cell;

[0181] For each bloodline path, extract the nodes, directed edges, node feature vectors and edge weights;

[0182] Serialize the queried bloodline path data into a data structure in JSON format;

[0183] The backend service sends the data structure in JSON format back to the frontend interface;

[0184] The front-end interface receives the bloodline path, parses it and builds a visual graph;

[0185] Use the visualization library to draw a lineage diagram to obtain the report lineage tree of the report cell.

[0186] It should be noted that the visualization library can be Cytoscape.js.

[0187] A data full-link end-to-end report lineage construction and display system, applied to a data full-link end-to-end report lineage construction and display method, comprising:

[0188] Data collection module collects multiple types of data tables, including source tables, intermediate tables and reports;

[0189] The data processing module divides the cells of the data table to obtain a plurality of cells; the cells are divided into a plurality of categories according to the data table type to which the cells belong, the data table name to which the cells belong and the parent cell to which the cells belong;

[0190] The data analysis module collects SQL statements in the data table extraction, conversion and loading process, and parses the SQL statements to obtain the dependencies between cells and build mapping functions between cells with dependencies;

[0191] The bloodline graph module constructs a bloodline graph, maps each cell to a node of the bloodline graph, maps the dependency relationship of the cell to a directed edge of the bloodline graph, and uses the mapping function between cells with dependency relationships as the weight of the directed edge;

[0192] The feature vector of a cell is constructed by the fields in the cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns of the data table to which it belongs;

[0193] The lineage path recognition module recognizes the lineage path from each report cell to the source table cell in the lineage graph, and classifies the lineage path according to the type of report cell;

[0194] The bloodline path analysis module identifies the characteristic bloodline paths and non-characteristic bloodline paths of each type of bloodline path, and reviews the non-characteristic bloodline paths of the same type based on the characteristic bloodline paths of the same type of bloodline paths to obtain the deviation area of ​​the non-characteristic bloodline paths;

[0195] The display platform construction module feeds back the deviation area to the human to obtain the corrected blood relationship map; and constructs the feature blood relationship display platform based on the corrected blood relationship map.

[0196] A computer device comprising:

[0197] one or more processors;

[0198] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method for constructing and displaying a report lineage for the entire data link end-to-end.

[0199] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are within the protection of the present embodiment.

Claims

1. A method for constructing and displaying a report lineage in an end-to-end data link, characterized in that: include: Step 1: Collect multiple types of data tables, including source tables, intermediate tables, and reports; Step 2, split the cells of the data table to obtain a plurality of cells; divide the cells into a plurality of categories according to the data table type to which the cells belong, the data table name to which the cells belong, and the parent cell to which the cells belong; Step 3: Collect SQL statements in the data table extraction, conversion and loading process, and parse the SQL statements to obtain the dependency relationship between cells and construct mapping functions between cells with dependency relationships; Step 4, construct a blood relationship graph, map each cell to a node of the blood relationship graph, map the dependency relationship of the cell to a directed edge of the blood relationship graph, and use the mapping function between the cells with dependency relationship as the weight of the directed edge; The feature vector of a cell is constructed by the fields in the cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns of the data table to which it belongs; Step 5, identifying the lineage path from each report cell to the source table cell in the lineage graph, and classifying the lineage path according to the type of the report cell; Step 6, identifying characteristic bloodline paths and non-characteristic bloodline paths of each type of bloodline path, and reviewing the same type of non-characteristic bloodline paths based on the characteristic bloodline paths of the same type of bloodline paths to obtain the deviation area of ​​the non-characteristic bloodline paths; wherein a characteristic bloodline path represents a representative bloodline path obtained by clustering a type of bloodline paths, and a non-characteristic bloodline path represents a bloodline path in a type of bloodline path that is different from the corresponding characteristic bloodline path; Step 7, feed back the deviation area to the human to obtain a corrected blood relationship map; build a feature blood relationship display platform based on the corrected blood relationship map.

2. According to the method for constructing and displaying a report lineage of a full data link end-to-end according to claim 1, it is characterized in that: Split the cells of the data table, including: Translate the table names of the data tables, unify the language, and obtain the same-language data tables; Semantically match the table names of the same-text data tables to determine the data tables with the same semantics; Classify data tables with the same semantics according to the types of data tables to obtain several homologous data tables; Identify the field information in each cell in the homologous data table, filter the field information by the preset filtering conditions, and determine the parent cell in the homologous data table; Classify the remaining cells in the source data table according to the parent cell in the source data table.

3. According to the method for constructing and displaying a report lineage of a full data link end-to-end according to claim 1, it is characterized in that: Parse SQL statements, including: Read the text in the SQL statement one by one and identify the keywords in the SQL statement; The data table type, the data table name, the parent cell in the data table, the number of rows and columns in the data table, and the fields in the cells of the data table are extracted as feature words; Decompose the keywords and feature words obtained from the text in the SQL statement according to the rules of the standard syntax tree, and generate the corresponding abstract syntax tree; Based on the abstract syntax tree, the dependency relationship and mapping function between cells are obtained.

4. According to the method for constructing and displaying a report lineage of a full data link end-to-end according to claim 1, it is characterized in that: Directed edges and edge weights, including: A directed edge consists of one or more starting points and an end point; Each starting point of a directed edge is connected to a dependent cell node, and the end point of a directed edge is connected to a dependent cell node; The weight of the edge indicates that the field in the dependent cell is taken as input. Through the mapping function corresponding to the weight of the directed edge, the input field is calculated based on the mapping function to obtain the field in the dependent cell.

5. According to the method for constructing and displaying a report lineage of a full data link end-to-end according to claim 1, it is characterized in that: Identify the characteristic bloodline paths and non-characteristic bloodline paths for each type of bloodline path, including: For each report cell, the bloodline path of each report cell is obtained by reversely identifying the directed edges; Take the report cell node in each bloodline path as the root node and obtain the degree of the remaining non-report cell nodes; Determine whether the lineage path of the report cell is a characteristic lineage path or a non-characteristic lineage path; If the parent cell, data table type and data table name of the cells mapped by the nodes of each level of the lineage path in which the same type of report cells exceed the preset threshold ratio are the same, and the directed edges and edge weights between adjacent levels are the same, then the lineage path is judged to be a characteristic lineage path; otherwise, the lineage path is judged to be a non-characteristic lineage path.

6. According to the method for constructing and displaying a report lineage of a full data link end-to-end according to claim 1, it is characterized in that: Review the same type of non-characteristic bloodline paths based on the characteristic bloodline paths of the same type of bloodline paths, including: For the fields in the report cell, the directed edges and edge weights of the characteristic lineage paths of the same type of report cells are reversed hierarchically to obtain the fitted lineage paths; The fitted lineage path is compared with the lineage path of the report cell to determine if the field in the cell where the node is mapped in the lineage path of the report cell is abnormal or the cell is missing.

7. A method for constructing and displaying a report lineage of a full data link end-to-end according to claim 6, characterized in that: Feedback the deviation area to the human to obtain the corrected bloodline map, including: Feedback the abnormal field or missing cell in the node mapping in the lineage path of the report cell to the manual, and obtain the corrected lineage path by manually modifying the non-feature lineage path; The corrected bloodline path is used to replace the corresponding non-characteristic bloodline path in the bloodline graph to obtain a corrected bloodline graph.

8. According to claim 1, a method for constructing and displaying a report lineage of a full data link end-to-end is characterized in that: Build a feature lineage display platform, including: Visual display platform, including front-end interface and back-end services; In the front-end interface, load the report and display the report content; In the backend service, based on the corrected bloodline graph, bloodline relationship query and data interaction interface are provided; In the front-end interface, bind a click event listener to each non-parent cell in the report. When the user clicks a non-parent cell in the report, the event processing function is triggered: Get the fields in the clicked cell, its parent cell, its data table type, its table name, and the number of rows and columns in the data table; The front-end sends a request to the back-end service, carrying the fields in the clicked cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns in the data table to which it belongs; In the corrected lineage graph, the backend service queries the lineage link based on the clicked cell and finds the lineage path from the cell to the source table cell; For each bloodline path, extract the nodes, directed edges, node feature vectors and edge weights; Serialize the queried bloodline path data into a data structure in JSON format; The backend service sends the data structure in JSON format back to the frontend interface; The front-end interface receives the bloodline path, parses it and builds a visual graph; Use the visualization library to draw a lineage diagram to obtain the report lineage tree of the report cell.

9. A data full-link end-to-end report lineage construction and display system, applied to a data full-link end-to-end report lineage construction and display method according to any one of claims 1-8, characterized in that: include: Data collection module collects multiple types of data tables, including source tables, intermediate tables and reports; The data processing module divides the cells of the data table to obtain a plurality of cells; the cells are divided into a plurality of categories according to the data table type to which the cells belong, the data table name to which the cells belong and the parent cell to which the cells belong; The data analysis module collects SQL statements in the data table extraction, conversion and loading process, and parses the SQL statements to obtain the dependencies between cells and build mapping functions between cells with dependencies; The bloodline graph module constructs a bloodline graph, maps each cell to a node of the bloodline graph, maps the dependency relationship of the cell to a directed edge of the bloodline graph, and uses the mapping function between cells with dependency relationships as the weight of the directed edge; The feature vector of a cell is constructed by the fields in the cell, the parent cell to which it belongs, the type of data table to which it belongs, the name of the data table to which it belongs, and the number of rows and columns of the data table to which it belongs; The lineage path recognition module recognizes the lineage path from each report cell to the source table cell in the lineage graph, and classifies the lineage path according to the type of report cell; The bloodline path analysis module identifies the characteristic bloodline paths and non-characteristic bloodline paths of each type of bloodline path, and reviews the non-characteristic bloodline paths of the same type according to the characteristic bloodline paths of the same type of bloodline paths to obtain the deviation area of ​​the non-characteristic bloodline paths; wherein a characteristic bloodline path represents a representative bloodline path obtained by clustering a type of bloodline paths, and a non-characteristic bloodline path represents a bloodline path in a type of bloodline path that is different from the corresponding characteristic bloodline path; The display platform construction module feeds back the deviation area to the human to obtain the corrected blood relationship map; and constructs the feature blood relationship display platform based on the corrected blood relationship map.

10. A computer device, characterized in that: include: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a data full-link end-to-end report lineage construction and display method as described in any one of claims 1 to 8.

Images