A Data Lineage Management Method in a Tax Scenario

By employing a full-chain data lineage management approach, the field-level lineage relationships of tax big data are automatically analyzed and visualized, solving the problem of data traceability difficulties and achieving efficient data governance and quality control.

CN123115268APending Publication Date: 2026-11-17CHINA NAT SOFTWARE & SERVICE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611610785.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-19
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In the context of big data in taxation, data sources are diverse, tracing is difficult, and manually locating upstream and downstream data is time-consuming and cumbersome. Traditional data lineage analysis methods cannot meet the needs of data governance and quality control.

Method used

By adopting a full-link data lineage management approach, through source data aggregation and pre-cleaning, data processing statement parsing, abstract syntax tree deconstruction, field-level lineage relationship extraction and visualization, and combined with relational database and graph database storage, automated and intelligent data lineage parsing and full-link visualization are achieved.

Benefits of technology

It implements field-level lineage mapping, penetrates temporary tables and subqueries, improves the efficiency of lineage extraction, ensures the integrity and accuracy of the link, supports high-performance impact analysis and link tracing, and adapts to the differences in data interfaces of different provinces, cities and external units.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115268A_ABST
    Figure CN123115268A_ABST
Patent Text Reader

Abstract

This invention relates to a method for end-to-end data lineage management in a tax context, belonging to the field of tax data governance. It includes data aggregation and pre-cleaning from source tax systems, establishing a unified data dictionary model; converting data processing statements into abstract syntax trees for deconstruction; eliminating semantic ambiguity using default execution libraries, table field mappings, alias mappings, and intermediate lineage relationships; achieving field-level lineage penetration for temporary tables and subqueries; storing lineage relationships in relational databases and graph databases respectively; and providing visualized displays for upstream and downstream tracing, link queries, and impact analysis. This invention solves the problems of multi-source heterogeneity, long processing paths, and difficulties in manual tracing of tax data, achieving automated extraction and end-to-end visualization of field-level lineage, effectively supporting tax data governance and quality control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of tax data governance, and in particular to a method for managing the entire data lineage in a tax context. Background Technology

[0002] With the deepening of the Golden Tax Project and the construction of smart taxation, my country's tax management system is facing unprecedented data management challenges. The tax data environment exhibits the following significant characteristics: (1) Data source diversity: The data sources are extremely wide, covering the internal tax system data of all provinces (autonomous regions, municipalities directly under the central government, and cities with independent planning status) across the country, including core business data such as tax declaration, collection, management, and inspection. At the same time, it also gathers exchange data from multiple external units such as industry and commerce, social security, customs, and banks. The diversity of data sources makes data traceability extremely difficult.

[0003] (2) Data heterogeneity: There are significant differences in data format, encoding standards and interface specifications among the data sources. The data interfaces and specifications provided by different provincial and municipal tax bureaus and external commissioning units are different. The data needs to undergo a lot of complex cleaning, transformation and loading (ETL) processing during the integration process, and the processing logic is highly complex.

[0004] (3) Long processing path: During the data processing process, the data needs to go through multiple stages such as storage, transformation, calculation and integration. Different business scopes correspond to different processing logics, which makes the data processing link too long, resulting in the complexity and depth of data flow between multiple tasks. Manually sorting out the data lineage is time-consuming and prone to errors.

[0005] (4) Poor visualization effect: Traditional data lineage analysis results are usually presented in tabular form, with a single display method and lack of intuitive graphical expression. It is difficult to clearly present the hierarchical relationship and flow path between data links, which is not conducive to business personnel's understanding and use.

[0006] Given the diverse data sources and highly complex data processing logic within the tax system, traditional data lineage analysis methods based on regular expressions or simple keyword matching are no longer sufficient to meet the basic needs of data governance, data quality control, and impact analysis. Therefore, there is an urgent need for a method capable of providing field-level, automated, refined, and intelligent data lineage analysis and end-to-end visualization for the complex processing scenarios of tax big data. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a data lineage management method in the tax scenario. This method addresses the problems of wide data sources, difficulty in tracing the source, excessive time consumption for manual upstream and downstream location, and cumbersome process in the current tax big data aggregation environment. It enables the automated extraction, intelligent analysis, and visualization of field-level lineage relationships in tax data.

[0008] This invention provides a method for end-to-end data lineage management in a tax context. The method includes: Step S1, Data Aggregation and Pre-cleaning of Source Tax System: Aggregate raw data from multiple tax-related data sources, standardize, process, check data quality, and pre-clean the aggregated raw data, and establish a unified data dictionary model; Step S2, Algorithm analysis of processing statements for tax aggregated data: Convert the data processing statements generated in step S1 into an abstract syntax tree, deconstruct the data processing statements based on the abstract syntax tree, identify data flow relationships, and extract field-level data lineage relationships; Step S3: Extract and process relational data for separate database storage: Store the parsed field-level data and lineage relationship data into a relational database and a graph database respectively; Step S4, Visualization of Data Lineage Analysis: Based on stored lineage data, it provides visualization functions for tracing upstream and downstream relationships, querying links to specified nodes, and analyzing impacts. Step S2 includes: defining and using a default execution library, table field mapping relationships, alias-table name mapping relationships, and intermediate data lineage relationships to eliminate semantic ambiguity of identifiers and achieve penetration of intermediate lineage relationships; deconstructing data processing statements into temporary table definition statements, data writing statements, and data query statements; extracting target-end information from data writing statements and extracting source-end information of query result data columns from data query statements; establishing field-level mapping relationships between the data columns of the query results of data query statements and the target-end table fields of data writing statements in sequence; if the source-end information involves temporary table or subquery fields, tracing back layer by layer using the intermediate data lineage relationships until the real physical source-end fields are obtained.

[0009] Optionally, the default execution library is used to eliminate semantic ambiguity and obtain a complete table name identifier when the parsed table name does not explicitly declare the database to which it belongs, in conjunction with the statement execution environment. The table field mapping relationship covers three types: temporary tables, external tables, and subquery output data columns. It is used to eliminate semantic ambiguity and obtain complete field identification when wildcards are used to refer to all fields or when the field name is not explicitly declared to belong to the table name. The alias-table name mapping relationship is used to restore the alias to the complete table name identifier when the parsed field name is declared to belong to the table through the table alias; The intermediate data lineage relationship is used to penetrate internal lineage information when using temporary tables or subqueries in data processing statements, so as to obtain the complete data lineage between the target end and the real physical source end.

[0010] Optionally, the deconstruction of the data processing statement includes: The data processing statements are deconstructed into temporary table definition statements starting with WITH, data write statements starting with INSERT or CREATE, and data query statements starting with SELECT.

[0011] Optionally, extracting target information from the data write statement includes: Extract target information from the data write statement, which includes the target table and target fields; Extract the target table and use the default execution library to eliminate semantic ambiguity and obtain the complete target table name; If the target field is explicitly declared, it will be extracted using comma-separated parameters; If not explicitly declared, the target field will be the field information from the target table in the table field mapping relationship. Extract the partition field from the partition information and append it as a special target field to the above target field; This ensures that the number of target fields corresponds sequentially to the number of data columns in the query result.

[0012] Optionally, the step of extracting the source information of the query result data columns from the data query statement includes: The data query statements are divided using the concatenation keyword UNION or UNION ALL as boundaries to obtain each SELECT query statement; For each SELECT query statement, it is broken down into SELECT syntax block, FROM syntax block, WHERE syntax block, GROUP syntax block, HAVING syntax block and ORDER syntax block, with the starting keyword of each syntax block as the boundary. The tables and fields involved in each syntax block are parsed in turn to obtain direct source information, column-domain condition source information, and global condition source information; The direct source information and column-domain condition source information are mapped one-to-one with the query result data columns, and all global condition source information is assigned to each result data column to complete the integration of the parsed results.

[0013] Optionally, step S2 may further include destructuring the field expression; The field expression refers to an expression that takes fields or constants as input parameters, is processed by functions, judgments, and calculations, and returns data in the form of constant values ​​or fields. The parsing of the field expression includes: Field expressions are categorized into three types: basic, composite, and subquery, with a total of 14 configurations. These 14 configurations include constant, field reference, wildcard, comparison, range judgment, special judgment, IF, CASE_WHEN, function, calculation, window, other expression, correlated subquery, and uncorrelated subquery. Identify the configuration of field expressions using an abstract syntax tree; If the field expression is a composite type, the field expression is decomposed into subfield expressions using a depth-first traversal method. The subfield expressions are then recursively parsed until the structure of the subfield expression is a basic type or a subquery type. The parsing result of a composite field expression is the union of the parsing results of all its subfield expressions. The parsing results are expanded using wildcards, disambiguated using field identifiers, and mapped to table aliases to obtain all source table fields used in the field expression. These source table fields are divided into two categories: source fields and condition fields.

[0014] Optionally, the classification rules for the parsing results of the composite field expression include: If the field expression is a comparison operation, range judgment, or special judgment type, the parsing result is the condition field; If the field expression is of type IF, the parsing result of the first parameter is the condition field, and the rest are the source fields; If the field expression is of type CASE_WHEN, the parsing result of the parameters after the keywords THEN and ELSE is the source field, and the rest are the condition fields; If the field expression is a window type, the parsing result of the parameters in the window function is the source field, and the rest are condition fields; If the field expression is a function, calculation, or other expression type, the parsed result is the source field; When merging the parsing results of subfield expressions, if the subfield expression is located in the source field position of the field expression, then the source field of the merged subfield expression becomes the source field of the field expression, and the condition field of the subfield expression becomes the condition field of the field expression; if the subfield expression is located in the condition field position of the field expression, then all fields of the merged subfield expression become the source field of the field expression.

[0015] Optionally, step S2 may further include parsing the temporary table definition statement; The temporary table definition statement contains the definition of multiple temporary tables. A temporary table contains three main parts: the temporary table name, the temporary table fields, and the temporary table data query statement. The parsing of the temporary table definition statement includes: Extract each temporary table from the temporary table definition statement and use the temporary table name as the target table name; If a temporary table field is explicitly declared, extract the target field name using comma separators; If not explicitly declared, the column names of the parsed query results will be used as the field names of the temporary table; Use the data query statement parsing method to parse the data query statement from the temporary table to obtain the column names and source information of the query results; Update the table and field mapping relationship using the temporary table name and temporary table field names; The table fields of the temporary table are mapped one-to-one with the source information to obtain lineage information and update the lineage relationship in the intermediate data.

[0016] Optionally, step S3 includes: Store a lineage relationship table in a relational database. Each record contains SCHEMA, TABLE, and FIELD information for the source field, SCHEMA, TABLE, and FIELD information for the target field, and relationship attribute information. In a graph database, each table field is treated as a node, and the lineage relationship between fields is treated as an edge. Node attributes include SCHEMA, TABLE, and FIELD, while edge attributes include relationship type and processing script identifier.

[0017] Optionally, step S4 includes: Upstream and downstream relationship tracing: Traverse upstream and downstream relationships at a specified depth and visualize them in the form of a lineage graph. In the lineage graph, nodes represent fields, edges represent lineage relationships, and the display style of the edges distinguishes between direct lineage and conditional lineage. Link query: Supports link queries between specified source and target fields, displaying the complete processing path and intermediate data items between the two fields; Impact Analysis: Input a specified field, query and list all downstream nodes of that field, and display the downstream table name, field name, relationship type and processing script source in tabular form.

[0018] In summary, the present invention has at least one of the following beneficial technical effects: 1. Automation and Intelligence: Through abstract syntax tree parsing and predefined metadata rules, it realizes the automated deconstruction of complex SQL processing statements, eliminating the need for manual line-by-line analysis, greatly improving the efficiency of lineage extraction and reducing labor costs.

[0019] 2. Field-level precision and penetration capability: Supports field-level lineage mapping and can penetrate temporary tables (CTEs) and subqueries, tracing the lineage layer by layer through intermediate data lineage to the real physical source field. This solves the problem that traditional methods cannot handle multi-level nested processing and ensures the integrity and accuracy of the lineage chain.

[0020] 3. Dual-database storage and efficient query: Combining the advantages of relational databases (storing structured lineage records) and graph databases (storing nodes and edges), it supports both transactional queries and deep path traversal, providing high-performance data support for subsequent impact analysis and link tracing.

[0021] 4. Visualized end-to-end display: Provides functions such as upstream and downstream relationship graphs, link query between specified nodes, and impact analysis. It clearly displays the data flow path and dependencies in a graphical way, making it easy for business personnel to intuitively understand the data processing process and effectively supporting data governance, quality control, and change impact assessment.

[0022] 5. Strong adaptability to tax scenarios: Taking into full account the characteristics of tax data such as multiple sources, heterogeneity, and long processing paths, it adapts to the differences in data interfaces of different provinces, cities and external units through standardized data dictionaries, default execution libraries, alias mapping and other mechanisms, and has good scalability and universality. Attached Figure Description

[0023] Figure 1 This is a flowchart of a data lineage management method for a tax scenario according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram of the data lineage of data processing statements in an embodiment of this application.

[0025] Figure 3 This is a schematic diagram of the data processing statement deconstruction in an embodiment of this application.

[0026] Figure 4 This is a schematic diagram of the data writing statement deconstruction in an embodiment of this application.

[0027] Figure 5 This is a schematic diagram illustrating the deconstruction of a data query statement in an embodiment of this application.

[0028] Figure 6 This is a schematic diagram of the deconstruction of the FROM syntax block in an embodiment of this application.

[0029] Figure 7 This is a schematic diagram showing the correspondence between the query results of the SELECT query statement and the source information in an embodiment of this application.

[0030] Figure 8 This is a schematic diagram illustrating the relationship between the query results of the SELECT query statement and the query results of the data query statement in an embodiment of this application.

[0031] Figure 9 This is a schematic diagram illustrating the classification of field expression types in an embodiment of this application.

[0032] Figure 10 This is a schematic diagram of the destructuring of the temporary table definition statement in an embodiment of this application.

[0033] Figure 11 This is a schematic diagram of database storage in an embodiment of this application. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to the accompanying drawings.

[0035] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of the present invention.

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] The embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.

[0038] like Figure 1 As shown, the present invention provides a flowchart of a data lineage management method for a tax scenario, as detailed below: Step S1: Data aggregation and pre-cleaning of the source tax system.

[0039] Raw data is aggregated from multiple tax-related data sources. The aggregated raw data undergoes standardization processing, data quality checks, and pre-cleaning. A unified data dictionary model is then established, as detailed below: In the context of big data in taxation, data aggregation is the foundation of lineage analysis. This step aggregates raw data from the internal tax systems (such as tax declaration, collection, and management) and external entities (such as industry and commerce, social security, customs, and banks) of 36 provinces (autonomous regions, municipalities, and cities with independent planning status) across the country.

[0040] After aggregation, the aggregated raw data undergoes standardization of the source physical tables, data processing, data quality checks, and pre-cleaning.

[0041] The structural information (table name, field name, field type) of the source physical table is stored in the metadata database according to a unified data dictionary model; at the same time, the ETL scripts, stored procedures, and SQL statements generated during data processing are summarized and stored to provide input for subsequent lineage analysis.

[0042] Step S2: Algorithm analysis of processing statements for tax data aggregation.

[0043] The data processing statements are converted into an abstract syntax tree (AST). Based on the AST, the data processing statements are deconstructed to identify data flow relationships and extract field-level data lineage relationships, as detailed below: like Figure 2 As shown, the data processing statement queries the preset database and returns the query results in the form of a two-dimensional table. The query results are then written to specified table fields, with each data column of the query result corresponding one-to-one with the fields of the table being written to. The SQL statement that queries the data is the data query statement, and the SQL statement that specifies the fields to be written to the target table is the data write statement. In the data processing statement, data flows from the query results to the table being written to; the underlying data flow relationship is known as data lineage.

[0044] Step S21: Metadata Management.

[0045] The following metadata is defined and used during the parsing process to eliminate semantic ambiguity of identifiers and penetrate the lineage relationships between data processing statements: Default execution library (PARENT): This refers to the name of the database to which the table belongs (DATABASE / SCHEMA / USER / PROJECT). The default value is the execution library used when data processing statements are executed. When the resolved table name does not explicitly declare its database (TABLE_NAME), the default execution library (PARENT.TABLE_NAME) is used to eliminate semantic ambiguity.

[0046] Table field mapping relationships (table structure information, i.e., data dictionary): covering three types: temporary tables (CTE), external tables (physical tables, views), and subquery output data columns. Temporary tables and subqueries are dynamically updated during parsing execution. When using wildcards ( ) refers to all fields, or when the field name is not explicitly declared to belong to the table name (COLUMN_NAME), the table field mapping relationship is used to eliminate semantic ambiguity (PARENT.TABLE_NAME.COLUMN_NAME).

[0047] Alias ​​to table mapping: The mapping between table aliases and table names / temporary table names / subqueries is dynamically updated during the parsing process. When declaring the table to which a parsed field name belongs using a table alias (ALIAS.COLUMN_NAME), the alias-table name mapping is used to eliminate semantic ambiguity (PARENT.TABLE_NAME.COLUMN_NAME).

[0048] Intermediate data lineage: The intermediate data lineage generated during the parsing of temporary tables and subqueries is the internal lineage information of the entire data processing statement, and it is dynamically updated during the parsing process. When temporary tables or subqueries are used in the data processing statement, and the data lineage information includes the temporary tables and subqueries, the intermediate data lineage is used to penetrate the internal lineage information and obtain the true data lineage.

[0049] Step S22: Determine the structure of the data processing statements.

[0050] like Figure 3 As shown, by summarizing and analyzing the syntax structure of data processing statements according to the SQL standard, we can obtain the structure of data processing statements, which includes three parts: temporary table definition statement (with WITH as the starting keyword), data writing statement (with INSERT or CREATE as the starting keyword), and data query statement (with SELECT as the starting keyword).

[0051] The core of data lineage parsing for data processing statements is to use the abstract syntax tree (AST) of the data processing statement to identify the type and content of each lexical unit (the smallest semantic unit of an SQL statement, i.e., the TOKEN of the abstract syntax tree) in the statement. It parses the source of the data from the data query statement (which table fields the data is queried from, i.e., the source end, the upstream of the data lineage) and the destination of the data from the data write statement (which table fields the data is written to, i.e., the destination end, the downstream of the data lineage). It also aligns the source end and the destination end at the field granularity to form a field-level data lineage.

[0052] The temporary table definition statement is an optional part of the data processing statement. It uses a set of data query statements to query data from the fields of the physical table in advance and store it in a virtual temporary table. It establishes an intermediate data lineage relationship from the fields of the physical table to the fields of the temporary table. The data processing statement uses the field names of the temporary table to access the data of the physical table behind the temporary table. With the help of the intermediate data lineage relationship, it is possible to penetrate the temporary table and obtain the data lineage relationship between the target end and the real source end.

[0053] Step S23: Structure and parsing of data writing statements.

[0054] like Figure 4As shown, the purpose of parsing the data write statement is to obtain the target end information, that is, the downstream of the data lineage. The target end information consists of two parts: the target table and the target field.

[0055] Step S231: The structure of the data writing statement.

[0056] Expand the data writing statement in step S22, which includes three parts: target table, target field, and target field. The partition field is a special target field, and the target field and partition field are optional.

[0057] Step S232: Parsing the data write statement.

[0058] Extract the target table (TARGET_TABLE) and use the default execution library to eliminate semantic ambiguity to obtain the complete target table name (PARENT.TARGET_TABLE). If the target table's field information is explicitly declared, the target field (TARGET_COLUMN) is extracted according to the comma separator; if the target table's field information is not explicitly declared, the target table's field information in the table field mapping relationship is used as the target field. Extract the partition field from the partition information and append it as a special target field to the aforementioned target field; After parsing, the complete target table name and target fields are obtained. The number of target fields is equal to the number of data columns in the query result of the data query statement, and they correspond one-to-one in order.

[0059] Step S24: Structure and parsing of data query statements.

[0060] like Figure 5 As shown, the data query statement queries the database and returns the query results in the form of a two-dimensional table. Each data column in the query results comes from a different table field during the query. The purpose of parsing the data query statement is to obtain the source information of the query result data columns (composed of the source table and source fields).

[0061] The source information includes two parts: direct source and conditional source. The direct source is the source table field of the data in the query result, and the conditional source is the table field involved in data restriction operations such as data filtering, windowing, aggregation, and sorting in the query. Condition source ends are divided into two categories: column-domain condition source ends and global condition source ends. Column-domain condition source ends apply to specific data columns in the query results, while global condition source ends apply to all data columns in the query results.

[0062] Step S241: Determine the structure of the data query statement.

[0063] A data query statement is the union of one or more SELECT query statements connected vertically using the UNION [ALL] keyword. Vertical connection means that the data and source information of each data column in the query result come from the data and source information of the corresponding data columns in the results of all SELECT query statements.

[0064] Using an abstract syntax tree, each SELECT query statement is obtained by dividing the data into UNION [ALL] boundaries. The core of data query statement parsing is to parse each SELECT query statement and integrate its parsing results.

[0065] Expanding the internal structure of a SELECT query statement reveals six syntax blocks: 1. SELECT syntax block: Starting with SELECT as the keyword, it consists of one or more field expressions, including the direct source fields of the query results and the column field condition source fields; 2. FROM syntax block: Starting with FROM as the key, it contains source table information and source field information for global conditions; 3. The WHERE syntax block, starting with WHERE as the keyword, consists of one or more field expressions, containing global condition source field information; 4. GROUP syntax block: Starting with GROUP BY as the keyword, it consists of one or more field expressions and contains global condition source field information; 5. HAVING syntax block: Starting with the keyword HAVING, it consists of one or more field expressions and contains global conditional source field information; 6. ORDER syntax block: Starting with ORDER BY as the keyword, it consists of one or more field expressions, containing global condition source field information.

[0066] Step S242: Parsing the data query statement.

[0067] The parsing logic for a SELECT query statement is to parse the tables and fields involved in each of the above parts in turn.

[0068] Syntax block decomposition: Using an abstract syntax tree, the SELECT query statement is decomposed with the starting keyword of each syntax block as the boundary, resulting in six syntax blocks.

[0069] Parsing the SELECT syntax block: The SELECT syntax block consists of one or more field expressions containing aliases (COL_EXP [ [ AS ] ALIAS], where the alias ALIAS can be omitted), separated by commas. Each field expression corresponds to a data column in the result of the SELECT query statement. Parsing the SELECT syntax block retrieves the table fields used by these field expressions.

[0070] Separate the SELECT syntax blocks using commas to obtain field expressions containing aliases. Parse each field expression one by one according to the following steps: Step A1: If an alias is included, extract the alias ALIAS and parse the remaining part according to the field expression parsing method in step S26.

[0071] Step A2: If no alias is included, the field expression is parsed according to the field expression parsing method in step S26. If the field expression is a wildcard or a field reference, the field name is used as the alias; otherwise, the sequence number is used to refer to the alias.

[0072] Step A3: The aliases in steps A1-A2 are the column names of the data columns in the query results of the SELECT query statement. The source table fields in the field expression parsing results are the direct source table fields of this data column, and the condition table fields are the column field condition source table fields of this data column.

[0073] Parsing of the FROM syntax block (e.g.) Figure 6 (As shown): The FROM syntax block has three forms: single-table query, implicit join multi-table query, and explicit join multi-table query. Its parsing steps are as follows: Step B1: Obtain the structural features of the syntax block using the lexical unit type information of the abstract syntax tree, determine the syntax block form, and parse it according to the form.

[0074] Step B2: If it is a single-table query, then it only contains one source table (SOURCE_TABLE [ [ AS ]ALIAS]); If the form is implicit join, then each source table is obtained by using commas as the separator; If the form is an explicit join, then each source table is obtained based on the join keyword JOIN. The explicit join condition (belonging to the global condition source table) is identified by the join condition keyword ON. The syntax and data lineage of the explicit join condition are the same as those of the WHERE syntax block, and it is parsed in the WHERE syntax block.

[0075] Step B3: In the source table SOURCE_TABLE [ [ AS ] ALIAS] obtained in step B2, SOURCE_TABLE is the source table name and ALIAS is the alias. If an alias exists, obtain the alias and update the mapping relationship between the alias and the table name. Use the default execution library to eliminate the semantic ambiguity of the source table name and obtain the source table name PARENT.SOURCE_TABLE.

[0076] Step B4: If the source table SOURCE_TABLE obtained in step B2 is a subquery, then use the method in step S27 for processing.

[0077] Parsing the WHERE syntax block and explicit join conditions: The WHERE syntax block is responsible for data filtering, while explicit join conditions are the filtering conditions used when tables are joined horizontally. Both are combinations of multiple filtering conditions using the keywords AND / OR and parentheses as boundaries. Each filtering condition is a comparison operation or logical judgment with fields or constants as operands. The parsing steps are as follows: Step C1: Using the keywords AND / OR and parentheses as the dividing criteria, decompose the WHERE syntax block and explicit join conditions to obtain each filter condition.

[0078] Step C2: Parse each filter condition according to the field expression parsing method in step S26 to obtain the table fields used in the field expressions.

[0079] Step C3: All table fields in the field expression are source table fields of the global conditions included in the WHERE syntax block and explicit join conditions.

[0080] Parsing of other syntax blocks (GROUP, HAVING, ORDER): The GROUP, HAVING, and ORDER syntax blocks have similar structures, all being combinations of multiple conditions bounded by commas. Their parsing steps are as follows: Step D1: Using commas as separators, extract each condition from the syntax block.

[0081] Step D2: Parse each condition one by one according to the field expression parsing method in step S26 to obtain the table fields used in the field expression.

[0082] Step D3: All table fields in the field expression are source table fields of the global condition contained in the GROUP, HAVING, ORDER syntax blocks.

[0083] like Figure 7As shown, after parsing the above steps S242, the direct source information, column field condition source information, global condition source information, and column name information of the query result data columns used in the SELECT query statement are obtained. The direct source information and column field condition source information are matched one by one with the result data columns, and all global condition source information is assigned to each result data column, thus completing the integration of the parsing results.

[0084] Step S243: Integrate the results of parsing the data query statement.

[0085] like Figure 8 As shown, for a data query statement that contains a SELECT query statement, the parsing result of the SELECT query statement is the parsing result of the data query statement.

[0086] For a data query statement containing multiple SELECT query statements, the column name of the data query result column is the column name of the data query result column of the first SELECT query statement; the source information of the data query result column is the union of the source information of the corresponding data columns of all SELECT query results.

[0087] Step S25: Data lineage construction.

[0088] Step S251: Mapping data write statements to data query statements.

[0089] The data columns of the query result of the data query statement correspond one-to-one with the fields of the target table of the data write statement. A mapping relationship is established between the target fields parsed in S232 and the source table fields of the data columns of the query result of the data query statement integrated in S243.

[0090] Step S252: Source information tracing.

[0091] If the source information involves temporary tables or subquery fields, the intermediate data lineage is used to trace the temporary tables and subquery fields layer by layer until no temporary tables or subquery fields are included. After tracing, the field-level data lineage is constructed.

[0092] Step S26: Structure and parsing of field expressions.

[0093] In this invention, a field expression (COL_EXP) refers to an expression that takes a field or constant as input parameters, processes them through functions, judgments, calculations, etc., and returns data in the form of a constant value or field.

[0094] The purpose of field expression parsing is to identify and extract the table fields used in the field expression and divide them into two categories: source fields and condition fields. The results of field expression parsing are all source table fields.

[0095] Step S261: Summarize and classify the structure of field expressions.

[0096] Based on SQL syntax, field expressions consist of six components: keywords (KEYWORD), identifiers (COL), function names (FUNC), operators, constants (CONST), and punctuation. The structural forms of field expressions are summarized into three categories: basic, composite, and subquery, totaling 14 configurations. These 14 configurations include constant, field reference, wildcard, comparison, range judgment, special judgment, IF, CASE_WHEN, function, calculation, window, other expression, correlated subquery, and uncorrelated subquery. Each configuration contains one or more forms, and the structural characteristics of each form are as follows: Figure 9 In summary: 1. Basic type: Indivisible atomic field expressions, including three types: constant type, field reference type, and wildcard type.

[0097] 2. Composite type: A non-atomic field expression consisting of one or more configurations that can be decomposed into subfield expressions.

[0098] 3. Subquery type: This type of query encapsulates a data query statement to reference a field.

[0099] Step S262: Parsing process of field expressions.

[0100] Step F1: Configuration recognition.

[0101] The structure of the field expression is obtained by using the lexical unit type of the abstract syntax tree, and compared with the structural features of the above 14 configurations to determine the configuration and form of the field expression.

[0102] Step F2: If the field expression is a basic type, obtain the fields used in the field expression according to the following cases, and all of them are source fields, then proceed to step F3: (1) If the field expression is a field reference type, the referenced field is the parsed result of the field expression; (2) If the field expression is a wildcard type (" The wildcard is expanded in subsequent steps to obtain the parsing result of the field expression; (3) If the field expression is a constant, the parsing result of the field expression is empty.

[0103] If the field expression is a subquery, process it using the method in step S27. All fields of the subquery temporary table obtained by parsing the subquery are used as the source fields of the subquery field expression parsing result, and then step F3 is executed.

[0104] If the field expression is a composite type, it contains one or more sub-field expressions. Based on the structural characteristics of the nine composite type configurations, the field expression is decomposed into sub-field expressions using a depth-first traversal method with the help of an abstract syntax tree. Steps F1-F2 are repeated for each sub-field expression until the configuration of the sub-field expression is a basic type or a subquery type and the parsing result is obtained. The parsing result of a composite field expression is the union of the parsing results of all its sub-field expressions. Then, step F3 is executed. (1) If the field expression is a comparison operation type, range judgment type, or special judgment type, the parsing result is the condition field; (2) If the field expression is of type IF, the parsing result of the first parameter is the condition field, and the rest are the source fields; (3) If the field expression is of type CASE_WHEN, the parsing result of the parameters after the keywords THEN and ELSE is the source field, and the rest are the condition fields; (4) If the field expression is a window type, the parsing result of the parameters in the window function is the source field, and the rest are condition fields; (5) If the field expression is a function, calculation, or other expression type, the parsing result is the source field; (6) When merging the parsing results of subfield expressions, if the subfield expression is in the source field position of the field expression, then the source field of the merged subfield expression is the source field of the field expression, and the condition field of the subfield expression is the condition field of the field expression; if the subfield expression is in the condition field position of the field expression, then all the source fields of the merged subfield expression are the source fields of the field expression.

[0105] Step F3: Wildcard expansion, field identifier disambiguation, and table alias mapping After steps F1-F2, the parsing result of the field expression only includes the referenced field and wildcards.

[0106] The referenced field exists in multiple forms. By using the table field mapping relationship and the default execution library, combined with the source table, the semantic ambiguity of the TABLE.COLUMN and COLUMN forms is eliminated; by using the alias-table name mapping relationship, the alias in the ALIAS.COLUMN form is restored to the table name.

[0107] The wildcard is expanded into fields using table field mapping relationships, resulting in field references. If the wildcard does not declare a source table, it refers to all fields of the source tables in the FROM syntax block. Combining the source tables of the data query statement to which the field expression belongs and the table field mapping relationships, the fields of the source tables are expanded sequentially in the order of appearance. If the wildcard has already declared a source table (such as TAB_NAME., CTE_NAME., ALIAS), the expansion is performed accordingly. If the table field mapping relationship is used, the fields of the specified source table will be expanded.

[0108] After the above steps F1-F3, all source table fields used in the field expression are obtained.

[0109] Step S27: Parsing the subquery.

[0110] In the parsing of the FROM syntax block and field expression of a data query statement, subqueries may exist. A subquery is a data query statement enclosed in parentheses. Using a subquery is the query result of the data query statement enclosed in the subquery. Parsing a subquery is essentially parsing a partial data query statement. This invention treats subqueries as a temporary table with a shorter lifespan, called a subquery temporary table. The parsing steps are as follows: Step G1: Create a unique identifier to serve as the table name (SUB_NAME) for the subquery temporary table.

[0111] Step G2: Using the method in step S24, parse the data query statement of the subquery to obtain the data column names and source information of the query results. Wherein: if it is a subquery in the FROM syntax block or an unrelated subquery in the field expression, the table field mapping relationship used during parsing only includes the table field mapping relationship of the physical table and the temporary table; if it is a related subquery in the field expression, the table field mapping relationship used during parsing includes not only the table field mapping relationship of the physical table and the temporary table, but also the table field mapping relationship of other subqueries at the same level as the current subquery defined in the data query statement of the current subquery.

[0112] Step G3: Use the column names from the query results in step G2 as the field names (SUB_COLUMN) of the subquery temporary table.

[0113] Step G4: Based on the table names and field names from steps G1 and G3, construct the table structure of the subquery temporary table, and update the table field mapping relationship accordingly.

[0114] Step G5: Match the table fields of the temporary table in the subquery of step G4 with the source information obtained in step G2 to obtain the lineage information and update the lineage relationship in the intermediate data.

[0115] Step S28: Structure and parsing of temporary table definition statements.

[0116] Step S281: Structure of the temporary table definition statement.

[0117] like Figure 10 As shown, the temporary table definition statement in the data processing statement syntax structure diagram in step S22 is expanded. The temporary table definition statement contains the definition of multiple temporary tables. A temporary table contains three main parts: the temporary table name, the temporary table fields, and the temporary table data query statement.

[0118] The temporary table data query statement is essentially the same as the data processing statement in S24. The temporary table can be regarded as obtaining the query result through the temporary table data query statement and writing it into the temporary table field. The execution of this local data processing statement takes the temporary table field as the target end and the physical table field queried in the temporary table data query statement as the source end. Based on this, the intermediate data lineage of the temporary table can be constructed.

[0119] The target information (temporary table name and field names) is obtained by parsing a fixed syntax structure, and the source information is obtained by parsing the method in step S24.

[0120] Step S282: Parsing of the temporary table definition statement.

[0121] Step H1: Split each temporary table from the temporary table definition statement and process them one by one according to steps H2-H6.

[0122] Step H2: Extract the temporary table name as the target table name (CTE_TABLE_NAME).

[0123] Step H3: If the temporary table field is explicitly declared, extract the target field name (CTE_COLUMN_NAME) using comma separators. If it is not explicitly declared, use the column names of the query result data obtained in step H4 as the temporary table field names (i.e., the target field names).

[0124] Step H4: Using the method in S24, parse the query results from the temporary table data query statement to obtain the column names and source information.

[0125] Step H5: Update the table and field mapping relationship using the temporary table name and the temporary table field name.

[0126] Step H6: Match the table fields of the temporary table in step H5 with the source information obtained in step H4 to obtain the lineage information, and update the lineage relationship of the intermediate data.

[0127] Step S3: Extract and process relational data and store it in separate databases.

[0128] like Figure 11As shown, the field-level lineage data is stored in two forms: relational database and graph database. The source field of a lineage relationship is called the source end, and the target field to which data is written is called the target end.

[0129] Relational database storage: In a relational database, the data lineage table stores the following information: source fields (SCHEMA, TABLE, FIELD), target fields (SCHEMA, TABLE, FIELD), and relation attributes (TYPE).

[0130] Graph database storage: In the data architecture of a graph database, nodes are table field information (including SCHEMA, TABLE, FIELD of the source and target ends), relations (or edges) are lineage information between fields, and attributes are lineage attribute information.

[0131] Step S4: Visualization of bloodline analysis data.

[0132] Upstream and downstream relationship tracing: Tracing upstream and downstream relationships to a specified depth and visually displaying them in the form of a kinship map.

[0133] Link query: Supports querying links between specified nodes, allowing you to view the entire data processing process, including intermediate data items, processing paths, and other information.

[0134] Impact Analysis: Performs impact analysis on the data and presents the results in a table, listing all downstream nodes of a specified node.

[0135] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for data bloodline full-link management in a tax scenario, characterized in that, include: Step S1, Data Aggregation and Pre-cleaning of Source Tax System: Aggregate raw data from multiple tax-related data sources, standardize, process, check data quality, and pre-clean the aggregated raw data, and establish a unified data dictionary model; Step S2, Algorithm analysis of processing statements for tax aggregated data: Convert the data processing statements generated in step S1 into an abstract syntax tree, deconstruct the data processing statements based on the abstract syntax tree, identify data flow relationships, and extract field-level data lineage relationships; Step S3: Extract and process relational data for separate database storage: Store the parsed field-level data and lineage relationship data into a relational database and a graph database respectively; Step S4, Visualization of Data Lineage Analysis: Based on stored lineage data, it provides visualization functions for tracing upstream and downstream relationships, querying links to specified nodes, and analyzing impacts. Step S2 includes: defining and using a default execution library, table field mapping relationships, alias-table name mapping relationships, and intermediate data lineage relationships to eliminate semantic ambiguity of identifiers and achieve penetration of intermediate lineage relationships; deconstructing data processing statements into temporary table definition statements, data writing statements, and data query statements; extracting target-end information from data writing statements and extracting source-end information of query result data columns from data query statements; establishing field-level mapping relationships between the data columns of the query results of data query statements and the target-end table fields of data writing statements in sequence; if the source-end information involves temporary table or subquery fields, tracing back layer by layer using the intermediate data lineage relationships until the real physical source-end fields are obtained. 2.The data blood relationship full-link management method in a tax scenario of claim 1, wherein, The default execution library is used to eliminate semantic ambiguity and obtain a complete table name identifier when the parsed table name does not explicitly declare the database to which it belongs, in combination with the statement execution environment. The table field mapping relationship covers three types: temporary tables, external tables, and subquery output data columns. It is used to eliminate semantic ambiguity and obtain complete field identification when wildcards are used to refer to all fields or when the field name is not explicitly declared to belong to the table name. The alias-table name mapping relationship is used to restore the alias to the complete table name identifier when the parsed field name is declared to belong to the table through the table alias; The intermediate data lineage relationship is used to penetrate internal lineage information when using temporary tables or subqueries in data processing statements, so as to obtain the complete data lineage between the target end and the real physical source end. 3.The data blood relationship full-link management method in a tax scenario of claim 1, wherein, The deconstruction of the data processing statements includes: The data processing statements are deconstructed into temporary table definition statements starting with WITH, data write statements starting with INSERT or CREATE, and data query statements starting with SELECT. 4.The data blood relationship full-link management method in a tax scenario of claim 1, wherein, The extraction of target end information from the data write statement includes: Extract target information from the data write statement, which includes the target table and target fields; Extract the target table and use the default execution library to eliminate semantic ambiguity and obtain the complete target table name; If the target field is explicitly declared, it will be extracted using comma-separated parameters; If not explicitly declared, the target field will be the field information from the target table in the table field mapping relationship. Extract the partition field from the partition information and append it as a special target field to the above target field; This ensures that the number of target fields corresponds sequentially to the number of data columns in the query result.

5. The data bloodline full-link management method in a tax scenario according to claim 1, characterized in that, The source information for extracting query result data columns from the data query statement includes: The data query statements are divided using the concatenation keyword UNION or UNION ALL as boundaries to obtain each SELECT query statement; For each SELECT query statement, it is broken down into SELECT syntax block, FROM syntax block, WHERE syntax block, GROUP syntax block, HAVING syntax block and ORDER syntax block, with the starting keyword of each syntax block as the boundary. The tables and fields involved in each syntax block are parsed in turn to obtain direct source information, column-domain condition source information, and global condition source information; The direct source information and column-domain condition source information are mapped one-to-one with the query result data columns, and all global condition source information is assigned to each result data column to complete the integration of the parsed results. 6.The data blood relationship full-link management method in a tax scenario of claim 1, wherein, Step S2 also includes the destructuring of field expressions; The field expression refers to an expression that takes fields or constants as input parameters, is processed by functions, judgments, and calculations, and returns data in the form of constant values ​​or fields. The parsing of the field expression includes: Field expressions are categorized into three types: basic, composite, and subquery, with a total of 14 configurations. These 14 configurations include constant, field reference, wildcard, comparison, range judgment, special judgment, IF, CASE_WHEN, function, calculation, window, other expression, correlated subquery, and uncorrelated subquery. Identify the configuration of field expressions using an abstract syntax tree; If the field expression is a composite type, the field expression is decomposed into subfield expressions using a depth-first traversal method. The subfield expressions are then recursively parsed until the structure of the subfield expression is a basic type or a subquery type. The parsing result of a composite field expression is the union of the parsing results of all its subfield expressions. The parsing results are expanded using wildcards, disambiguated using field identifiers, and mapped to table aliases to obtain all source table fields used in the field expression. These source table fields are divided into two categories: source fields and condition fields.

7. The data bloodline full-link management method in a tax scenario according to claim 6, characterized in that, The classification rules for the parsing results of the composite field expression include: If the field expression is a comparison operation, range judgment, or special judgment type, the parsing result is the condition field; If the field expression is of type IF, the parsing result of the first parameter is the condition field, and the rest are the source fields; If the field expression is of type CASE_WHEN, the parsing result of the parameters after the keywords THEN and ELSE is the source field, and the rest are the condition fields; If the field expression is a window type, the parsing result of the parameters in the window function is the source field, and the rest are condition fields; If the field expression is a function, calculation, or other expression type, the parsed result is the source field; When merging the parsing results of subfield expressions, if the subfield expression is located in the source field position of the field expression, then the source field of the merged subfield expression becomes the source field of the field expression, and the condition field of the subfield expression becomes the condition field of the field expression; if the subfield expression is located in the condition field position of the field expression, then all fields of the merged subfield expression become the source field of the field expression.

8. A method for full-link data lineage management in a tax scenario according to claim 1, characterized in that, Step S2 also includes parsing the temporary table definition statement; The temporary table definition statement contains the definition of multiple temporary tables. A temporary table contains three main parts: the temporary table name, the temporary table fields, and the temporary table data query statement. The parsing of the temporary table definition statement includes: Extract each temporary table from the temporary table definition statement and use the temporary table name as the target table name; If a temporary table field is explicitly declared, extract the target field name using comma separators; If not explicitly declared, the column names of the parsed query results will be used as the field names of the temporary table; Use the data query statement parsing method to parse the data query statement from the temporary table to obtain the column names and source information of the query results; Update the table and field mapping relationship using the temporary table name and temporary table field names; The table fields of the temporary table are mapped one-to-one with the source information to obtain lineage information and update the lineage relationship in the intermediate data.

9. A method for end-to-end data lineage management in a tax scenario according to claim 1, characterized in that, Step S3 includes: Store a lineage relationship table in a relational database. Each record contains SCHEMA, TABLE, and FIELD information for the source field, SCHEMA, TABLE, and FIELD information for the target field, and relationship attribute information. In a graph database, each table field is treated as a node, and the lineage relationship between fields is treated as an edge. Node attributes include SCHEMA, TABLE, and FIELD, while edge attributes include relationship type and processing script identifier.

10. A method for end-to-end data lineage management in a tax scenario according to claim 1, characterized in that, Step S4 includes: Upstream and downstream relationship tracing: Traverse upstream and downstream relationships at a specified depth and visualize them in the form of a lineage graph. In the lineage graph, nodes represent fields, edges represent lineage relationships, and the display style of the edges distinguishes between direct lineage and conditional lineage. Link query: Supports link queries between specified source and target fields, displaying the complete processing path and intermediate data items between the two fields; Impact Analysis: Input a specified field, query and list all downstream nodes of that field, and display the downstream table name, field name, relationship type and processing script source in tabular form.