Database table field provenance method and apparatus
Patent Information
- Application Number
- CN202611239965.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-10-30
AI Technical Summary
随着数据加工链路日益复杂,一个下游表字段往往经过多层抽取-转换-加载(Extract-Transform-Load,ETL)、嵌套查询和存储过程加工才能生成,其数据来源追溯极为困难
在本技术方案中,获取数据库中待分析的SQL脚本;识别SQL脚本中的嵌套子查询,将嵌套子查询替换为临时表并在原位置建立对临时表的引用,并将临时表的定义前置,使SQL脚本转化为线性化脚本;从线性化脚本中提取所引用的物理表,关联存储以建立表级依赖索引;解析数据写入操作的目标字段与来源字段之间的映射关系,提取过滤条件信息,关联存储以构建字段级血缘关系库;基于字段级血缘关系库和表级依赖索引,对目标字段进行溯源,得到从目标字段到原始基础表的字段的血缘链路。由此,通过将嵌套子查询线性化,降低了复杂SQL脚本的解析难度;通过提取字段级映射关系并关联过滤条件信息,实现了从表级血缘到字段级血缘的精细化追溯;通过递归溯源临时表,打通了从目标字段到原始基础表字段的全链路血缘,从而提高了数据血缘追溯的准确性和效率。
Smart Images

Figure CN123011951A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a method and apparatus for tracing the source of database table fields. Background Technology
[0002] In the fields of data warehousing and data governance, data lineage analysis is a crucial step in supporting data quality control, troubleshooting, and impact analysis. As data processing chains become increasingly complex, a single downstream table field often undergoes multiple layers of extraction-transform-load (ETL), nested queries, and stored procedures before being generated, making it extremely difficult to trace the data source.
[0003] In related technologies, data lineage construction mainly relies on ETL tool metadata parsing or manual review of Structured Query Language (SQL) scripts. This can only identify table-level dependencies and cannot achieve field-level lineage tracing. Furthermore, when faced with multi-level nested subqueries and temporary table scenarios, the lineage link is prone to breakage and the parsing accuracy is poor. Summary of the Invention
[0004] This disclosure provides a method and apparatus for tracing the origin of database table fields, thereby at least partially addressing one of the technical problems in related technologies. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a database table field tracing method is provided, comprising: obtaining an SQL script to be analyzed in a database; identifying nested subqueries in the SQL script, replacing the nested subqueries with temporary tables and establishing a reference to the temporary tables in their original positions, while simultaneously forwarding the definition of the temporary tables, thereby transforming the SQL script into a linearized script that does not contain nested subqueries; extracting the referenced physical tables from the linearized script, and associating and storing the physical tables with the linearized script to establish a table-level dependency index; for data write operations in the linearized script, parsing the mapping relationship between the target field and the source field of the data write operation, extracting the filtering condition information in the data write operation, and associating and storing the mapping relationship and the filtering condition information to construct a field-level lineage database; and tracing the target field based on the field-level lineage database and the table-level dependency index to obtain the lineage link from the target field to the fields of the original base table, wherein the original base table is a physical table that does not depend on other fields for processing.
[0005] According to a second aspect of the present disclosure, a database table field tracing apparatus is provided, comprising: an acquisition module for acquiring an SQL script to be analyzed in a database; a conversion module for identifying nested subqueries in the SQL script, replacing the nested subqueries with temporary tables and establishing a reference to the temporary tables in their original positions, while simultaneously forwarding the definition of the temporary tables, so that the SQL script is converted into a linearized script that does not contain nested subqueries; a first storage module for extracting referenced physical tables from the linearized script, associating and storing the physical tables with the linearized script to establish a table-level dependency index; a second storage module for parsing the mapping relationship between the target field and the source field of a data write operation in the linearized script, extracting filtering condition information in the data write operation, and associating and storing the mapping relationship and the filtering condition information to construct a field-level lineage database; and a tracing module for tracing the target field based on the field-level lineage database and the table-level dependency index to obtain the lineage link from the target field to the field of the original base table, wherein the original base table is a physical table that does not depend on other fields for processing.
[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the database table field tracing method as described in the first aspect of the present disclosure.
[0007] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the database table field tracing method as described in the first aspect of the present disclosure.
[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the database table field tracing method as described in the first aspect of the present disclosure.
[0009] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: This technical solution involves: acquiring the SQL script to be analyzed from the database; identifying nested subqueries in the SQL script, replacing them with temporary tables and establishing references to these temporary tables in their original positions, and placing the definition of the temporary tables at the forefront to transform the SQL script into a linearized script; extracting the referenced physical tables from the linearized script and storing them in association to establish table-level dependent indexes; parsing the mapping relationship between the target field and the source field of the data write operation, extracting filtering condition information, and storing it in association to construct a field-level lineage database; and tracing the target field based on the field-level lineage database and the table-level dependent indexes to obtain the lineage link from the target field to the fields in the original base table. Thus, by linearizing the nested subqueries, the parsing difficulty of complex SQL scripts is reduced; by extracting field-level mapping relationships and associating filtering condition information, fine-grained tracing from table-level lineage to field-level lineage is achieved; and by recursively tracing the temporary table, the entire lineage link from the target field to the fields in the original base table is established, thereby improving the accuracy and efficiency of data lineage tracing.
[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.
[0012] Figure 1 This is a flowchart illustrating the database table field tracing method shown in the first embodiment of this disclosure; Figure 2 This is a flowchart illustrating the database table field tracing method shown in the second embodiment of this disclosure; Figure 3 This is a flowchart illustrating the database table field tracing method shown in the third embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of the database table field tracing device shown in the fourth embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Implementation
[0013] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.
[0014] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0015] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.
[0016] In the fields of data warehousing and data governance, data lineage analysis is a crucial step in supporting data quality control, troubleshooting, and impact analysis. As data processing chains become increasingly complex, a single downstream table field often undergoes multiple layers of extraction-transform-load (ETL), nested queries, and stored procedures before being generated, making it extremely difficult to trace the data source.
[0017] In related technologies, data lineage construction mainly relies on ETL tool metadata parsing or manual review of Structured Query Language (SQL) scripts. This can only identify table-level dependencies and cannot achieve field-level lineage tracing. Furthermore, when faced with multi-level nested subqueries and temporary table scenarios, the lineage link is prone to breakage and the parsing accuracy is poor.
[0018] The database table field tracing method and apparatus of this disclosure are described below with reference to the accompanying drawings.
[0019] Figure 1 This is a flowchart illustrating the database table field tracing method shown in the first embodiment of this disclosure.
[0020] like Figure 1 As shown, the database table field tracing method includes the following steps: Step 101: Obtain the SQL script to be analyzed from the database.
[0021] It should be noted that the SQL scripts to be analyzed can come from various data sources, including but not limited to: processing scripts exported by ETL tools, SQL statements embedded in Perl / Shell scripts, view, stored procedure, and function definitions in the database, as well as data manipulation language (DML) and data definition language (DDL) statements captured from database logs (such as Binlog and Redo Log).
[0022] Because scripts from different data sources have inconsistent formats and contain interfering information such as comments, blank lines, and macro variables, direct parsing can lead to parser errors or inaccurate results. Therefore, script standardization preprocessing is necessary. In this embodiment, obtaining the SQL script to be analyzed from the database includes: obtaining a multi-source SQL script set from the database; and performing standardization preprocessing on each SQL script in the SQL script set to obtain a standardized SQL script to be analyzed. The standardized preprocessing includes: Each SQL script is divided into independent SQL statements or stored procedures according to logical units, and a unique script identifier is assigned to each logical unit and stored as an independent script; for example, a hash algorithm can be used to generate a unique identifier (SCRIPT_ID) to associate the logical unit (script) to which each lineage record belongs in the subsequent relationship table.
[0023] Remove comments and blank lines from each logical unit; specifically, identify and remove single-line comments (--) and multi-line comments ( / ... / ) in the SQL script, and delete meaningless blank lines to reduce parsing interference.
[0024] Standardize the case of keywords in each logical unit; for example, it can be uniformly converted to uppercase or lowercase according to the database type to eliminate parsing interference caused by inconsistent case.
[0025] Expand and replace the macro definitions or variables in each logical unit to restore the most accurate execution SQL.
[0026] Therefore, by standardizing the preprocessing of multi-source heterogeneous SQL scripts, scripts from different technology stacks can be unified into a standard format that can be recognized by subsequent parsers, effectively avoiding parsing errors caused by format differences and improving the success rate and accuracy of SQL script parsing.
[0027] Step 102: Identify nested subqueries in the SQL script, replace the nested subqueries with temporary tables and create references to the temporary tables in their original positions, while moving the definition of the temporary tables to the front, so that the SQL script is transformed into a linearized script that does not contain nested subqueries.
[0028] Complex data processing scripts often contain numerous multi-level nested subqueries. Directly parsing deeply nested SQL statements can lead to parser memory overflow or parsing failure, and it is difficult to trace the field mapping relationships layer by layer in the nested logic. To address this, this disclosure introduces a nested subquery flattening and temporary table reconstruction mechanism to transform the unparsable nested structure into a linear structure that can be parsed layer by layer.
[0029] As an example, by flattening nested subqueries into temporary tables and defining them in advance, complex nested subqueries that were originally not directly parsable are transformed into a linear execution flow that is easy to parse layer by layer. On the one hand, because the temporary table definition is placed before the main query, each temporary table is independently defined as a clear intermediate node, which decomposes the originally intertwined nested logic into a sequentially executed hierarchical structure, significantly reducing the complexity of a single parsing, avoiding parser memory overflow or parsing failure due to excessive nesting, and improving the success rate and stability of SQL script parsing. On the other hand, in the linearized script, each temporary table corresponds to an independent definition statement. Subsequent steps can perform table-level dependency extraction and field-level lineage parsing for each temporary table layer by layer, without having to process the entire nested structure in one parsing, thus providing a structured foundation for subsequent fine-grained lineage tracing.
[0030] Step 103: Extract the referenced physical table from the linearization script, and associate the physical table with the linearization script for storage to establish a table-level dependency index.
[0031] To store script-level table dependencies in a structured manner and provide table-level index support for subsequent field-level lineage tracing, thereby enabling rapid location of the script to which the target field belongs and the range of physical tables it depends on during the tracing process, and avoiding repeated scanning of the entire script, in this embodiment of the disclosure, the referenced physical tables are extracted from the linearized script, and the physical tables are associated with the linearized script for storage to establish a table-level dependency index. This includes: parsing the linearized script, identifying and extracting the database object names after excluding SQL keywords and system function names, as the physical tables referenced by the logical units corresponding to the linearized script; obtaining the script identifier of the logical unit, associating each extracted physical table with the script identifier, generating a correspondence record between each physical table and the logical unit; and storing the correspondence record in a first relation table (e.g., named SCRIPT_TABLE_MP) to form a table-level dependency index with the script identifier as the dimension. Therefore, by establishing table-level dependency indexes, the relationship between the logical unit corresponding to the linearized script and the physical table it references is stored in a structured manner in a relational database. This provides table-level index support for subsequent field-level lineage tracing, enabling the logical unit where the target field is located and the range of physical tables it depends on to be quickly located during the tracing process. This avoids repeated scanning of the entire script and improves tracing efficiency.
[0032] The first relational table's fields include: primary key (ID), script identifier (SCRIPT_ID), and physical table name (TABLE_NAME). Optionally, it may also include read / write type (ACCESS_TYPE, used to identify whether the physical table is used as a source table (SOURCE) or a target table (TARGET) in the logical unit.
[0033] In summary, by constructing table-level dependency indexes, a coarse-grained relationship was established between the logical units corresponding to the linearized scripts and the physical tables, providing table-level index support for subsequent field-level lineage tracing.
[0034] Step 104: For the data write operation in the linearization script, parse the mapping relationship between the target field and the source field of the data write operation, extract the filtering condition information in the data write operation, and store the mapping relationship and the filtering condition information together to build a field-level lineage database.
[0035] In this embodiment, the table-level dependency index can only reflect the dependency relationship between the logical unit corresponding to the linearized script and the physical table, which cannot meet the requirements of fine-grained field-level management. Therefore, it is necessary to perform field-level mapping resolution for each data write operation in the logical unit corresponding to the linearized script.
[0036] As an example, by locating each data write operation in the linearization script, the target field and source field are matched one by one, and the processing logic expression and filtering condition information are extracted and stored in a relational table. This constructs a field-level lineage database, enabling each field mapping record to be fully traced back to its source table, source field, processing logic, and filtering constraints. This achieves a leap from table-level lineage to field-level lineage, providing a precise data foundation for subsequent field-level data tracing and impact analysis, and solving the problem in related technologies that can only identify table-level dependencies but cannot trace field-level processing logic.
[0037] Step 105: Based on the field-level lineage database and table-level dependency index, trace the source of the target field to obtain the lineage link from the target field to the field in the original base table. The original base table is a physical table that does not depend on other fields for processing.
[0038] In practical applications, database systems typically contain a large number of data processing scripts. Therefore, when obtaining the SQL script to be analyzed, all data processing scripts in the system can be traversed according to the script dependency order or a preset execution order. Each data processing script is then executed sequentially as the SQL script to be analyzed, following steps 101 to 104, until the lineage of all SQL scripts has been extracted and stored in the field-level lineage database and the table-level dependency index. Thus, through full script batch processing, all data processing scripts in the system can be processed uniformly, ensuring that the field-level lineage database records the lineage information of all SQL scripts, providing a complete data foundation for tracing the origin of any subsequent target field.
[0039] After all SQL scripts are parsed and inserted into the database, the field-level lineage database stores the mapping relationships between each field. However, since the data processing chain may contain multiple layers of temporary tables, directly querying the source of a target field may only yield the source field of the previous layer. If this source field is a temporary table field, the lineage chain breaks at the temporary table, making full-chain tracing impossible. Therefore, a recursive tracing mechanism is needed to establish a complete chain.
[0040] Specifically, in order to establish a complete lineage from the target field to the original base table field and avoid the lineage link being broken at the temporary table, in this embodiment of the disclosure, the target field is traced based on a field-level lineage database and a table-level dependency index to obtain the lineage link from the target field to the field of the original base table. This includes: starting from the target field, querying the field-level lineage database to obtain the mapping relationship corresponding to the target field, and determining the direct source field of the target field based on the mapping relationship; determining the type of the direct source field: if the direct source field is a field of a physical table, querying the table-level dependency index, and if it is confirmed that the physical table is the original base table, using the direct source field as a field of the original base table, and using the currently traced fields to form the lineage link from the target field to the field of the original base table; if the direct source field is a field of a temporary table, obtaining the definition statement of the temporary table, using the definition statement as a new SQL script to continue tracing upwards until the original base table field is reached, and using the currently traced fields to form the lineage link from the target field to the field of the original base table. Therefore, by determining the type of the direct source field, different processing paths are applied to fields in the physical table and fields in the temporary table. Fields in the physical table, after being confirmed as originating from the original base table via table-level dependent indexes, serve as the endpoint for tracing. Fields in the temporary table are recursively retrieved from their definition statements as new scripts to continue tracing upwards until the original base table is reached. This mechanism enables lineage tracing to penetrate through the temporary tables layer by layer, establishing a complete lineage from the target field to the original base table field. This solves the technical problem of lineage breaking at the temporary table in related technologies, providing complete link information for field-level data source tracing and impact analysis.
[0041] Here, "physical table" refers to the table in the database that actually stores the data, as opposed to temporary tables. "Original base table" refers to a physical table that does not depend on other fields for processing, i.e., the physical table located at the end of the lineage chain. By querying the first relation table (table-level dependent index), it can be confirmed whether the physical table has an upstream source. If the physical table does not have a record as the target table in the table-level dependent index, it indicates that it is the original base table, and the tracing terminates.
[0042] In summary, the process involves: obtaining the SQL script to be analyzed from the database; identifying nested subqueries within the SQL script; replacing these nested subqueries with temporary tables and establishing references to these temporary tables in their original positions, while also placing the temporary table definitions at the forefront to transform the SQL script into a linearized script; extracting the referenced physical tables from the linearized script and storing them in relation to create table-level dependent indexes; parsing the mapping relationship between the target field and the source field of the data write operation, extracting filtering condition information, and storing it in relation to construct a field-level lineage database; and tracing the target field based on the field-level lineage database and the table-level dependent indexes to obtain the lineage link from the target field to the original base table. Thus, by linearizing nested subqueries, the parsing difficulty of complex SQL scripts is reduced; by extracting field-level mapping relationships and associating filtering condition information, fine-grained tracing from table-level lineage to field-level lineage is achieved; and by recursively tracing the temporary table, the entire lineage link from the target field to the original base table is established, thereby improving the accuracy and efficiency of data lineage tracing.
[0043] To clearly illustrate how the nested subqueries in the SQL script are identified in the above embodiments, the nested subqueries are replaced with temporary tables and references to the temporary tables are established in the original positions, and the definition of the temporary tables is brought forward, so that the SQL script is transformed into a linearized script that does not contain nested subqueries, this disclosure proposes another database table field tracing method.
[0044] Figure 2 This is a flowchart illustrating the database table field tracing method shown in the second embodiment of this disclosure.
[0045] like Figure 2 As shown, the database table field tracing method includes the following steps: Step 201: Obtain the SQL script to be analyzed from the database.
[0046] Step 202: Parse the SQL script into an abstract syntax tree, and identify the original nesting position of the nested subqueries based on the abstract syntax tree.
[0047] In this embodiment of the disclosure, the Abstract Syntax Tree (AST) parser parses the SQL statements of the SQL script into a tree structure, accurately identifies the start position, end position and hierarchical relationship of each nested subquery, thereby accurately locating the nested subqueries within the parentheses of the FROM clause or JOIN clause, providing accurate location information for subsequent extraction and replacement.
[0048] As an example, an SQL script is input into an SQL parser, which outputs a corresponding abstract syntax tree (AST). Based on this AST, all nodes are traversed, nodes of type nested subqueries are identified, and their parent node information and position offset in the original SQL statement are recorded as the original nesting position of the nested subquery. Therefore, by using an AST parser instead of traditional regular expression matching, multi-level nested structures can be accurately handled, avoiding the technical shortcomings of regular expressions in handling nesting levels, and the tendency to miss or mismatch. This ensures that the position information of each nested subquery is accurately recorded, providing a reliable positional basis for subsequent extraction, replacement, and linearization, thereby effectively improving the accuracy of nested subquery identification and the stability of parsing.
[0049] Step 203: Extract nested subqueries based on the original nesting position, and generate a unique temporary table name for each extracted nested subquery.
[0050] In this embodiment, based on the original nesting position identified in step 202, the subquery statement fragments (i.e., the complete multi-level query logic) corresponding to the nested subqueries are extracted from the SQL script. Simultaneously, a unique temporary table name is generated for each extracted nested subquery. For example, the temporary table name can be generated using "TMP_" followed by a sequence number (such as TMP_001, TMP_002) or a hash value, ensuring the uniqueness of the temporary table name within the same script execution context. This allows subsequent steps to accurately associate each subquery statement fragment with its corresponding temporary table. Therefore, by generating a unique temporary table name for each extracted nested subquery, each temporary table is uniquely identified and referenced within the SQL script context, avoiding name conflicts in temporary tables after multiple nested subqueries are flattened. This provides an accurate identification basis for subsequently replacing the nested subqueries with the temporary table name at the original nesting position and establishing reference relationships.
[0051] Step 204: Replace the nested subquery with the name of the temporary table in the original nested position to form a reference to the temporary table, and generate the definition statement of the temporary table at the same time.
[0052] In this embodiment, the location of the nested subquery in the SQL script is replaced with the name of the temporary table, so that the original nested location no longer contains a nested subquery statement, but instead becomes a name reference to the temporary table. Simultaneously, a corresponding "CREATE TEMPORARY TABLE temporary_table_name AS (subquery)" statement is generated as the definition statement for the temporary table. Thus, by replacing the nested subquery with the name of the temporary table in the original nested location, a name reference to the temporary table is formed in the main query, and a definition statement for the temporary table is generated. This transforms the originally executed nested subquery logic into a temporary table object that can be independently defined and referenced, thereby stripping the nested structure from the main query and laying the foundation for obtaining a linearized script that does not contain nested subqueries.
[0053] Step 205: Place the generated definition statement before the main query that references the temporary table, thus converting the nested execution logic into sequential execution logic.
[0054] In this embodiment, the temporary table definition statement generated in step 204 is inserted before the main query that references the temporary table. This ensures that the temporary table has been defined by the time the main query is executed, thus making the originally nested subqueries execute sequentially. This facilitates the subsequent parsing of the field mapping relationships of each temporary table layer by layer. Therefore, the definition logic of each temporary table is independently parsed as an intermediate layer. Subsequent steps do not need to handle complex nested structures; they only need to parse the field mapping relationships of each temporary table layer by layer according to the script execution order to complete lineage tracing.
[0055] Step 206: Traverse all nested subqueries in the SQL script, and perform replacement, generation definition statement and predefinition statement operations on each nested subquery to obtain a linearized script that does not contain nested subqueries.
[0056] In this embodiment of the disclosure, for multiple nested subqueries in the same SQL script, steps 202 to 205 are repeated until all nested subqueries are flattened, ultimately resulting in a linearized script that does not contain any nested subqueries. In this linearized script, all the original nested subqueries are replaced with temporary table references, and the corresponding temporary table definition statements are placed at the beginning. The execution flow of the entire script becomes a top-down sequential execution, thereby reducing the complexity of subsequent table-level dependency extraction and field-level lineage resolution, and avoiding parsing failures due to excessive nesting.
[0057] Step 207: Extract the referenced physical table from the linearization script, and associate the physical table with the linearization script for storage to establish a table-level dependency index.
[0058] Step 208: For the data write operation in the linearization script, parse the mapping relationship between the target field and the source field of the data write operation, extract the filtering condition information in the data write operation, and store the mapping relationship and the filtering condition information together to build a field-level lineage database.
[0059] Step 209: Based on the field-level lineage database and table-level dependency index, trace the source of the target field to obtain the lineage link from the target field to the field in the original base table. The original base table is a physical table that does not depend on other fields for processing.
[0060] It should be noted that the execution process of steps 201 and steps 207 to 209 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.
[0061] In summary, the SQL script is parsed into an abstract syntax tree (AST), and the original nesting positions of nested subqueries are identified based on the AST. Nested subqueries are extracted from their original nesting positions, and a unique temporary table name is generated for each extracted subquery. The nested subqueries are then replaced with the temporary table names at their original nesting positions, creating a reference to the temporary table and generating a definition statement for it. This definition statement is placed before the main query referencing the temporary table, transforming the nested execution logic into sequential execution logic. Finally, all nested subqueries in the SQL script are traversed, and for each subquery, replacement, definition statement generation, and preceding definition statement operations are performed, resulting in a linearized script that does not contain nested subqueries. Thus, complex nested queries that were originally difficult to parse directly are transformed into a linear execution flow that is easy to parse layer by layer, solving the technical challenge of parsing nested queries.
[0062] To clearly illustrate how the field-level lineage database is constructed in the above embodiments, this disclosure proposes another method for tracing database table fields.
[0063] Figure 3 This is a flowchart illustrating the database table field tracing method shown in the third embodiment of this disclosure.
[0064] like Figure 3 As shown, the database table field tracing method includes the following steps: Step 301: Obtain the SQL script to be analyzed from the database.
[0065] Step 302: Identify nested subqueries in the SQL script, replace the nested subqueries with temporary tables and create references to the temporary tables in their original positions, and simultaneously move the definition of the temporary tables to the front, so that the SQL script is transformed into a linearized script that does not contain nested subqueries.
[0066] Step 303: Extract the referenced physical table from the linearization script, and store the physical table in association with the linearization script to establish a table-level dependency index.
[0067] Step 304: Locate each data write operation in the linearization script and determine the target field list of the target table corresponding to each data write operation.
[0068] In this embodiment, data write operations include, but are not limited to, INSERT INTO statements, CREATE VIEW statements, CREATE TABLE AS SELECT statements, and MERGE INTO statements. By locating these statements, the target table and its target field list can be determined. Specifically, in an INSERT INTO statement, the target table is the table name specified after the INSERT INTO keyword, and the target field list is the list of fields within parentheses after the table name; in a CREATE VIEW statement, the target table is the view name, and the target field list is the list of fields before the SELECT statement in the view definition. Thus, by accurately locating and parsing different types of data write operations, the target table and target field list corresponding to each write statement can be accurately extracted, providing accurate target information for the subsequent establishment of field-level mapping relationships. Regardless of the write method used by the data processing script, its write target can be uniformly identified, ensuring the integrity and compatibility of field-level lineage resolution and avoiding the omission of target tables or target fields due to differences in write operation types.
[0069] Step 305: Parse the source field list for each data write operation, match each target field in the target field list and the source field list with the corresponding source field one by one, and establish a mapping relationship between the target field and the source field; The mapping relationships include: direct mapping where the source field is directly assigned to the target field, and derived mapping where the source field is assigned to the target field after function operation, arithmetic operation, or string concatenation.
[0070] In this embodiment of the disclosure, direct mapping occurs when a source field is directly assigned to a target field (e.g., "SELECT col_aFROM ..."). In this case, there is a one-to-one direct correspondence between the target field and the source field. Derivative mapping occurs when a source field is assigned to a target field after function operations, arithmetic operations, or string concatenation (e.g., "SELECT func(col_b) + col_cFROM ..."). In this case, the target field depends on multiple source fields, and the corresponding processing logic expression needs to be recorded (e.g., "func(col_b)+col_c").
[0071] As an example, the field list in a SELECT statement is parsed. For each target field, the corresponding expression is found in the SELECT list. If the expression is a single field reference (e.g., "col_a"), a direct mapping relationship is established; if the expression contains function operations, arithmetic operations, or string concatenation (e.g., "func(col_b)+col_c"), a derived mapping relationship is established, and all field references in the expression are extracted as the source field list, while the complete processing logic expression is recorded. By distinguishing between direct and derived mapping, the correspondence and processing logic between target fields and source fields can be accurately recorded. Direct mapping preserves the direct dependencies between fields, while derived mapping fully records the transformation logic of fields after function operations, arithmetic operations, or string concatenation. This ensures that each field-level lineage can be traced back to a specific source field and processing expression, providing accurate processing link information for subsequent field-level lineage analysis and influence analysis.
[0072] Step 306: Extract the source table information and filter condition information from each data write operation.
[0073] In this embodiment, the FROM and JOIN clauses in the data write operation are parsed to extract all involved table names as source table information. Simultaneously, the conditional expressions in the WHERE clause are parsed to extract filtering condition information. This filtering condition information can serve as field processing constraints or as a basis for impact analysis, used to determine the effective range of associated fields during subsequent tracing. Thus, by extracting source table information and filtering condition information, a complete context of table-level source and row-level constraints is provided for each field mapping relationship. The source table information clarifies which tables the field data originates from, while the filtering condition information limits the effective range of field values. The combination of both allows field-level lineage relationships to answer not only "which field the data comes from" but also "under what conditions the data was generated," providing a more complete basis for subsequent data quality checks and impact analysis.
[0074] Step 307: Store the mapping relationship between each target field and the source field in the second relation table, and store the corresponding source table information and filter condition information in the third relation table. The second relation table and the third relation table are associated through the script identifier and source table information of the logical unit corresponding to the same linearization script, so that each mapping relationship can be associated with the corresponding source table information and filter condition information, and complete the associated storage of mapping relationship and filter condition information to build a field-level lineage relationship library.
[0075] As an example, the second relation table (e.g., named SCR_COL_TAR_SRC_TBWH_MP) includes the following fields: ID (primary key), SCRIPT_ID (script identifier), TARGET_TABLE (target table name), TARGET_COLUMN (target field name), SOURCE_TABLE (source table name), SOURCE_COLUMN (source field name), and LOGIC (processing logic expression). The third relation table (e.g., named SCR_TAB_WHE) includes the following fields: TBWH_ID (primary key), SCRIPT_ID (script identifier), TABLE_NAME (source table name), and FILTER_CONDITION (filtering condition information). The two tables are linked through SCRIPT_ID and TABLE_NAME (i.e., the source table name), allowing each field mapping record to be traced back to its corresponding source table and filtering condition. The SOURCE_TABLE (source table name) can be determined based on the source table information.
[0076] Therefore, by constructing a second and a third relation table and associating them through related fields, not only are the mapping relationships between fields recorded, but also the processing logic expressions and filtering condition information are preserved. This allows each field-level lineage relationship to be fully traced back to its source table and filtering constraints, resulting in a much higher information density than the solution that only records table-level dependencies.
[0077] Step 308: Based on the field-level lineage database and table-level dependency index, trace the source of the target field to obtain the lineage link from the target field to the field in the original base table. The original base table is a physical table that does not depend on other fields for processing.
[0078] It should be noted that the execution processes of steps 301 to 303 and 308 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit these processes and will not elaborate further.
[0079] In summary, through Figure 3In the illustrated embodiment, each data write operation in the linearization script is located, and the target field list of the target table corresponding to each data write operation is determined. The source field list of each data write operation is parsed, and each target field in the target field list and the source field list is matched one by one with the corresponding source field to establish a mapping relationship between the target field and the source field. The mapping relationship includes direct mapping and derived mapping. The source table information and filter condition information in each data write operation are extracted. The mapping relationship between each target field and the source field is stored in a second relation table, and the corresponding source table information and filter condition information are stored in a third relation table. The two tables are associated through the script identifier and the source table information to complete the associated storage of the mapping relationship and the filter condition information, so as to construct a field-level lineage relationship database. Thus, this disclosure can accurately extract field-level mapping relationships and associate processing logic and filter conditions, realizing fine-grained tracing from table-level lineage to field-level lineage.
[0080] In addition, to facilitate users' intuitive viewing of the lineage relationships between fields, after constructing the lineage link from the target field to the fields of the original base table, this embodiment of the disclosure also includes graph construction and query steps; graph construction: further includes: constructing a directed acyclic graph (DAG) with the mapping relationship in the field-level lineage relationship database as edges and the target field and source field involved in the mapping relationship as nodes; the nodes in the directed acyclic graph are combinations of "table name.field name", and the edges are mapping relationships.
[0081] Query: In response to a query request for a target field, the system traverses the directed acyclic graph (DAG), queries the DAG, and displays the lineage links from the target field to the fields in the original base table. Specifically, when the user inputs any target field, the system starts from the target field node and traverses the DAG in the reverse direction of the directed edges (i.e., from the target node to the upstream source node), outputting the complete processing path from the target field to the fields in the original base table.
[0082] Traversing the directed acyclic graph (DAG) can be achieved using either recursive common table expressions (RCAs) or consecutive left joins in relational databases, or directed edge traversal algorithms in graph databases such as Neo4j. By constructing a DAG and performing graph traversal in response to query requests, the dependencies between fields can be visually presented, enabling rapid identification of data sources and their impact, significantly improving the efficiency of data problem investigation and impact analysis.
[0083] Figure 4 This is a schematic diagram of the structure of the database table field tracing device shown in the seventh embodiment of this disclosure.
[0084] like Figure 4 As shown, the database table field tracing device 400 includes: an acquisition module 401, a conversion module 402, a first storage module 403, a second storage module 404, and a tracing module 405.
[0085] The system includes the following modules: an acquisition module 401, which acquires the SQL script to be analyzed from the database; a conversion module 402, which identifies nested subqueries in the SQL script, replaces the nested subqueries with temporary tables, establishes a reference to the temporary tables in their original positions, and simultaneously forwards the definition of the temporary tables, thus converting the SQL script into a linearized script without nested subqueries; a first storage module 403, which extracts the referenced physical tables from the linearized script and stores the physical tables in association with the linearized script to establish a table-level dependency index; a second storage module 404, which, for data write operations in the linearized script, parses the mapping relationship between the target field and the source field of the data write operation, extracts the filtering condition information in the data write operation, and stores the mapping relationship and the filtering condition information in association to construct a field-level lineage database; and a tracing module 405, which, based on the field-level lineage database and the table-level dependency index, traces the target field to obtain the lineage link from the target field to the field in the original base table, wherein the original base table is a physical table that does not depend on other fields for processing.
[0086] As one possible implementation of this disclosure, the acquisition module 401 is specifically used for: acquiring a multi-source SQL script set from a database; performing standardization preprocessing on each SQL script in the SQL script set to obtain a standardized SQL script to be analyzed; wherein, the standardization preprocessing includes: cutting each SQL script into independent SQL statements or stored procedures according to logical units, and assigning a unique script identifier to each logical unit; removing comments and blank lines in each logical unit; unifying the capitalization of keywords in each logical unit; and expanding and replacing macro definitions or variables in each logical unit.
[0087] As one possible implementation of this disclosure, the conversion module 402 is specifically configured to: parse the SQL script into an abstract syntax tree, identify the original nesting position of the nested subqueries based on the abstract syntax tree; extract the nested subqueries according to the original nesting position, and generate a unique temporary table name for each extracted nested subquery; replace the nested subquery with the table name of the temporary table at the original nesting position to form a reference to the temporary table, and simultaneously generate a definition statement for the temporary table; place the generated definition statement before the main query that references the temporary table, so that the nested execution logic is converted into sequential execution logic; traverse all nested subqueries in the SQL script, and perform the replacement, definition statement generation, and pre-definition statement operations on each nested subquery to obtain a linearized script that does not contain nested subqueries.
[0088] As one possible implementation of this disclosure, the first storage module 403 is specifically used for: parsing the linearized script, identifying and extracting the database object name after excluding SQL keywords and system function names, as the physical table referenced by the logical unit corresponding to the linearized script; obtaining the script identifier of the logical unit, associating each extracted physical table with the script identifier, generating a correspondence record between each physical table and the logical unit; and storing the correspondence record in a first relation table to form a table-level dependency index with the script identifier as the dimension.
[0089] As one possible implementation of this disclosure, the second storage module 404 is specifically used for: Locate each data write operation in the linearization script and determine the target field list of the target table corresponding to each data write operation; Parse the source field list for each data write operation, and match each target field in the target field list and the source field list with the corresponding source field one by one to establish a mapping relationship between the target field and the source field; the mapping relationship includes: direct mapping where the source field is directly assigned to the target field, and derived mapping where the source field is assigned to the target field after function operation, arithmetic operation or string concatenation; Extract the source table information and filter condition information from each data write operation; The mapping relationship between each target field and the source field is stored in the second relation table, and the corresponding source table information and filter condition information are stored in the third relation table. The second relation table and the third relation table are associated with the script identifier and source table information of the logical unit corresponding to the same linearized script, so that each mapping relationship can be associated with the corresponding source table information and filter condition information, thus completing the associated storage of the mapping relationship and the filter condition information to construct a field-level lineage relationship library.
[0090] As one possible implementation of this disclosure, the tracing module 405 is specifically configured to: starting from the target field, query the field-level lineage database to obtain the mapping relationship corresponding to the target field, and determine the direct source field of the target field based on the mapping relationship; determine the type of the direct source field: if the direct source field is a field of a physical table, query the table-level dependent index, and if it is confirmed that the physical table is the original base table, use the direct source field as a field of the original base table, and use the currently traced fields to form a lineage link from the target field to the field of the original base table; if the direct source field is a field of a temporary table, obtain the definition statement of the temporary table, use the definition statement as a new SQL script to continue tracing upwards until the field of the original base table is reached, and use the currently traced fields to form a lineage link from the target field to the field of the original base table.
[0091] As one possible implementation of this disclosure, after constructing the lineage link from the target field to the fields of the original base table, the apparatus further includes: a construction module, configured to construct a directed acyclic graph using the mapping relationships in the field-level lineage relationship database as edges and the target field and source field involved in the mapping relationships as nodes; and a display module, configured to traverse the directed acyclic graph in response to a query request for the target field, query and display the lineage link from the target field to the fields of the original base table in the directed acyclic graph.
[0092] As one possible implementation of this disclosure, traversing the directed acyclic graph includes: using a recursive common table expression or a continuous left join query in a relational database; or, using a directed edge traversal algorithm in a graph database.
[0093] The database table field tracing device of this disclosure acquires the SQL script to be analyzed in the database; identifies nested subqueries in the SQL script, replaces the nested subqueries with temporary tables and establishes a reference to the temporary tables in their original positions, and forwards the definition of the temporary tables, thus transforming the SQL script into a linearized script; extracts the referenced physical tables from the linearized script, and associates and stores them to establish a table-level dependency index; parses the mapping relationship between the target field and the source field of the data write operation, extracts filtering condition information, and associates and stores it to construct a field-level lineage database; based on the field-level lineage database and the table-level dependency index, it traces the source of the target field to obtain the lineage link from the target field to the field in the original base table. Thus, by linearizing the nested subqueries, the parsing difficulty of complex SQL scripts is reduced; by extracting field-level mapping relationships and associating filtering condition information, fine-grained tracing from table-level lineage to field-level lineage is achieved; by recursively tracing the temporary tables, the entire lineage link from the target field to the field in the original base table is established, thereby improving the accuracy and efficiency of data lineage tracing.
[0094] In an exemplary embodiment, an electronic device is also proposed.
[0095] The electronic devices include: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the database table field tracing method as proposed in any of the foregoing embodiments.
[0096] As an example, Figure 5 This is a schematic diagram of the structure of an electronic device 500 as shown in an exemplary embodiment of this disclosure, as follows: Figure 5 As shown, the above-mentioned electronic device 500 may further include: The system includes a memory 510 and a processor 520, and a bus 530 connecting different components (including the memory 510 and the processor 520). The memory 510 stores a computer program, and when the processor 520 executes the program, it implements the database table field tracing method described in this embodiment of the disclosure.
[0097] Bus 530 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.
[0098] Electronic device 500 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 500, including volatile and non-volatile media, removable and non-removable media.
[0099] Memory 510 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 540 and / or cache memory 550. The server may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 560 can be used to read and write non-removable, non-volatile magnetic media (…). Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 530 via one or more data media interfaces. Memory 510 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.
[0100] A program / utility 580 having a set (at least one) of program modules 570 may be stored, for example, in memory 510. Such program modules 570 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 570 typically perform the functions and / or methods described in the embodiments of this disclosure.
[0101] Electronic device 500 can also communicate with one or more external devices 590 (e.g., keyboard, pointing device, display 591, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 592. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 593. As shown, network adapter 593 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0102] The processor 520 performs various functional applications and data processing by running programs stored in the memory 510.
[0103] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the database table field tracing method of this disclosure embodiment, and will not be repeated here.
[0104] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to perform the database table field tracing method proposed in any of the above embodiments. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0105] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the database table field tracing method proposed in any of the above embodiments.
[0106] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0107] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A method for tracing the origin of database table fields, characterized in that, include: Retrieve the SQL script to be analyzed from the database; Identify nested subqueries in the SQL script, replace the nested subqueries with temporary tables and establish references to the temporary tables in their original positions, while simultaneously moving the definition of the temporary tables to the front, thus transforming the SQL script into a linearized script that does not contain nested subqueries. Extract the referenced physical tables from the linearization script, and associate and store the physical tables with the linearization script to establish a table-level dependency index; For the data writing operation in the linearization script, the mapping relationship between the target field and the source field of the data writing operation is parsed, the filtering condition information in the data writing operation is extracted, and the mapping relationship and the filtering condition information are associated and stored to construct a field-level lineage database. Based on the field-level lineage database and the table-level dependency index, the target field is traced to obtain the lineage link from the target field to the field in the original base table, wherein the original base table is a physical table that does not depend on other fields for processing.
2. The method according to claim 1, characterized in that, The process of obtaining the SQL script to be analyzed from the database includes: Retrieve a multi-source SQL script set from the database; Each SQL script in the SQL script set is preprocessed to obtain a standardized SQL script to be analyzed. The standardization preprocessing includes: Each SQL script is divided into independent SQL statements or stored procedures according to logical units, and a unique script identifier is assigned to each logical unit. Remove comments and blank lines from each logical unit; Standardize the capitalization of keywords in all logical units; Expand and replace macro definitions or variables in each logic unit.
3. The method according to claim 1, characterized in that, The process of identifying nested subqueries in the SQL script, replacing the nested subqueries with temporary tables and establishing references to the temporary tables in their original positions, while simultaneously moving the definition of the temporary tables to the front, transforming the SQL script into a linearized script that does not contain nested subqueries, includes: The SQL script is parsed into an abstract syntax tree, and the original nesting position of the nested subqueries is identified based on the abstract syntax tree; Based on the original nested position, extract the nested subqueries and generate a unique temporary table name for each extracted nested subquery; Replace the nested subquery with the table name of the temporary table at the original nested position to form a reference to the temporary table, and at the same time generate the definition statement of the temporary table; The generated definition statement is placed before the main query that references the temporary table, thus converting the nested execution logic into sequential execution logic. By iterating through all nested subqueries in the SQL script and performing the replacement, generation definition statement, and preceding definition statement operations on each nested subquery, a linearized script without nested subqueries is obtained.
4. The method according to claim 1, characterized in that, The step of extracting the referenced physical table from the linearization script and associating the physical table with the linearization script for storage to establish a table-level dependency index includes: The linearized script is parsed to identify and extract the database object names after excluding SQL keywords and system function names, which are used as the physical tables referenced by the logical units corresponding to the linearized script. Obtain the script identifier of the logical unit, associate each extracted physical table with the script identifier, and generate a correspondence record between each physical table and the logical unit; The corresponding relationship records are stored in the first relationship table to form a table-level dependency index with the script identifier as the dimension.
5. The method according to claim 1, characterized in that, The process for handling data write operations in the linearization script involves parsing the mapping relationship between the target field and the source field of the data write operation, extracting the filtering condition information from the data write operation, and associating and storing the mapping relationship and the filtering condition information to construct a field-level lineage database, including: Locate each data write operation in the linearization script and determine the target field list of the target table corresponding to each data write operation; Parse the source field list for each data write operation, and match each target field in the target field list and the source field list with the corresponding source field one by one to establish a mapping relationship between the target field and the source field; the mapping relationship includes: direct mapping where the source field is directly assigned to the target field, and derived mapping where the source field is assigned to the target field after function operation, arithmetic operation or string concatenation; Extract the source table information and filter condition information from each data write operation; The mapping relationship between each target field and the source field is stored in the second relation table, and the corresponding source table information and filter condition information are stored in the third relation table. The second relation table and the third relation table are associated with the script identifier and source table information of the logical unit corresponding to the same linearized script, so that each mapping relationship can be associated with the corresponding source table information and filter condition information, thus completing the associated storage of the mapping relationship and the filter condition information to construct a field-level lineage relationship library.
6. The method according to claim 1, characterized in that, The step of tracing the target field based on the field-level lineage database and the table-level dependency index to obtain the lineage link from the target field to the fields in the original base table includes: Starting with the target field, query the field-level lineage database to obtain the mapping relationship corresponding to the target field, and determine the direct source field of the target field based on the mapping relationship; Determine the type of the direct source field: If the direct source field is a field of a physical table, query the table-level dependent index. When it is confirmed that the physical table is the original base table, use the direct source field as a field of the original base table, and use the currently traced fields to form a lineage link from the target field to the field of the original base table. If the direct source field is a field of a temporary table, obtain the definition statement of the temporary table, use the definition statement as a new SQL script to continue tracing upwards until the original base table field is reached, and use the currently traced fields to form a lineage link from the target field to the original base table field.
7. The method according to claim 1, characterized in that, After constructing the lineage link from the target field to the fields in the original base table, the following is also included: A directed acyclic graph is constructed using the mapping relationships in the field-level lineage database as edges and the target and source fields involved in the mapping relationships as nodes. In response to a query request for a target field, the directed acyclic graph is traversed, and the lineage links from the target field to the fields in the original base table are queried and displayed in the directed acyclic graph.
8. The method according to claim 7, characterized in that, The traversal of the directed acyclic graph includes: Use recursive common table expressions or consecutive left join queries from a relational database; or, A directed edge traversal algorithm for graph databases is adopted.
9. A database table field tracing device, characterized in that, include: The acquisition module is used to acquire the SQL script to be analyzed from the database; The conversion module is used to identify nested subqueries in the SQL script, replace the nested subqueries with temporary tables and establish references to the temporary tables in the original positions, and simultaneously move the definition of the temporary tables to the front, so that the SQL script is converted into a linearized script that does not contain nested subqueries. The first storage module is used to extract the referenced physical table from the linearization script, and store the physical table in association with the linearization script to establish a table-level dependency index. The second storage module is used to parse the mapping relationship between the target field and the source field of the data writing operation in the linearization script, extract the filtering condition information in the data writing operation, and store the mapping relationship and the filtering condition information together to build a field-level lineage database. The tracing module is used to trace the source of a target field based on the field-level lineage database and the table-level dependency index to obtain the lineage link from the target field to the field in the original base table, wherein the original base table is a physical table that does not depend on other fields for processing.
10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.
11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.
12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.
