A data weight quantification method and system based on operator-level consanguinity analysis

Abstract

The application discloses a data weight quantification method and system based on operator-level bloodline analysis. The method parses a target scene corresponding to a to-be-analyzed SQL statement into an abstract syntax tree; traverses the abstract syntax tree, extracts various operators, and constructs a dependency relationship among the various operators; constructs an operator directed acyclic graph based on the various operators and the dependency relationship; constructs an operator-level bloodline storage model containing operator nodes, table nodes and an association relationship among the nodes based on the operator directed acyclic graph, wherein the attribute of the association relationship contains an edge weight value, and the edge weight value is obtained by weighted summation of a data flow weight, an operator complexity weight, a dependency relationship weight and a path depth weight; and the edge weight value is normalized to obtain a normalized edge weight value, so as to realize data weight quantification. The application can provide fine-grained and quantifiable bloodline analysis support for data quality traceability in data governance and key operator positioning in query optimization.

Description

Technical Field

[0001] This application relates to the field of data governance technology, and in particular to a data weight quantification method and system based on operator-level lineage analysis. Background Technology

[0002] Data lineage refers to the relationships formed during the entire lifecycle of data, from its generation, processing, and circulation. It records the source, transformation process, and dependencies of the data. Traditional data lineage analysis is mainly divided into two categories: table-level lineage and field-level lineage. The lineage relationships between data are constructed by parsing SQL statements, ETL scripts, or data processing jobs.

[0003] However, existing methods have significant limitations. First, in terms of analytical granularity, they cannot reveal the true transformation logic of data within SQL. Although the database query optimizer generates internal, operator-based execution plans (such as PROJECT, JOIN, FILTER, and AGGREGATE) when executing SQL, these plans are transient and serve query execution; their information is not persisted or modeled, preventing data governance personnel from accessing and utilizing this fine-grained processing information. Second, in terms of relational representation, existing lineages are mostly qualitative descriptions, lacking quantitative evaluation mechanisms. This makes it impossible to objectively determine which data processing stage has the greatest impact on downstream processes, or which operator is a critical propagation node for data quality issues when facing complex data chains. Finally, in terms of quantitative attempts, existing methods typically rely on a single indicator (such as the number of data rows), failing to comprehensively consider multi-dimensional factors such as the dynamic changes in data flow, the computational complexity of the operator itself, the centrality of the operator in the entire data flow topology, and the complexity of the processing path.

[0004] Therefore, there is an urgent need for a technology that can transform the operator-level execution logic within a database into a persistent, queryable, and quantifiable data governance model to solve the technical problems of coarse granularity and lack of effective quantitative evaluation in lineage analysis. Summary of the Invention

[0005] This application aims to propose a data weight quantification method and system based on operator-level lineage analysis, which can provide fine-grained and quantifiable lineage analysis support for data quality tracing and key operator positioning in query optimization in data governance.

[0006] In a first aspect, embodiments of this application provide a data weight quantization method based on operator-level lineage analysis, the method comprising:

[0007] Parse the SQL statement to be parsed corresponding to the target scenario into an abstract syntax tree;

[0008] Traverse the abstract syntax tree, extract various operators, and construct the dependencies between the various operators;

[0009] Based on the various operators and the dependencies, an operator directed acyclic graph is constructed;

[0010] Based on the operator directed acyclic graph, an operator-level lineage storage model is constructed, which includes operator nodes, table nodes, and the relationships between nodes. The attributes of the relationships include edge weight values, which are obtained by weighted summation of data flow weight, operator complexity weight, dependency weight, and path depth weight.

[0011] The edge weight values ​​are normalized to obtain normalized edge weight values, thereby realizing data weight quantization.

[0012] Compared with the prior art, the first aspect of this application has the following beneficial effects:

[0013] This method parses the SQL statement to be parsed for the target scenario into an abstract syntax tree; traverses the abstract syntax tree, extracts various operators, and constructs dependencies between these operators; based on these operators and dependencies, it constructs an operator-directed acyclic graph (DAG); based on the DAG, it constructs an operator-level lineage storage model containing operator nodes, table nodes, and the relationships between nodes. The attributes of these relationships include edge weights, which are obtained by weighted summation of data flow weights, operator complexity weights, dependency weights, and path depth weights; the edge weights are then normalized to obtain normalized edge weight values, thus achieving data weight quantification. In this way, by constructing dependencies between operators, building an operator-directed acyclic graph, and constructing an operator-level lineage storage model, operator information can be persisted or modeled, enabling data governance personnel to access and utilize this fine-grained processing information; by comprehensively considering data flow weights, operator complexity weights, dependency weights, and path depth weights to calculate edge weight values, the lineage relationships between operators can be effectively quantified and evaluated. Therefore, this method can provide fine-grained and quantifiable lineage analysis support for data quality tracing and key operator location in query optimization in data governance.

[0014] In some implementations, constructing an operator-directed acyclic graph based on the multiple operators and the dependencies includes:

[0015] Initialize an empty list of operators and an empty dependency graph, wherein the dependency graph is in the form of an adjacency list, the key of the adjacency list is the unique ID of the operator, and the value of the adjacency list is a list of predecessor operator IDs of the operator.

[0016] Each operator is added to the operator list, and the unique ID of the operator that is dependent on the current operator is added to the predecessor operator ID list in the dependency graph to construct the initial operator directed acyclic graph.

[0017] Cycle detection is performed on the initial operator directed acyclic graph. If a cycle is detected in the initial operator directed acyclic graph, the dependency relationship between the current operator pairs connected to the cycle is corrected to obtain the corrected dependency relationship. The unique ID of the operator corresponding to the corrected dependency relationship is added to the predecessor operator ID list. When there is no cycle in the initial operator directed acyclic graph, the operator directed acyclic graph is constructed.

[0018] In some implementations, the step of constructing an operator-level lineage storage model based on the operator-directed acyclic graph, which includes operator nodes, table nodes, and the relationships between nodes, includes:

[0019] Based on the operator directed acyclic graph, operator nodes and table nodes are constructed. The attributes of the operator node include a unique identifier for the operator, an enumeration value of the operator type, the SQL fragment of the operator, a list of input data sources, an output field mapping, a complexity score, and an execution order. The attributes of the table node include a unique identifier for the table, the full name of the table, and a list of table fields.

[0020] Based on the operator directed acyclic graph, the association relationships between nodes are constructed. The association relationships include the dependency relationships between operator nodes, the data flow relationships between table nodes and operator nodes, and the data flow relationships between operator nodes and table nodes.

[0021] A single attribute index is established on the operator type enumeration value, complexity score, and execution order of the operator node;

[0022] Based on the operator nodes, the table nodes, the relationships between the nodes, and the single attribute index, an operator-level lineage storage model is constructed.

[0023] In some implementations, the edge weight value is obtained by weighted summation of data flow weight, operator complexity weight, dependency weight, and path depth weight, including:

[0024] Calculate the data flow weight based on the number of rows of operator output data and the number of rows of operator input data.

[0025] A complexity score is preset for each operator, and the complexity score is used as the operator complexity weight;

[0026] The importance score of each operator in the operator directed acyclic graph is calculated using PageRank, and the dependency weights are calculated based on the importance scores.

[0027] The path depth weight is calculated based on the number of upstream tables and downstream tables corresponding to each operator. The number of upstream tables is the number of tables from the input table to the operator, and the number of downstream tables is the number of tables from the output table to the output table.

[0028] The edge weight value is obtained by weighting and summing the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight.

[0029] In some implementations, calculating the data flow weight based on the number of operator output data rows and the number of operator input data rows includes:

[0030] For data reduction operators, the data flow weight corresponding to the data reduction operator is calculated based on the number of output data rows and the number of input data rows of the operator.

[0031] For data reconstruction operators, the data flow weight corresponding to the data reconstruction operator is calculated based on the number of left input data rows, right input data rows, and output data rows of the data reduction operator.

[0032] In some implementations, calculating the path depth weight based on the number of upstream and downstream tables corresponding to each operator includes:

[0033] Get the number of upstream and downstream tables connected to each operator;

[0034] Add the number of upstream tables and the number of downstream tables to get the sum;

[0035] The summation result is multiplied by a preset coefficient to obtain the path depth weight.

[0036] In some implementations, the step of weighted summing of the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight to obtain the edge weight value includes:

[0037] Obtain the weight coefficients corresponding to the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight;

[0038] The data flow weight, the operator complexity weight, the dependency weight, and the path depth weight are used to construct a feature vector;

[0039] The feature vector is input into the trained logistic regression model to obtain the target weight coefficients;

[0040] Based on the target weight coefficient, the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight are weighted and summed to obtain the edge weight value.

[0041] Secondly, embodiments of this application also provide a data weight quantization system based on operator-level lineage analysis, the system comprising:

[0042] The data parsing unit is used to parse the SQL statement to be parsed corresponding to the target scenario into an abstract syntax tree;

[0043] The operator extraction unit is used to traverse the abstract syntax tree, extract various operators, and construct the dependency relationships between the various operators;

[0044] The first construction unit is used to construct an operator directed acyclic graph based on the multiple operators and the dependencies.

[0045] The second construction unit is used to construct an operator-level lineage storage model based on the operator directed acyclic graph, which includes operator nodes, table nodes and the relationships between nodes. The attributes of the relationships include edge weight values, which are obtained by weighted summation of data flow weight, operator complexity weight, dependency weight and path depth weight.

[0046] The weight quantization unit is used to normalize the edge weight values ​​to obtain normalized edge weight values, thereby realizing data weight quantization.

[0047] Thirdly, embodiments of this application also provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to execute a data weight quantization method based on operator-level lineage analysis as described above.

[0048] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a data weight quantization method based on operator-level lineage analysis as described above.

[0049] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0050] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0051] Figure 1 This is a flowchart illustrating an embodiment of the data weight quantization method based on operator-level lineage analysis provided in this application;

[0052] Figure 2 This is a schematic diagram of the operator parsing process in the best embodiment of the data weight quantization method based on operator-level lineage parsing provided in this application;

[0053] Figure 3 This is a schematic diagram of the weight calculation process in the best embodiment of the data weight quantization method based on operator-level lineage analysis provided in this application;

[0054] Figure 4 This is a schematic diagram of the SQL table structure for marine meteorological element fusion analysis in the best embodiment of the data weight quantization method based on operator-level lineage analysis provided in this application;

[0055] Figure 5 This is a schematic diagram of the structure of an embodiment of the data weight quantization system based on operator-level lineage analysis provided in this application;

[0056] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0057] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0058] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0059] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0060] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0061] Existing methods have significant limitations. First, in terms of analytical granularity, they fail to reveal the true transformation logic of data within SQL. Although the database query optimizer generates internal, operator-based execution plans (such as PROJECT, JOIN, FILTER, and AGGREGATE) when executing SQL, these plans are transient and serve query execution; their information is not persisted or modeled, preventing data governance personnel from accessing and utilizing this fine-grained processing information. Second, in terms of relational representation, existing lineages are mostly qualitative descriptions, lacking quantitative evaluation mechanisms. This makes it impossible to objectively determine which data processing stage has the greatest impact on downstream processes, or which operator is a critical propagation node for data quality issues when facing complex data chains. Finally, in terms of quantitative attempts, existing methods typically rely on a single metric (such as the number of data rows), failing to comprehensively consider multi-dimensional factors such as the dynamic changes in data flow, the computational complexity of the operator itself, the centrality of the operator in the entire data flow topology, and the complexity of the processing path.

[0062] To address the problems existing in the prior art, this application proposes a data weight quantization method and system based on operator-level lineage analysis.

[0063] Reference Figure 1 This application provides a flowchart illustrating a data weight quantization method based on operator-level lineage analysis. This method is applied to electronic devices, such as servers or mobile terminals. Figure 1 As shown, the data weight quantization method based on operator-level lineage analysis may include the following steps:

[0064] Step S101: Parse the SQL statement to be parsed corresponding to the target scenario into an abstract syntax tree;

[0065] Step S102: Traverse the abstract syntax tree, extract various operators, and construct the dependency relationships between the various operators;

[0066] Step S103: Construct an operator directed acyclic graph based on multiple operators and dependencies;

[0067] Step S104: Based on the operator directed acyclic graph, construct an operator-level lineage storage model that includes operator nodes, table nodes, and the relationships between nodes. The attributes of the relationships include edge weight values, which are obtained by weighted summation of data flow weight, operator complexity weight, dependency weight, and path depth weight.

[0068] Step S105: Normalize the edge weight values ​​to obtain normalized edge weight values, thereby realizing data weight quantization.

[0069] In this embodiment, the SQL statement to be parsed corresponding to the target scenario is parsed into an abstract syntax tree; the abstract syntax tree is traversed to extract various operators and construct dependencies between them; based on the various operators and dependencies, an operator-directed acyclic graph (DAG) is constructed; based on the DAG, an operator-level lineage storage model is constructed, which includes operator nodes, table nodes, and the relationships between nodes. The attributes of the relationships include edge weight values, which are obtained by weighted summation of data flow weight, operator complexity weight, dependency weight, and path depth weight; the edge weight values ​​are normalized to obtain normalized edge weight values, thereby achieving data weight quantification. Thus, by constructing dependencies between operators, building an operator-directed acyclic graph, and constructing an operator-level lineage storage model, operator information can be persisted or modeled, enabling data governance personnel to access and utilize this fine-grained processing information; by comprehensively considering data flow weight, operator complexity weight, dependency weight, and path depth weight to calculate edge weight values, the lineage relationships between operators can be effectively quantified and evaluated. Therefore, this embodiment can provide fine-grained and quantifiable lineage analysis support for data quality tracing and key operator location in query optimization in data governance.

[0070] The above-mentioned parsing of the SQL statement to be parsed corresponding to the target scenario into an abstract syntax tree can be achieved by using the SQL parsing engine of the JSqlParser tool to parse the SQL statement to be parsed corresponding to the target scenario into an abstract syntax tree.

[0071] The target scenario mentioned above can be a marine meteorological data fusion and analysis scenario or other practical application scenarios. This embodiment does not make any specific limitations.

[0072] The above-mentioned traversal of the abstract syntax tree to extract various operators can be achieved by traversing the abstract syntax tree and parsing the filtering conditions, JOIN operations, projection fields, GROUP BY clauses, and aggregate functions in the FROM clause and WHERE clause to extract various operators.

[0073] The above-described operator-directed acyclic graph (AAG) is constructed based on various operators and dependencies. It can be a directed graph without loops, built upon the dependencies between operators. The algorithm for constructing operator-directed AAGs follows the execution order of the SQL standard and adopts a bottom-up construction strategy to ensure that the graph structure is consistent with the actual SQL execution logic.

[0074] In some implementations, an operator-directed acyclic graph is constructed based on multiple operators and dependencies, including:

[0075] Initialize an empty list of operators and an empty dependency graph. The dependency graph is in the form of an adjacency list, where the key of the adjacency list is the unique ID of the operator, and the value of the adjacency list is a list of the IDs of the predecessor operators of the operator.

[0076] Each operator is added to the operator list, and the unique ID of the operator that is dependent on the current operator is added to the predecessor operator ID list in the dependency graph to construct the initial operator directed acyclic graph.

[0077] Cycle detection is performed on the initial operator directed acyclic graph. If a cycle is detected in the initial operator directed acyclic graph, the dependency relationship between the current operator pairs connected to the cycle is corrected to obtain the corrected dependency relationship. The unique ID of the operator corresponding to the corrected dependency relationship is added to the predecessor operator ID list. When there is no cycle in the initial operator directed acyclic graph, the operator directed acyclic graph is constructed.

[0078] In this embodiment, constructing an operator-directed acyclic graph lays a solid data foundation for the subsequent construction of an operator-level lineage storage model, thereby supporting the complete storage and efficient querying of operator-level lineage information.

[0079] The aforementioned cycle detection of the initial operator directed acyclic graph can be performed using a depth-first search method.

[0080] The depth-first search algorithm described above can be an algorithm used to traverse or search a tree or graph. It traverses the nodes of the tree along its depth, searching the branches as deeply as possible. When all edges containing a node have been explored, or when a node no longer meets certain conditions during the search, the search backtracks to the starting node of the edge from which the node was found. This process is repeated until all nodes have been visited.

[0081] In some implementations, an operator-level lineage storage model is constructed based on an operator-directed acyclic graph, including operator nodes, table nodes, and the relationships between nodes, comprising:

[0082] Based on the operator directed acyclic graph, operator nodes and table nodes are constructed. The attributes of the operator node include the operator unique identifier, operator type enumeration value, operator SQL fragment, input data source list, output field mapping, complexity score and execution order. The attributes of the table node include the table unique identifier, table full name and table field list.

[0083] Based on the operator-directed acyclic graph, the association relationships between nodes are constructed. These relationships include the dependency relationships between operator nodes, the data flow relationships between table nodes and operator nodes, and the data flow relationships between operator nodes and table nodes.

[0084] Establish a single attribute index on the operator type enumeration value, complexity score, and execution order of the operator node;

[0085] Based on operator nodes, table nodes, the relationships between nodes, and single attribute indexes, an operator-level lineage storage model is constructed.

[0086] In this embodiment, by constructing an operator-level lineage storage model that includes operator nodes, table nodes, and the relationships between nodes, operator information can be persisted or modeled, enabling data governance personnel to access and utilize this fine-grained processing information.

[0087] The unique identifier of the aforementioned operator can be ensured by using the SQL statement MD5 hash value - the operator's sequence number in the parsing order.

[0088] The SQL fragments of the above operators can be, for example, the content of the WHERE clause corresponding to the FILTER operator, or the content of the JOIN clause corresponding to the JOIN operator.

[0089] The above list of input data sources can record information about the tables or operators that the upstream depends on.

[0090] The above output field mapping can record the correspondence between operator output fields and upstream input fields.

[0091] The above complexity score can be preset according to the operator type.

[0092] The above execution order can be the sequence number of the operators in the SQL execution flow, starting from 1 and incrementing.

[0093] The unique identifier for the above table can be an MD5 hash value named three levels from catalog.schema.table.

[0094] The table field list above can record all field names and data types.

[0095] In some implementations, the edge weight is obtained by weighted summation of data flow weight, operator complexity weight, dependency weight, and path depth weight, including:

[0096] Calculate the data flow weight based on the number of rows of operator output data and the number of rows of operator input data.

[0097] A complexity score is preset for each operator, and the complexity score is used as the operator complexity weight;

[0098] The importance score of each operator in the operator directed acyclic graph is calculated using PageRank, and the dependency weights are calculated based on the importance scores.

[0099] The path depth weight is calculated based on the number of upstream tables and downstream tables corresponding to each operator. The number of upstream tables is the number of tables from the input table to the operator, and the number of downstream tables is the number of tables from the output table to the output table.

[0100] The edge weight value is obtained by weighting and summing the data flow weight, operator complexity weight, dependency weight, and path depth weight.

[0101] In this embodiment, edge weight values ​​are calculated by comprehensively considering data flow weight, operator complexity weight, dependency weight, and path depth weight, which enables effective quantitative evaluation of the lineage relationship between operators.

[0102] The complexity score of each operator mentioned above can be determined based on the inherent complexity of the operator type, reflecting the difficulty of the operator processing logic and the level of resource consumption. Based on the operator execution cost in database query optimization theory, a set of operator complexity scoring tables for the complexity of lineage quantification analysis was designed.

[0103] The aforementioned database query optimization theory can be considered a system of methods to improve database performance. It mainly involves reducing data redundancy and improving query efficiency and storage resource utilization by rationally designing table structures and optimizing indexes and query statements.

[0104] The PageRank algorithm mentioned above is a proprietary Google algorithm used to measure the importance of a specific webpage relative to other webpages in the search engine index. In this embodiment, it is used to calculate the importance score of each operator in the operator directed acyclic graph.

[0105] In some implementations, the data flow weight is calculated based on the number of operator output data rows and the number of operator input data rows, including:

[0106] For data reduction operators, the data flow weight corresponding to the data reduction operator is calculated based on the number of output data rows and the number of input data rows of the operator.

[0107] For data reconstruction operators, the data flow weight corresponding to the data reconstruction operator is calculated based on the number of left input data rows, right input data rows, and output data rows of the data reduction operator.

[0108] In this embodiment, by calculating the data flow weight based on the number of operator output data rows and the number of operator input data rows, the influence of the operator on the amount of data can be reflected, which can be used to quantify the contribution of data compression / expansion to the importance of lineage.

[0109] The aforementioned data reduction operators can include the FILTER operator and the GROUP operator.

[0110] The aforementioned data reconstruction operators can include JOIN operators.

[0111] In some implementations, the path depth weight is calculated based on the number of upstream and downstream tables corresponding to each operator, including:

[0112] Get the number of upstream and downstream tables connected to each operator;

[0113] Add the number of upstream tables and the number of downstream tables to get the sum;

[0114] The summation result is multiplied by a preset coefficient to obtain the path depth weight.

[0115] In this embodiment, by calculating the path depth weight based on the number of upstream and downstream tables corresponding to each operator, the complexity of the lineage relationship can be reflected, laying a good data foundation for the effective quantitative evaluation of the lineage relationship between operators.

[0116] The aforementioned preset coefficients can be pre-set coefficients and can be changed according to actual conditions. This embodiment does not impose specific limitations.

[0117] In some implementations, the edge weight value is obtained by weighted summation of data flow weight, operator complexity weight, dependency weight, and path depth weight, including:

[0118] Obtain the weight coefficients corresponding to data flow weight, operator complexity weight, dependency weight, and path depth weight;

[0119] The data flow weight, operator complexity weight, dependency weight, and path depth weight are used to construct a feature vector;

[0120] The feature vectors are input into the trained logistic regression model to obtain the target weight coefficients;

[0121] Based on the target weight coefficient, the weights of data flow, operator complexity, dependency, and path depth are summed in a weighted manner to obtain the edge weight value.

[0122] In this embodiment, by constructing a feature vector from data traffic weight, operator complexity weight, dependency weight, and path depth weight, and then inputting the feature vector into a trained logistic regression model, the target weight coefficient is obtained. This allows for adaptive optimization of the weight coefficient, enabling dynamic adjustment of the weight to adapt to different business scenarios. Furthermore, by comprehensively considering data traffic weight, operator complexity weight, dependency weight, and path depth weight to calculate the edge weight value, the lineage relationship between operators can be effectively quantified and evaluated.

[0123] The trained logistic regression model described above can be based on a pre-constructed loss function and gradient descent algorithm.

[0124] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:

[0125] This embodiment addresses the technical problems in existing technologies, such as the inability to utilize SQL's internal operator execution logic for data governance and the lack of an effective quantitative evaluation model for lineage relationships. This embodiment explicitly and persistently presents operator execution plans and innovatively constructs a multi-dimensional, adaptive weight quantification model, providing precise and quantifiable decision-making basis for data quality tracing, critical link identification, and query optimization.

[0126] This embodiment of the method deeply analyzes SQL statements to identify and extract PROJECT, JOIN, FILTER, GROUP operators and their execution order, constructing an operator dependency graph. It uses the Neo4j graph database to store operator nodes and table nodes, designing a three-layer naming format: catalog.schema.table. It innovatively establishes a four-dimensional weight calculation model that integrates data flow impact, operator computational complexity, topological dependency importance (i.e., dependency relationship), and path complexity (i.e., path depth), and combines this with an improved PageRank algorithm for data governance scenarios to effectively quantify and evaluate the lineage relationships between operators. Finally, it provides a machine learning-based adaptive optimization mechanism for weight coefficients, enabling the model to learn from historical data and dynamically adjust to adapt to different business scenarios.

[0127] The purpose of this embodiment is to provide a data weight quantification method and system based on operator-level lineage analysis, to solve the technical problems of coarse granularity of lineage analysis, lack of quantitative evaluation mechanism, and single dimension of weight calculation in the prior art. The coarse granularity of lineage analysis is reflected in its coverage only at the table and field levels, failing to parse operators such as PROJECT, JOIN, FILTER, and GROUP in the SQL execution process; the lack of a quantitative evaluation mechanism is reflected in its qualitative description of lineage relationships, lacking means to assess the importance of lineage; and the single dimension of weight calculation is reflected in its reliance on a single indicator, without integrating multi-dimensional factors. This embodiment can provide fine-grained and quantifiable lineage analysis support for data quality tracing in data governance and key operator location in query optimization.

[0128] To achieve the above objectives, the technical solution of this embodiment specifically includes the following steps:

[0129] Figure 2 This diagram illustrates the operator parsing process, showcasing the entire process of a SQL statement from parsing to final storage. It covers the core processing steps: First, the SQL statement (the SQL statement to be parsed in the target scenario) is parsed into an Abstract Syntax Tree (AST). Then, operator identification and extraction are performed based on the AST. Next, a Directed Acyclic Graph (DAG) of operators is constructed. Finally, based on the DAG, an extended operator-level lineage storage model based on Neo4j is built.

[0130] Step S1: The SQL statement is parsed into an Abstract Syntax Tree (AST).

[0131] This step uses the JSqlParser framework to parse the input SQL statement into an abstract syntax tree, providing a structured syntactic representation for subsequent operator recognition. Specifically, it includes:

[0132] 1. SQL statement preprocessing: Perform syntax checking and formatting on the input SQL statement.

[0133] 2. AST Generation: The SQL string is converted into a syntax tree object by the SQL parsing engine of the JSqlParser tool, forming a complete abstract syntax tree structure.

[0134] Step S2: Operator identification and extraction based on AST.

[0135] This step involves a depth-first traversal of the abstract syntax tree to identify and extract various operators from the SQL statement, establishing a mapping between syntax tree nodes and operator types. Specifically, this includes:

[0136] 1. Create the table scan operator TABLE_SCAN: First, parse the FROM clause to identify the table references, which include base tables and views. Create a unique TABLE_SCAN operator for each table reference. The operator attributes record the three levels of naming information of the table: catalog, schema, and table.

[0137] 2. Create the FILTER operator: Parse the filtering conditions in the WHERE clause. The filtering conditions include comparison conditions and logical conditions. Create the FILTER operator, set the predecessor dependency of the operator to the corresponding TABLE_SCAN operator, and record the complete filtering condition expression in the operator attributes.

[0138] 3. Create the JOIN operator: Identify the JOIN operation in the SQL statement. The JOIN operation includes INNER JOIN, LEFT JOIN, and RIGHT JOIN. Based on the connection type identifier after the JOIN keyword, create the corresponding type of JOIN operator. Set the TABLE_SCAN or FILTER operator participating in the connection as the predecessor dependency of the JOIN operator. The operator attributes record the connection conditions and the three-level naming information of the connected tables. The connection conditions are the content of the ON clause.

[0139] 4. Create GROUP aggregate operator: Parse the GROUP BY clause and aggregate functions such as SUM, COUNT and AVG, create GROUP operator, set JOIN operator or FILTER operator as the predecessor dependency of GROUP operator, and record grouping field, aggregate function type and aggregate field information in operator attributes.

[0140] 5. Create a PROJECT operator: Process the projected fields in the SELECT clause. Projected fields include original fields, alias fields, and calculated fields. Create a PROJECT operator and set the GROUP operator, JOIN operator, or FILTER operator as the predecessor dependency of the PROJECT operator. The operator properties record the mapping relationship between the projected fields and the output fields of the upstream operators.

[0141] 6. Create the SORT operator: If the SQL contains an ORDER BY clause, parse the sorting field and sorting direction. If the sorting direction is ASC or DESC, create the SORT operator. Set the PROJECT operator or GROUP operator as the predecessor dependency of the SORT operator. Record the sorting field and direction information in the operator properties.

[0142] Step S3: Construct the operator directed acyclic graph (DAG).

[0143] The operator-based directed acyclic graph construction algorithm in this step follows the execution order of the SQL standard and adopts a bottom-up construction strategy to ensure that the graph structure is consistent with the actual SQL execution logic. Specifically, it includes:

[0144] 1. Initialization phase: Create an empty list of operators and a dependency graph data structure. The list of operators is used to store all the parsed operators. The dependency graph is in the form of an adjacency list. The key of the adjacency list is the unique ID of the operator, and the value is a list of the IDs of the predecessor operators of that operator.

[0145] 2. Table Scan Phase: Add all parsed TABLE_SCAN operators to the operator list. In the dependency graph, initialize an empty predecessor operator ID list for each TABLE_SCAN operator. The TABLE_SCAN operator has no upstream dependencies.

[0146] 3. Filtering phase: Add the parsed FILTER operators to the operator list, and add the corresponding TABLE_SCAN operator unique ID to the predecessor operator ID list for each FILTER operator in the dependency graph.

[0147] 4. Joining Phase: Add the parsed JOIN operators to the operator list, and add the unique ID of the TABLE_SCAN operator or the unique ID of the FILTER operator participating in the join to the predecessor operator ID list for each JOIN operator in the dependency graph.

[0148] 5. Grouping and Aggregation Phase: Add the parsed GROUP operators to the operator list, and add the corresponding unique JOIN operator ID or FILTER operator ID to the predecessor operator ID list for each GROUP operator in the dependency graph; if there is an aggregation function, update the aggregation attribute of the GROUP operator synchronously.

[0149] 6. Projection phase: Add the resolved PROJECT operators to the operator list, and add the corresponding unique GROUP operator ID, JOIN operator ID, or FILTER operator ID to the predecessor operator ID list for each PROJECT operator in the dependency graph.

[0150] 7. Sort phase: If the SORT operator exists, add it to the operator list and add the corresponding PROJECT operator unique ID or GROUP operator unique ID to the predecessor operator ID list in the dependency graph.

[0151] 8. Verification Phase: After construction, cycle detection is performed on the directed acyclic graph of the operators using a depth-first search method. If a cycle is detected, the cyclically dependent operator pair is located, and the dependency relationship is corrected according to the SQL syntax logic to ensure that the graph structure is acyclic.

[0152] Step S4: Construct an extended operator-level lineage storage model based on Neo4j.

[0153] This step, to support complete storage and efficient querying of operator-level lineage information, constructs an extended operator-level lineage storage model based on Neo4j, using an operator-directed acyclic graph. The data model of the Neo4j graph database is extended, and the structure of operator nodes, table nodes, and relationships is designed as follows:

[0154] 1. Design the node structure.

[0155] (1) Design of operator nodes: The core attributes of operator nodes include the unique identifier of the operator, the enumeration value of the operator type, the corresponding SQL fragment, the list of input data sources, the output field mapping, the complexity score, and the execution order. Among them, the generation rule of the unique identifier of the operator is: the MD5 hash value of the SQL statement - the sequence number of the operator in the parsing order, to ensure global uniqueness; the enumeration value of the operator type includes TABLE_SCAN, FILTER, JOIN, GROUP, PROJECT, and SORT; the corresponding SQL fragments are such as the content of the WHERE clause corresponding to the FILTER operator and the content of the JOIN clause corresponding to the JOIN operator; the list of input data sources records the information of the tables or operators that the upstream depends on; the output field mapping records the correspondence between the output fields of the operator and the input fields of the upstream; the complexity score is preset according to the operator type, referring to the subsequent weight model; the execution order is the sequence number of the operator in the SQL execution flow, starting from 1 and increasing.

[0156] (2) Design table nodes: The core attributes of a table node include the table's unique identifier, the table's full name, and the table's field list. The table's unique identifier is the MD5 hash value of the three-level naming of catalog.schema.table; the table's full name is catalog.schema.table; and the table's field list records all field names and data types.

[0157] 2. Design the relationship structure.

[0158] (1) Design the FLOWS_TO_OPERATOR relationship: This relationship is used to represent the dependency relationship between operator nodes, with the source node being the upstream operator and the target node being the downstream operator; the relationship attributes include the data volume change rate, the estimated processing time, and the edge weight value (i.e., the weight calculated by the multidimensional weight calculation model in step S5). The data volume change rate is the ratio of the input data volume of the downstream operator to the output data volume of the upstream operator; the estimated processing time is calculated based on the operator type and data volume, such as JOIN operator estimated at 10ms / 10,000 rows of data; the edge weight value is initially 0, and is subsequently calculated and assigned through a multi-dimensional weight calculation model.

[0159] (2) Design the FEEDS_TO_OPERATOR relationship: This relationship is used to represent the data flow relationship from the table node to the operator node. The source node is the table node and the target node is the TABLE_SCAN operator. The relationship attributes include the data read timestamp, which is the time when the operator reads the table data.

[0160] (3) Design the OUTPUTS_TO_TABLE relationship: This relationship is used to represent the data flow relationship from the operator node to the table node. The source node is the PROJECT operator or SORT operator, the PROJECT operator or SORT operator is the final output operator, and the target node is the table node. The relationship attributes include the data write timestamp and the number of output data rows. The data write timestamp is the time when the operator writes data to the table.

[0161] 3. Design the index structure.

[0162] To improve query efficiency, single-attribute indexes are created for the operator type enumeration value, complexity score, and execution order attribute of the operator node. A range index is created for the edge weight value attribute of the FLOWS_TO_OPERATOR relationship, supporting fast filtering queries based on conditions such as edge weight value greater than MinVal and edge weight value between [MinVal, MaxVal]. MinVal is the lower limit of the weight value, used to filter low-weight lineage relationships, and MaxVal is the upper limit of the weight value, used to limit the upper bound of the weight value range query. It should be noted that MinVal and MaxVal can be user-preset values; this embodiment does not impose specific limitations.

[0163] Step S5: Construct a multidimensional weight calculation model.

[0164] The multi-dimensional weight calculation model proposed in this step comprehensively considers data flow weight. Operator complexity weights Dependency weights Path depth weight The importance of kinship among operators is quantified through weighted summation of four dimensions of features, referring to... Figure 3 The specific process is as follows:

[0165] 1. Determine the mathematical model for weight calculation.

[0166] The mathematical model for weight calculation is expressed as follows:

[0167] ;

[0168] in, , , and These are weighting coefficients, with initial default values ​​set to empirical values. , , and This ensures that the four dimensions contribute reasonably to the final weights, and that the total weights equal 1. However, in complex and dynamically changing business environments, manually configured coefficients may not achieve optimal results. Therefore, this embodiment provides a machine learning-based adaptive optimization method (see step S7) to automatically learn and generate optimal weight coefficients from historical data, thereby achieving intelligent and adaptive adjustment of the model. The final weight value (i.e. edge weight value) of the lineage relationship between operators is taken in the range of [0,1]. The larger the value, the higher the importance of the lineage relationship in the data flow.

[0169] 2. Calculate data flow weights : This reflects the degree of impact of the operator on the amount of data, and is used to quantify the contribution of data compression / expansion to the importance of lineage. The calculation rules are as follows:

[0170] (1) For data reduction operators: The number of output rows for these operators is usually less than or equal to the number of input rows, mainly including FILTER and GROUP BY. Their weights aim to quantify the strength of data filtering or aggregation. The calculation formula is:

[0171] ;

[0172] in, Output the number of data rows for the operator. Input the number of data rows for the operator, if If no data is found after filtering, then ,like That is, without data filtering or aggregation, then . and All data can be obtained from the SQL execution plan.

[0173] (2) For data reconstruction operators: These operators (mainly JOIN) reshape the data flow according to the connection logic. The relationship between the number of output rows and the number of input rows is uncertain and may increase, decrease or remain unchanged. Therefore, the weight calculation needs to consider both data addition and data filtering.

[0174] The calculation of the JOIN operator weights depends on estimates of the size of its input and output data, including the number of rows of the left input data ( ), right input the number of data rows ( ) and the number of rows of output data ( The above estimates are not obtained through real-time sampling, but rather from the statistical information system maintained by the database query optimizer during the execution plan generation phase. Therefore, they are defined as:

[0175] ;

[0176] in, The ratio reflects the change in the amount of data in a JOIN operation. This indicates an increase in the amount of output data; if This indicates that JOIN acts as a filter; if This indicates that the amount of data remains unchanged.

[0177] The data flow weight of JOIN is defined as follows:

[0178] ;

[0179] This function form allows the weights to change smoothly between (0,1), avoiding amplification in extreme cases, and without introducing additional constants.

[0180] 3. Determine the operator complexity weights : Based on the inherent complexity of operator types, reflecting the difficulty of operator processing logic and resource consumption level, and according to the operator execution cost in database query optimization theory, a set of operator complexity scoring tables for lineage quantification analysis was designed, as shown in Table 1. The specific scores are as follows: TABLE_SCAN operator is 0.1, only reading table data, with the simplest logic; PROJECT operator is 0.1, only handling field projection, with simple logic; FILTER operator is 0.3, requiring parsing of filter conditions, with medium logic; SORT operator is 0.5, requiring the execution of sorting algorithms, with medium resource consumption; GROUP operator is 0.7, requiring the execution of grouping and aggregation calculations, with complex logic and high resource consumption; JOIN operator is 0.8, requiring the handling of table join logic, with complex data association and high resource consumption. Take the score of the corresponding operator type directly from the operator complexity rating table.

[0181] Table 1 shows the operator complexity scoring table.

[0182]

[0183] 4. Calculate dependency weights : The PageRank algorithm is used to calculate the topological importance of operators in the lineage graph (i.e., the directed acyclic graph of operators), reflecting the coreness of operators in the lineage propagation path. The calculation process is as follows:

[0184] (1) Initialize importance score: Let the total number of operator nodes in the lineage graph be . Initial importance score for each operator node .

[0185] (2) The importance score for iterative calculation is:

[0186] ;

[0187] in, For the first In the next iteration, the upstream operator Importance rating For the number of iterations, The damping coefficient is set to 0.85 (this can be changed according to the actual situation, and this embodiment does not impose a specific limitation), referring to the standard PageRank algorithm; For the first In the next iteration, the upstream operator Importance rating; upstream operator The out-degree, i.e. the number of downstream dependent operators.

[0188] (3) Determine the iteration termination condition: When the absolute value of the difference in importance scores of all operator nodes in two adjacent iterations is less than 1. Stop iterating when the time is right.

[0189] (4) Calculation All operator nodes (including upstream and downstream operators) in the lineage graph use the same PageRank iterative importance score calculation formula for importance score calculation; for any lineage relationship between operators, the product of the final importance score of its downstream operator and the final importance score of its upstream operator is taken as the dependency weight value of the lineage relationship.

[0190] 5. Calculate path depth weights : The complexity of blood relations is calculated based on the number of input and output tables involved in the operator; the calculation formula is as follows:

[0191] ;

[0192] in, This represents the number of upstream tables that the operator directly depends on (i.e., the number of input tables). For example, the JOIN operator joins two tables. It equals 2; This refers to the number of downstream tables (i.e., the number of output tables) that the operator directly outputs data to. For example, the PROJECT operator outputs to only one table. It equals 1; The value range is [0.1, 1.0]. When equal to 1, Equal to 0.1; when When it equals 10, It equals 1.0.

[0193] To avoid unlimited growth of path depth weights in complex queries (such as multi-level JOINs), this embodiment will... The maximum valid value for this sum is set to 10. When this sum exceeds 10, Fixed at 1.0.

[0194] Step S6: Normalization method.

[0195] This step ensures the reliability and comparability of the final weight values, uniformly mapping the weight values ​​of all FLOWS_TO_OPERATOR relationships to the range [0.1, 1.0]. The specific process is as follows:

[0196] 1. Determine global boundary values: Collect the initial weight values ​​of all FLOWS_TO_OPERATOR relationships within the current analysis batch, obtained through a multidimensional weight calculation model. Traverse all Value, determining the global maximum value in this batch of data. and global minimum .

[0197] 2. Perform linear normalization calculation: for each initial weight value Apply the following minimization normalization formula to linearly map it to the target interval [0.1, 1.0].

[0198] ;

[0199] in, This represents the normalized weights. It is a single initial weight value to be normalized. It is the minimum value among all initial weight values ​​in this batch. It is the maximum value among all initial weight values ​​in this batch; if If all normal weight values ​​are equal, then... .

[0200] 3. Interval mapping stage: The results after segmented normalization are checked again to ensure that all weight values ​​are mapped to the interval [0.1, 1.0]. If there are values ​​outside this interval, the value is caused by calculation error. Values ​​less than 0.1 are set to 0.1, and values ​​greater than 1.0 are set to 1.0.

[0201] Step S7: Machine learning-based weight coefficient optimization method.

[0202] 1. Construct the training dataset.

[0203] (1) Constructing feature vectors: For each FLOWS_TO_OPERATOR relationship in the pedigree graph, extract its four-dimensional weight components calculated in step S5. , , , The feature vector that constitutes this sample .

[0204] (2) Collect label data: Through labeling, for each feature vector Assign a binary importance label .Label This indicates that the blood relationship has been proven to be significant. This indicates a normal range. Data sources for label 'a' include: data quality monitoring systems, where any data quality issues caused by this lineage path are tagged with 1; and query performance logs, where operator-related lineages identified as performance bottlenecks are tagged with 1.

[0205] 2. Define the loss function and optimization objective.

[0206] This step aims to establish a mathematical model to measure the current weighting coefficients. The accuracy of classifying training samples.

[0207] (1) Establishing a prediction model: Using a logistic regression model, for the input feature vector X and weight coefficients First, calculate the linear combination. Then through the Sigmoid function Convert it to a probability value It should be noted that the logistic regression model can be any model known to those skilled in the art, and will not be specifically described in this embodiment.

[0208] (2) Define the loss function: Use the log loss function to quantify the prediction probability. With real labels The difference between them. For a single sample, the loss function is... The optimization objective is to minimize the total loss over the entire training set.

[0209] 3. Perform gradient descent optimization.

[0210] This step involves iterative calculations to gradually adjust the weighting coefficients. The goal is to find the optimal solution that minimizes the total loss function.

[0211] (1) Calculate the gradient: Calculate the total loss function Regarding each weighting coefficient The partial derivatives are used to obtain the gradient vector. The gradient vector indicates the direction in which the loss function rises the fastest.

[0212] (2) Update parameters: Use the gradient descent algorithm to update the weight coefficients according to the opposite direction of the gradient: ,in For learning rate, This represents the weight coefficients before the update.

[0213] (3) Convergence determination: Repeat gradient calculation and parameter update until the total loss function converges to below the preset threshold, or the maximum number of iterations is reached. The final parameter vector obtained at this point... This is the optimal weighting coefficient (i.e., the target weighting coefficient). ).

[0214] This embodiment also provides specific implementation examples for practical application scenarios, including:

[0215] Taking marine meteorological data fusion analysis as an example, this embodiment illustrates its technical effectiveness in a practical application scenario. A marine monitoring center needs to perform correlation analysis on sea surface temperature and wind speed data to generate a marine environmental assessment report. The original SQL business logic is as follows: Figure 4 As shown, Figure 4 SQL table for fusion analysis of marine meteorological elements.

[0216] This embodiment uses an operator-level lineage method to parse the SQL table for marine meteorological element fusion analysis, establishing the weight relationship between the upstream data table and the downstream result table `marine_environment`. According to the multidimensional weight calculation model, the lineage weight from the `temperature_data` table to the `marine_environment` table is 0.54, and the lineage weight from the `wind_data` table to the `marine_environment` table is 0.46. This weight difference reflects the dominant role of temperature data in environmental index calculation.

[0217] Data quality checks revealed outliers in the `environment_index` values ​​for multiple sites in the `marine_environment` table. Traditional lineage analysis methods only indicate that the data in this table originates from the `temperature_data` and `wind_data` tables, requiring technicians to examine all data in both source tables simultaneously. This implementation sorts the data based on lineage weight: the `temperature_data` table (weight 0.54) has a greater weight than the `wind_data` table (weight 0.46). Technicians prioritize checking the `temperature_data` table, which has the highest weight. Through weight-driven lineage tracing, the problematic data was located in the `temperature_data` table, eliminating the need to simultaneously check the `wind_data` table. After correcting the data and re-executing the `CREATE TABLE` statement, the data quality in the `marine_environment` table returned to normal. This method demonstrates improved efficiency compared to traditional approaches.

[0218] This embodiment provides a scientific tracing path for data quality issues by quantifying the lineage weights between upstream data tables and downstream result tables. When anomalies are detected in downstream table data, the system guides the investigation of upstream data sources according to the lineage weights from high to low; higher weights indicate a greater impact on the results and should be checked first. This weight-based lineage tracing method significantly improves the efficiency and accuracy of data governance.

[0219] Reference Figure 5 This application also provides a data weight quantization system based on operator-level lineage analysis. The system includes a data parsing unit 501, an operator extraction unit 502, a first construction unit 503, a second construction unit 504, and a weight quantization unit 505, wherein:

[0220] Data parsing unit 501 is used to parse the SQL statement to be parsed corresponding to the target scenario into an abstract syntax tree;

[0221] The operator extraction unit 502 is used to traverse the abstract syntax tree, extract various operators, and build the dependency relationships between various operators;

[0222] The first building unit 503 is used to build an operator directed acyclic graph based on multiple operators and dependencies;

[0223] The second building unit 504 is used to build an operator-level lineage storage model based on the operator directed acyclic graph, which includes operator nodes, table nodes and the relationships between nodes. The attributes of the relationships include edge weight values, which are obtained by weighted summation of data flow weight, operator complexity weight, dependency weight and path depth weight.

[0224] The weight quantization unit 505 is used to normalize the edge weight values ​​to obtain normalized edge weight values, thereby realizing data weight quantization.

[0225] It should be noted that since the data weight quantization system based on operator-level lineage analysis in this embodiment is based on the same inventive concept as the data weight quantization method based on operator-level lineage analysis described above, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.

[0226] Reference Figure 6 This application also provides an electronic device, which includes:

[0227] At least one memory;

[0228] At least one processor;

[0229] At least one program;

[0230] The program is stored in memory, and the processor executes at least one program to implement the data weight quantization method based on operator-level lineage analysis described above in this disclosure.

[0231] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0232] The electronic devices according to embodiments of this application will now be described in detail.

[0233] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0234] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 using the data weight quantization method based on operator-level lineage analysis according to the embodiments of this disclosure.

[0235] The input / output interface 1800 is used to implement information input and output.

[0236] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0237] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0238] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0239] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described data weight quantization method based on operator-level lineage analysis.

[0240] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0241] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0242] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0243] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0244] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0245] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0246] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0247] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0248] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0249] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0250] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The embodiments of this application have been described in detail above with reference to the accompanying drawings, but this application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of this application.

[0251] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A data weight quantization method based on operator-level lineage analysis, characterized in that, The method includes: Parse the SQL statement to be parsed corresponding to the target scenario into an abstract syntax tree; Traverse the abstract syntax tree, extract various operators, and construct the dependencies between the various operators; Based on the various operators and the dependencies, an operator directed acyclic graph is constructed; Based on the directed acyclic graph of the operators, an operator-level lineage storage model is constructed, which includes operator nodes, table nodes, and the relationships between nodes. The attributes of the relationships include edge weight values, which are obtained by weighted summation of data flow weight, operator complexity weight, dependency weight, and path depth weight, including: Calculate the data flow weight based on the number of rows of operator output data and the number of rows of operator input data. A complexity score is preset for each operator, and the complexity score is used as the operator complexity weight; The importance score of each operator in the operator directed acyclic graph is calculated using PageRank, and the dependency weights are calculated based on the importance scores. The path depth weight is calculated based on the number of upstream tables and downstream tables corresponding to each operator. The number of upstream tables is the number of tables from the input table to the operator, and the number of downstream tables is the number of tables from the output table to the output table. The edge weight value is obtained by weighted summing of the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight. The edge weight values ​​are normalized to obtain normalized edge weight values, thereby realizing data weight quantization.

2. The data weight quantization method based on operator-level pedigree analysis according to claim 1, characterized in that, The construction of the operator directed acyclic graph based on the various operators and the dependencies includes: Initialize an empty list of operators and an empty dependency graph, wherein the dependency graph is in the form of an adjacency list, the key of the adjacency list is the unique ID of the operator, and the value of the adjacency list is a list of predecessor operator IDs of the operator. Each operator is added to the operator list, and the unique ID of the operator that is dependent on the current operator is added to the predecessor operator ID list in the dependency graph to construct the initial operator directed acyclic graph. Cycle detection is performed on the initial operator directed acyclic graph. If a cycle is detected in the initial operator directed acyclic graph, the dependency relationship between the current operator pairs connected to the cycle is corrected to obtain the corrected dependency relationship. The unique ID of the operator corresponding to the corrected dependency relationship is added to the predecessor operator ID list. When there is no cycle in the initial operator directed acyclic graph, the operator directed acyclic graph is constructed.

3. The data weight quantization method based on operator-level lineage analysis according to claim 1, characterized in that, The construction of an operator-level lineage storage model based on the operator-directed acyclic graph, including operator nodes, table nodes, and the relationships between nodes, includes: Based on the operator directed acyclic graph, operator nodes and table nodes are constructed. The attributes of the operator node include a unique identifier for the operator, an enumeration value of the operator type, the SQL fragment of the operator, a list of input data sources, an output field mapping, a complexity score, and an execution order. The attributes of the table node include a unique identifier for the table, the full name of the table, and a list of table fields. Based on the operator directed acyclic graph, the association relationships between nodes are constructed. The association relationships include the dependency relationships between operator nodes, the data flow relationships between table nodes and operator nodes, and the data flow relationships between operator nodes and table nodes. A single attribute index is established on the operator type enumeration value, complexity score, and execution order of the operator node; Based on the operator nodes, the table nodes, the relationships between the nodes, and the single attribute index, an operator-level lineage storage model is constructed.

4. The data weight quantization method based on operator-level lineage analysis according to claim 1, characterized in that, The step of calculating the data flow weight based on the number of operator output data rows and the number of operator input data rows includes: For data reduction operators, the data flow weight corresponding to the data reduction operator is calculated based on the number of output data rows and the number of input data rows of the operator. For data reconstruction operators, the data flow weight corresponding to the data reconstruction operator is calculated based on the number of left input data rows, right input data rows, and output data rows of the data reduction operator.

5. The data weight quantization method based on operator-level lineage analysis according to claim 1, characterized in that, The step of calculating the path depth weight based on the number of upstream and downstream tables corresponding to each operator includes: Get the number of upstream and downstream tables connected to each operator; Add the number of upstream tables and the number of downstream tables to get the sum; The summation result is multiplied by a preset coefficient to obtain the path depth weight.

6. The data weight quantization method based on operator-level lineage analysis according to claim 1, characterized in that, The step of weighting and summing the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight to obtain the edge weight value includes: Obtain the weight coefficients corresponding to the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight; The data flow weight, the operator complexity weight, the dependency weight, and the path depth weight are used to construct a feature vector; The feature vector is input into the trained logistic regression model to obtain the target weight coefficients; Based on the target weight coefficient, the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight are weighted and summed to obtain the edge weight value.

7. A data weight quantization system based on operator-level lineage analysis, characterized in that, The system includes: The data parsing unit is used to parse the SQL statement to be parsed corresponding to the target scenario into an abstract syntax tree; The operator extraction unit is used to traverse the abstract syntax tree, extract various operators, and construct the dependency relationships between the various operators; The first construction unit is used to construct an operator directed acyclic graph based on the multiple operators and the dependencies. The second construction unit is used to construct an operator-level lineage storage model based on the operator-directed acyclic graph, which includes operator nodes, table nodes, and the relationships between nodes. The attributes of the relationships include edge weight values, which are obtained by weighted summation of data flow weight, operator complexity weight, dependency weight, and path depth weight, including: Calculate the data flow weight based on the number of rows of operator output data and the number of rows of operator input data. A complexity score is preset for each operator, and the complexity score is used as the operator complexity weight; The importance score of each operator in the operator directed acyclic graph is calculated using PageRank, and the dependency weights are calculated based on the importance scores. The path depth weight is calculated based on the number of upstream tables and downstream tables corresponding to each operator. The number of upstream tables is the number of tables from the input table to the operator, and the number of downstream tables is the number of tables from the output table to the output table. The edge weight value is obtained by weighted summing of the data flow weight, the operator complexity weight, the dependency weight, and the path depth weight. The weight quantization unit is used to normalize the edge weight values ​​to obtain normalized edge weight values, thereby realizing data weight quantization.

8. An electronic device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the data weight quantization method based on operator-level lineage analysis as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the data weight quantization method based on operator-level lineage analysis as described in any one of claims 1 to 6.

Images