Blood relationship analysis method based on data dependency relationship, electronic equipment and medium

Abstract

The invention provides a blood relationship analysis method based on a data dependency relationship, electronic equipment and a medium, and relates to the technical field of data blood relationship analys.The method comprises the steps that an SQL statement vector corresponding to each SQL statement is generated according to all table names and field names corresponding to each SQL statement; obtaining an SQL statement vector list A; all SQL statement vectors in A are clustered to obtain a cluster list B; analyzing each SQL statement in the Bj to extract a corresponding table name, a field name and an operation type, and establishing a field-level blood relationship graph Xj between the table names and the field names of the SQL statements in the Bj; establishing a cross-cluster blood relationship mapping table QR corresponding to the B; fusing the clusters in the B to obtain a global blood relationship analysis graph QW; displaying the QW in a layered interactive view; according to the method, the bottlenecks of the traditional blood relationship analysis method in accuracy, efficiency and availability are solved, and efficient and reliable technical support is provided for large-scale data management.

Description

Technical Field

[0001] The present invention relates to the technical field of data lineage analysis, and in particular to a lineage analysis method, electronic equipment and medium based on data dependency. Background Art

[0002] In the field of data governance, data lineage analysis is a core technology for tracking the complete flow path of data from generation to application. Its core value is reflected in two aspects: one is to improve data transparency, by clarifying the source, processing logic and usage scenarios of the data, to ensure that abnormal data can be quickly traced; the other is to optimize resource efficiency, identify redundant data and inefficient links, and reduce storage and computing costs.

[0003] Traditional lineage analysis methods mainly rely on static parsing and metadata collection; static parsing extracts the structured dependencies of data objects through syntax analysis tools, while metadata collection directly obtains data flow records from logs or systems. However, these methods have significant defects: first, they lack support for complex logic, and have weak parsing capabilities for dynamically generated queries, nested logic, or cross-system interaction scenarios, resulting in broken lineage links; second, they have low computational efficiency, and the complexity of global dependency analysis grows exponentially with the amount of data, making it difficult to cope with large-scale scenarios; therefore, how to perform efficient lineage analysis on the dependencies of the entire data process from source to terminal has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above technical problems, the technical solution adopted by the present invention is: According to a first aspect of the present application, a method for blood relationship analysis based on data dependency is provided, the method comprising the following steps: S100, generating a SQL statement vector corresponding to each SQL statement according to all table names and field names corresponding to each SQL statement; to obtain a SQL statement vector list A=(A1, A2, ..., A i , …, A n ), i=1, 2,...,n; among them, A i is the SQL statement vector corresponding to the i-th SQL statement, and n is the number of SQL statements; S200, cluster all SQL statement vectors in A to obtain a cluster list B = (B1, B2, ..., B j , …, B m ), j = 1, 2, ..., m; where B j is the jth cluster obtained by clustering, and m is the number of clusters obtained by clustering; S300, Analysis B j Extract the corresponding table name, field name and operation type for each SQL statement in B jField-level blood relationship diagram X between the table name and field name in the SQL statement j ; where X j includes several nodes and intra-cluster directed edges connecting two nodes; each node corresponds to a table name or a field name, and there is a blood relationship between the tables or fields corresponding to the two nodes connected by the intra-cluster directed edges; S400, establish a cross-cluster blood relationship mapping table QR corresponding to B; where QR records several clusters with dependency relationships; S500, according to QR, fuse the clusters in B to obtain a global blood relationship analysis diagram QW; where QW includes nodes corresponding to each table name and field name, intra-cluster directed edges connecting nodes in the same cluster, and inter-cluster directed edges connecting nodes in different clusters; S600, display QW in a hierarchical interactive view; where the first-level view of QW shows the nodes corresponding to each cluster and the inter-cluster directed edges, and the second-level view shows the nodes corresponding to the table name and field name and the intra-cluster directed edges.

[0005] According to another aspect of the present application, there is also provided a non-transitory computer-readable storage medium, in which at least one instruction or at least one program segment is stored, and at least one instruction or at least one program segment is loaded and executed by a processor to implement the above-mentioned blood relationship analysis method based on data dependency relationships.

[0006] According to another aspect of the present application, there is also provided an electronic device, including a processor and the above-mentioned non-transitory computer-readable storage medium.

[0007] The present invention has at least the following beneficial effects: The blood relationship analysis method based on data dependency relationships of the present invention combines in-depth analysis of semantic features such as extracting table and field information, clusters SQL statements with similar business logics by learning the semantic relationships between table names and field names, and realizes the combination of structural information and semantic information; first constructs the intra-cluster field-level blood relationship, and performs blood relationship analysis on the tables and fields corresponding to the SQL statements within each cluster to ensure the accuracy of the local blood relationship. Then, it performs the fusion of inter-cluster dependency relationships, constructs a global blood relationship analysis diagram by analyzing the association relationships between clusters. Initially, only the intra-cluster blood relationship is displayed, which is convenient for users to quickly understand the SQL logic of a certain cluster. When more detailed information is needed, users can further expand the inter-cluster blood relationship to view the cross-cluster data flow relationship; compared with directly calculating the global blood relationship, first calculating the intra-cluster and then fusing the inter-cluster relationships as needed greatly reduces the amount of calculation; the hierarchical display avoids the complexity caused by directly displaying the global blood relationship, enabling users to gradually explore the relationships between SQL statements; through triple innovations of semantic vectorization, hierarchical calculation, and interactive visualization, it solves the bottlenecks in accuracy, efficiency, and usability of traditional blood relationship analysis methods, and provides an efficient and reliable technical support for large-scale data governance. Brief Description of the Drawings

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0009] Figure 1 It is a flowchart of the blood relationship analysis method based on data dependency provided by the embodiment of the present invention. Detailed Embodiments

[0010] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

[0011] It should be noted that based on the present disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, this device and / or method can be implemented using other structures and / or functions in addition to one or more of the aspects described herein.

[0012] The following will refer to Figure 1 the flowchart of the blood relationship analysis method based on data dependency shown in the figure to introduce a blood relationship analysis method based on data dependency.

[0013] The blood relationship analysis method based on data dependency may include the following steps: S100, generate a SQL statement vector corresponding to each SQL statement according to all table names and field names corresponding to each SQL statement; to obtain a list of SQL statement vectors A = (A1, A2,..., A i ,..., A n ), i = 1, 2,..., n; where A i is the SQL statement vector corresponding to the i-th SQL statement, and n is the number of SQL statements.

[0014] In this embodiment, the targeted scenario is the same type of business scenario. In the same type of business scenario, the table names and field names involved in the SQL statements have high similarity; each SQL statement used within a preset historical time period can be obtained. For each SQL statement, the corresponding table name and field name can be obtained, and then the table name and field name are vectorized to obtain A.

[0015] Further, A i can be obtained through the following steps: S110, obtain all the table names and field names corresponding to the i-th SQL statement S i to obtain the table name list C i and field name list D i ; where C i = (C i , C i，1 , …, C i，2 , …, C i，r , …, C i，f（i） ), r = 1, 2, …, f(i); C i，r is the r-th table name corresponding to S i , and f(i) is the number of table names corresponding to S i ; D i = (D i，1 , D i，2 , …, D i，p , …, D i，g（i） ), p = 1, 2, …, g(i); D i，p is the p-th field name corresponding to S i , and g(i) is the number of field names corresponding to S i .

[0016] In this embodiment, first, all SQL statements are preprocessed and standardized: all SQL statements are unified in case to avoid problems caused by inconsistent case; aliases in the SQL statements are removed, and escape characters or special symbols in the SQL statements are processed to provide standardized data input for subsequent analysis.

[0017] It can be understood that since in this embodiment, the number of table names and field names corresponding to each SQL statement is different, therefore, in this embodiment, f(i) does not refer to a specific function or function result value, but refers to a value that may vary with the specific value of i. For example, when i = 1, f(i) = 3; when i = 2, f(i) = 4; when i = 3, f(i) = 3; the same is true for g(i).

[0018] S120, use a preset corpus model to vectorize each table name in C i and each field name in D iEach field name in is converted into a corresponding vector to obtain C i The corresponding list of table name vectors XC i and D i The corresponding list of field name vectors XD i ; where XC i = (XC i，1 , XC i，2 , …, XC i，r , …, XC i，f（i） ); XC i，r is the table name vector corresponding to C i，r ; XD i = (XD i，1 , XD i，2 , …, XD i，p , …, XD i，g（i） ); XD i，p is the field name vector corresponding to D i，p .

[0019] Construct a Word2Vec model: Use Word2Vec to train a corpus model based on the business context of SQL statements, map each table name and field name in the SQL statement to a vector space; use the Skip-gram or CBOW model to generate word vectors according to the context of tables and fields in the SQL statement.

[0020] Word embedding representation: Use Word2Vec to represent the table names and field names in SQL as low-dimensional dense vectors. Word2Vec is a context-based word vector model that can capture the semantic similarity between tables and fields.

[0021] S130, determine i and XD i , determine ; where ω table is the preset table name weight, ω field is the preset field name weight; ω table > ω field .

[0022] In this embodiment, different weights for tables and fields are set, and the weight of the table is greater than that of the field, because the table is the core element of the SQL query (such as the users table determines the data source), the main carrier affecting the query result, and the field (such as name) is the specific element of the query operation, only describing the attributes of the table, and has less impact on the query structure; assigning a higher weight to the table name is to highlight its semantic importance and reduce the noise caused by field redundancy.

[0023] The optimal weight distribution of table names and field names in SQL vectorization can be determined through experiments (e.g., ω table= 0.7, ω field = 0.3). Build a corpus of SQL, train a Word2Vec model to generate word vectors, and calculate the semantic relevance and blood relationship analysis accuracy (Acc) of weighted vectors. Use one-way ANOVA and a comprehensive scoring formula to determine the selection of weights, achieving the best balance between semantic expression and business requirements, and providing a quantitative basis for SQL semantic analysis.

[0024] Use Word2Vec to convert the table names and field names in each SQL statement into vector form for clustering analysis. Ignoring the SQL operation types (such as SELECT, INSERT) is to avoid misclassification caused by different operation types and introduce noise. Each statement is converted into a low-dimensional vector representation, and the similarity between SQL statements can be judged by calculating the similarity between vectors.

[0025] S200, cluster all the SQL statement vectors in A to obtain a cluster list B = (B1, B2,..., B j ,..., B m ), j = 1, 2,..., m; where B j is the j-th cluster obtained by clustering, and m is the number of clusters obtained by clustering.

[0026] Furthermore, step S200 may include the following steps: S210, use the elbow method to determine the value of K.

[0027] In this embodiment, the elbow method is used to determine the value of K (the number of clusters). By calculating the sum of squared errors (SSE) under different values of K, select the value of K when the SSE drops rapidly and then levels off. At the same time, appropriately adjust the value of K according to the structure of the clusters. If the differences between SQL statements within each cluster are small, more clusters may be needed. On the contrary, if the differences between SQL statements are large, fewer clusters may be needed. Comprehensively evaluate the clustering effects under different values of K to determine the appropriate number of clusters.

[0028] Specifically, calculate the sum of squared errors (SSE) corresponding to different values of K (such as K = 1 to K = 10), that is, the sum of the squares of the distances between data points within each cluster and the cluster center.

[0029] Plot a curve graph of the value of K and SSE, and select the "inflection point" where the SSE drops significantly more slowly as the optimal value of K.

[0030] Example: When K = 3, SSE = 1000; when K = 4, SSE = 800; when K = 5, SSE = 750; when K = 6, SSE = 740.

[0031] From K = 4 to K = 5, the SSE decline rate decreases significantly (from 200 to 50), so K = 5 is selected.

[0032] S220, cluster all SQL statement vectors in A using the K-Means clustering algorithm; wherein, the similarity between different SQL statement vectors is calculated through cosine similarity.

[0033] Use Word2Vec to construct SQL statement feature vectors, calculate the similarity between vectors based on the semantic similarity of tables and fields using cosine similarity, avoid the curse of dimensionality in high dimensions, and better reflect the SQL semantic association. Use these vectors as the input for K-Means clustering, and group SQL statements with similar business logics into the same cluster to facilitate subsequent lineage analysis.

[0034] S300, parse B j Extract the corresponding table names, field names, and operation types for each SQL statement in B, and establish a field-level lineage graph X j between the table names and field names of the SQL statements in B j ; wherein, X j includes several nodes and intra-cluster directed edges connecting two nodes; each node corresponds to a table name or a field name, and there is a lineage relationship between the tables or fields corresponding to the two nodes connected by the intra-cluster directed edges.

[0035] Furthermore, step S300 may include the following steps: S310, using a lexical analyzer and a syntax analyzer, traverse the syntax tree nodes of each SQL statement, recursively parse the field-level dependency relationships, first determine the operation type through the syntax tree root node, determine the table names and field names of the input tables and the table names and field names of the output tables corresponding to each SQL statement, and obtain the list of table name and field name groups ZB for B j = (ZB j , ZB j，1 , …, ZB j，2 , …, ZB j，q , …, ZB j，u（j） ), q = 1, 2, …, u(j); wherein, ZB j，q is the table name and field name group corresponding to the qth SQL statement in B, and u(j) is the number of SQL statements in B j ; ZB j contains the operation type, input table set, output table set, input field dictionary, and output field dictionary; the keys of the input field dictionary and the output field dictionary are the corresponding table names, and the values are the corresponding lists of field names. j，q

[0036] ​In this embodiment, the ANTLR is used to parse SQL statements, and an SQL syntax analysis tree (Abstract Syntax Tree, AST) is generated through ANTLR. The table names, field names, field operations (such as JOIN, WHERE, SELECT, etc.), and the relationships between fields are extracted from the AST corresponding to each SQL statement, providing structured information for subsequent clustering and lineage analysis.

[0037] Input table set: Parse the FROM clause, JOIN tables, and subquery sources.

[0038] Output table set: Parse target tables such as INSERT INTO and CREATE TABLE AS.

[0039] Field dictionary: Classify the input / output fields into key-value pairs by table name (such as {"table1":["id","name"],"table2":["value"]}).

[0040] S320, for any two table name and field name group lists ZB j in ZB j _A and ZB j _B, if ZB j the table name MZA of the output table of ZB j _A is the same as the table name MZB of the input table of ZB j _B, and the output field dictionary of ZB j _A intersects with the input field dictionary of ZB j _B, then for any field name GQ in the intersection, add a field-level lineage edge pointing from the field node corresponding to GQ in ZB j _A to the field node corresponding to GQ in ZB

[0041] In this embodiment, first, table name matching is performed, that is, the table name (MZA) of the upstream output table is the same as the table name (MZB) of the downstream input table; then, the field intersection is obtained: the output fields of MZA overlap with the input fields of MZB (such as the id field exists simultaneously); finally, lineage edge generation: table-level lineage edge: from MZA to MZB (table-level dependency); field-level edge: for each field in the intersection (such as id), generate a field-level lineage edge from MZA.id to MZB.id.

[0042] S330, if the table name QSA of the input table of ZB j _A is the same as the table name QSB of the input table of ZB j _B, then add a table-level sharing edge pointing from the table node corresponding to QSA to the table node corresponding to QSB; enter S340.

[0043] In this embodiment, the triggering condition of S330 is: input tables of different SQL statements have the same name (for example, multiple tasks read the same original table).

[0044] Shared edge generation: QSA to QSB (table nodes have no blood relationship but a shared relationship).

[0045] Application scenarios include: Optimize resources: Identify frequently accessed shared tables and cache or partition them in a targeted manner.

[0046] Impact analysis: When the shared table structure changes, quickly locate dependent tasks.

[0047] S340, if ZB j _A's field names are QZA and ZB j _B has the same field name as QZB, creating a slave ZB j _A's QZA field node points to ZB j The field-level bloodline of the QZB field node of _B is used to reflect the direct reference of the field between different statements.

[0048] In this embodiment, QZA can be the field name of the input field or the field name of the output field, and QZB can also be the field name of the input field or the field name of the output field; there are fields with the same name in different SQL statements (such as user_id exists in both table A and table B).

[0049] Bloodline edge generation: QZA to QZB (cross-table references between field nodes).

[0050] Through the above steps, based on the strict matching of input and output tables, incorrect connections can be avoided; only the SQL in the cluster needs to be traversed, and the complexity is greatly reduced.

[0051] Then, through the above steps, directed edges are added within the cluster to the nodes corresponding to other tables except the input table and the output table corresponding to different SQL statements; the directed edges within the cluster can clearly show how the data comes, which processes or stages it has gone through, from which tables and which fields it is calculated, indicating the source and destination of the data.

[0052] The method in the above steps has at least the following beneficial effects: Accurate data tracing: When locating data errors, it can be accurate to the field level. For example, if a report value is abnormal, it can be traced back to the calculation logic of a field in a specific ETL step.

[0053] Impact analysis: When a field is modified, the affected downstream fields and tasks are automatically identified. For example, if the orders.amount field is deleted, all queries and reports that depend on the field will be prompted.

[0054] Resource optimization: Identify frequently accessed tables by sharing edges and optimize the allocation of storage or computing resources.

[0055] Data governance: Field-level lineage supports compliance detection, such as tracking the propagation path of the sensitive field phone_number.

[0056] Logical error troubleshooting: Detect invalid lineage, such as a field being processed multiple times but not being used by any task in the end.

[0057] S400, establish a cross-cluster lineage mapping table QR corresponding to B; where QR records several clusters with dependency relationships.

[0058] Further, step S400 may include the following steps: S410, based on B j Establish an index for the input table and output table of the corresponding SQL statement of B j ; where the index includes an input table index and an output table index; the key of the input table index is the table name, and the value is the list of SQL statements using the corresponding table as the input; the key of the output table index is the table name, and the value is the list of SQL statements using the corresponding table as the output.

[0059] In this embodiment, after analyzing the intra-cluster lineage relationship, it is necessary to analyze the SQL statements between different clusters to identify the cross-table and field dependency relationships between them.

[0060] S420, for the indexes corresponding to any two clusters, if there is an intersection GU between the key of the input table index corresponding to one cluster and the key of the output table index corresponding to the other cluster, it is determined that there is a cross-cluster dependency relationship between the two clusters.

[0061] S430, determine the cluster with the key in GU as the output table as the source cluster, and determine the cluster with the key in the intersection as the input table as the target cluster.

[0062] S440, construct a dictionary with the source cluster as the key and the target cluster as the value to obtain QR.

[0063] In this embodiment, based on each SQL statement in the syntax analysis stage, generate the corresponding AST, and extract its input table and output table information. Build indexes for the input tables and output tables of each SQL statement. When finding cross-cluster dependency relationships, the indexes can quickly locate the SQL statements involving specific tables, avoiding traversing all SQL statements. For the SQL statements in different clusters, check whether the output table of one statement is the input table of another statement. If this is the case, it is considered that there is a cross-cluster dependency relationship between the two clusters.

[0064] Establish cross-cluster lineage mapping. Based on the identified cross-cluster dependencies, establish cross-cluster lineage mapping. This mapping records which clusters have dependencies. Specifically, construct a dictionary with the source cluster as the key and the list of target clusters that depend on the source cluster as the value. This dictionary is the cross-cluster lineage mapping.

[0065] The above steps have the following beneficial effects: Quick positioning: Query cross-cluster dependencies through indexing to avoid traversing all SQLs.

[0066] Lightweight storage: The index structure only needs to maintain the mapping relationship between table names and SQLs.

[0067] S500. According to QR, fuse the clusters in B to obtain the global lineage analysis graph QW. Among them, QW includes nodes corresponding to each SQL statement and inter-cluster directed edges connecting nodes in the same cluster and connecting nodes in different clusters.

[0068] Furthermore, step S500 may include the following steps: S510. Set the cluster identifier corresponding to each node within each cluster in B.

[0069] S520. Traverse QR. If the output table corresponding to the SQL statement in the source cluster is the same as the input table corresponding to the SQL statement in the target cluster, add an inter-cluster directed edge from the node corresponding to the SQL statement in the source cluster to the node corresponding to the SQL statement in the target cluster to obtain QW.

[0070] In this embodiment, add corresponding nodes for the SQL statements within each cluster in the global graph, and add corresponding edges according to the intra-cluster lineage graph. To distinguish nodes in different clusters, a cluster identifier will be added to each node.

[0071] According to QR, check the input table and output table relationships of SQL statements between different clusters. If the output table of an SQL statement in one cluster is the input table of an SQL statement in another cluster, add an inter-cluster directed edge from the node corresponding to the SQL statement in the source cluster to the node corresponding to the SQL statement in the target cluster in the global graph.

[0072] The above steps have the following beneficial effects: Global traceability: Completely display the data flow across business modules.

[0073] Hierarchical separation: The cluster identifier avoids node naming conflicts and supports clear visualization.

[0074] S600. Display QW in a hierarchical interactive view. Among them, the first-level view of QW shows the nodes corresponding to each cluster and the inter-cluster directed edges, and the second-level view shows the nodes corresponding to the table names and field names and the intra-cluster directed edges.

[0075] In this embodiment, a graph visualization tool (Neo4j) is used to display the lineage analysis graph, and users can view the intra-cluster dependencies. An interactive view is provided to further view the detailed inter-cluster relationships, display the comprehensive data flow and dependencies between SQL statements, and present the detailed lineage analysis results in the form of a visual graph.

[0076] Hierarchical display (cluster level → SQL level), supporting interactive exploration; Example: Users click on the cluster node to view the internal SQL link, and search for the table name to highlight the path; reducing the user's cognitive burden and improving the troubleshooting efficiency.

[0077] The method in this embodiment has at least the following beneficial effects: (1) Combining syntax parsing and vectorization to improve the feature expression ability of SQL statements Traditional SQL parsing methods usually only extract structural information, while this method deeply parses by combining semantic features such as extracting table and field information. By learning the semantic relationships between table names and field names, SQL statements with similar business logics are clustered to achieve the combination of structural information and semantic information.

[0078] (2) Assigning different weights to tables and fields when vectorizing SQL statements to enhance the SQL semantic expression ability Use Word2Vec to learn the semantic relationships between table names and field names through context. When constructing vectors, set the weight of the table name higher than that of the field name. The table is the core of the SQL query, determining the data source, while the field is only a data unit within the table, and the field is used to describe the specific data in the table. Treating it with a low weight can reduce unnecessary noise while retaining semantic information. Determine the specific weight through experiments to make the vector representation of SQL statements more in line with business logic.

[0079] Solve the problem of improper clustering of SQL statements caused by field redundancy, so that different queries under the same table can be correctly clustered together without being assigned to different clusters due to individual field differences. Maintain the business consistency of SQL statements and ensure that SQL statements with the same business function have higher similarity.

[0080] (3) Based on K-Means clustering, achieve efficient classification of SQL statements Generate the vector representation of SQL statements with the help of Word2Vec, and use the K-Means algorithm to cluster them, grouping similar SQL statements into the same cluster. When clustering, use cosine similarity to calculate similarity, because it is not affected by the vector length and can better reflect the SQL semantic association, avoiding the "curse of dimensionality" of Euclidean distance in high dimensions. Use the elbow method to select the number of clusters K and combine manual adjustment to improve the clustering stability.

[0081] Compared with directly calculating the lineage of SQL statements in full volume, the computational overhead of SQL statement lineage analysis can be reduced. By clustering first and then calculating intra-cluster lineage, the computational amount can be significantly reduced.

[0082] (4) Intra-cluster lineage analysis and inter-cluster fusion to construct a hierarchical lineage analysis framework First, construct the intra-cluster lineage relationship. Perform lineage analysis on the tables and fields corresponding to the SQL statements within each cluster to ensure the accuracy of local lineage relationships. Then, perform inter-cluster dependency relationship fusion. By analyzing the association relationships between clusters, construct a global lineage analysis graph.

[0083] Hierarchical display strategy. Initially, only display intra-cluster lineage to facilitate users to quickly understand the data flow logic between SQL statements in a certain cluster. When more detailed information is needed, users can further expand the inter-cluster lineage to view the data flow relationships across clusters.

[0084] Reduce computational overhead and improve visualization effects: Compared with directly calculating the global lineage relationship, calculate intra-cluster first and then fuse inter-cluster relationships as needed, significantly reducing the computational amount. Hierarchical display avoids the complexity caused by directly displaying the global lineage relationship, enabling users to gradually explore the relationships between SQL statements.

[0085] In addition, although the steps of the methods in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be executed in that specific order, or that all the steps shown must be executed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.

[0086] Embodiments of the present invention also provide a non-transitory computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one segment of a program related to a method for implementing a method in a method embodiment. The at least one instruction or the at least one segment of the program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0087] The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0088] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0089] The program code contained on the readable medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0090] The program code for performing the operations of this application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0091] Embodiments of the present invention also provide an electronic device, including a processor and the foregoing non-transitory computer-readable storage medium.

[0092] The electronic device is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of this application.

[0093] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include but are not limited to: at least one of the foregoing processors, at least one of the foregoing memories, and a bus connecting different system components (including the memory and the processor).

[0094] Wherein, the memory stores program code, and the program code can be executed by the processor, so that the processor executes the steps in the various embodiments described in this specification.

[0095] The memory may include a readable medium in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0096] The memory may also include program / utility with a set (at least one) of program modules, and such program modules include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. An implementation of a network environment may be included in each or some combination of these examples.

[0097] The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus structures.

[0098] The electronic device may also communicate with one or more external devices (such as a keyboard, a pointing device, a Bluetooth device, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device, and / or communicate with any device that enables the electronic device to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be carried out through an input / output (I / O) interface. Moreover, the electronic device may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through the bus. It should be understood that although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0099] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which may be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0100] The embodiments of the present invention also provide a computer program product, which includes program code. When the program product runs on an electronic device, the program code is used to cause the electronic device to execute the steps in the methods according to various exemplary embodiments of the present invention described above in this specification.

[0101] Although some specific embodiments of the present invention have been described in detail by way of examples, those skilled in the art should understand that the above examples are only for illustration purposes and not for limiting the scope of the present invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A lineage analysis method based on data dependency relationships, characterized in that, The method includes the following steps: S100. Generate an SQL statement vector corresponding to each SQL statement based on all table names and field names corresponding to each SQL statement, so as to obtain a list of SQL statement vectors A = (A1, A2,..., A i ,..., A n ), where i = 1, 2,..., n; among them, A i is the SQL statement vector corresponding to the i-th SQL statement, and n is the number of SQL statements; S200, cluster all SQL statement vectors in A to obtain a cluster list B = (B1, B2,..., B j ,..., B m ), where j = 1, 2,..., m; among them, B j is the j-th cluster obtained by clustering, and m is the number of clusters obtained by clustering; S300, parse B j Extract the corresponding table names, field names, and operation types for each SQL statement in B, and establish B j Field-level lineage graph X between the table names and field names of the SQL statements in B j ; where X j Includes several nodes and intra-cluster directed edges connecting two nodes; each node corresponds to a table name or a field name, and there is a lineage relationship between the tables or fields corresponding to the two nodes connected by the intra-cluster directed edges; S400. Establish a cross-cluster blood relationship mapping table QR corresponding to B; wherein, QR records several clusters with dependency relationships; S500. According to QR, fuse the clusters in B to obtain a global blood relationship analysis graph QW; wherein, QW includes nodes corresponding to each table name and field name, as well as intra-cluster directed edges connecting nodes in the same cluster and inter-cluster directed edges connecting nodes in different clusters; S600. Display QW in a hierarchical interactive view; wherein, the first-level view of QW shows the nodes corresponding to each cluster and the inter-cluster directed edges, and the second-level view shows the nodes corresponding to the table name and field name and the intra-cluster directed edges.

2. The method for lineage analysis based on data dependency according to claim 1, wherein A i obtained by the following steps: S110, obtain the i-th SQL statement S i corresponding to all table names and field names to obtain S i corresponding table name list C i and field name list D i ; Among them, C i = (C i，1 , C i，2 , …, C i，r , …, C i，f（i） ), r = 1, 2, …, f(i); C i，r is the r-th table name corresponding to S i , and f(i) is the number of table names corresponding to S i ; D i = (D i，1 , D i，2 , …, D i，p , …, D i，g（i） ), p = 1, 2, …, g(i); D i，p is the p-th field name corresponding to S i , and g(i) is the number of field names corresponding to S i . S120, using a preset corpus model, convert each table name in C i and each field name in D i into corresponding vectors to obtain the list XC i of table name vectors corresponding to C i and the list XD i of field name vectors corresponding to D i ; where XC i = (XC i，1 , XC i，2 , …, XC i，r , …, XC i，f（i） ); XC i，r is the table name vector corresponding to C i，r ; XD i = (XD i，1 , XD i，2 , …, XD i，p , …, XD i，g（i） ); XD i，p is the field name vector corresponding to D i，p . S130, determine according to XC i and XD i , determine ; where ω table is the preset table name weight, and ω field is the preset field name weight; ω table > ω field .

3. The method for lineage analysis based on data dependency according to claim 1, wherein Step S300 includes the following steps: S310. Using a lexical analyzer and a syntax analyzer, traverse the syntax tree nodes of each SQL statement, recursively parse the field-level dependency relationships, first determine the operation type through the syntax tree root node, and determine the table name and field name of the input table and the table name and field name of the output table corresponding to each SQL statement to obtain B j The corresponding list of table name and field name groups ZB j = (ZB j，1 , ZB j，2 , …, ZB j，q , …, ZB j，u（j） ), q = 1, 2, …, u(j); where ZB j，q is the table name and field name group corresponding to the q-th SQL statement in B j , and u(j) is the number of SQL statements in B j ; ZB j，q contains the operation type, the input table set, the output table set, the input field dictionary, and the output field dictionary; the keys of the input field dictionary and the output field dictionary are the corresponding table names, and the values are the corresponding lists of field names; S320, for ZB j in any two table name and field name group lists ZB j _A and ZB j _B, if the table name MZA of the output table of ZB j _A is the same as the table name MZB of the input table of ZB j _B, and the output field dictionary of ZB j _A intersects with the input field dictionary of ZB j _B, then for any field name GQ in the intersection, add a field-level lineage edge from the field node corresponding to GQ in ZB j _A to the field node corresponding to GQ in ZB j _B, and add a table-level lineage edge from the table node corresponding to MZA to the table node corresponding to MZB; S330, if ZB j The table name QSA of the input table of _A is the same as that of ZB j _B, then add a table-level shared edge pointing from the table node corresponding to QSA to the table node corresponding to QSB; go to S340; S340, if ZB j The field name QZA of _A is the same as ZB j _B, create from ZB j The QZA field node of _A points to ZB j The field-level lineage edge of the QZB field node of _B to reflect the direct reference of the field between different statements.

4. The method for lineage analysis based on data dependency according to claim 3, wherein Step S400 includes the following steps: S410, according to B j Based on the input table and output table of the corresponding SQL statement, establish B j The index of the corresponding SQL statement; among them, the index includes an input table index and an output table index; the key of the input table index is the table name, and the value is a list of SQL statements that use the corresponding table as input; the key of the output table index is the table name, and the value is a list of SQL statements that use the corresponding table as output; S420. For the indexes corresponding to any two clusters, if there is an intersection GU between the key of the input table index corresponding to one cluster and the key of the output table index corresponding to the other cluster, it is determined that there is a cross-cluster dependency relationship between the two clusters; S430. Determine the cluster with the key in GU as the output table as the source cluster, and determine the cluster with the key in the intersection as the input table as the target cluster; S440. Construct a dictionary with the source cluster as the key and the target cluster as the value to obtain QR.

5. The method for lineage analysis based on data dependency according to claim 4, wherein Step S500 includes the following steps: S510. Set the cluster identifier corresponding to each node in each cluster in B; S520. Traverse QR. If the output table corresponding to the SQL statement in the source cluster is the same as the input table corresponding to the SQL statement in the target cluster, add an inter-cluster directed edge from the node corresponding to the SQL statement in the source cluster to the node corresponding to the SQL statement in the target cluster to obtain QW.

6. The method for lineage analysis based on data dependency according to claim 1, characterized in that Step S200 includes the following steps: S210. Use the elbow method to determine the value of K; S220. Use the K-Means clustering algorithm to cluster all SQL statement vectors in A; wherein, the similarity between different SQL statement vectors is calculated by cosine similarity.

7. The blood relationship analysis method based on data dependency according to claim 2, characterized in that, The preset corpus model includes: the Word2Vec model.

8. A non-transitory computer-readable storage medium storing at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program is loaded and executed by a processor to implement the blood relationship analysis method based on data dependency relationships as described in any one of claims 1-7.

9. An electronic device, characterized in that, It includes a processor and the non-transitory computer-readable storage medium described in claim 8.

Images