A blood relationship analysis method, electronic device and medium based on data dependency

Abstract

The present invention provides a lineage analysis method, electronic device and medium based on data dependency, which relates to the technical field of data lineage analysis. The method comprises: generating an SQL statement vector corresponding to each SQL statement according to all table names and field names corresponding to each SQL statement; obtaining an SQL statement vector list A; clustering all SQL statement vectors in A to obtain a cluster list B; parsing B j Extract the corresponding table name, field name and operation type for each SQL statement in B j Field-level relationship diagram between table names and field names in SQL statements X j ; Establish a cross-cluster blood relationship mapping table QR corresponding to B; Merge the clusters in B to obtain a global blood relationship analysis diagram QW; Display QW in a layered interactive view; The present invention solves the bottlenecks of traditional blood relationship analysis methods in accuracy, efficiency and usability, and provides efficient and reliable technical support for large-scale data governance.

Description

Technical Field

[0001] The present invention relates to the technical field of data lineage analysis, and in particular to a lineage analysis method, electronic equipment, and medium based on data dependency. Background Art

[0002] In the field of data governance, data lineage analysis is the core technology for tracking the complete flow path of data from generation to application. Its core value is reflected in two aspects: first, it improves data transparency by clarifying the source, processing logic and usage scenarios of the data to ensure that abnormal data can be quickly traced; second, it optimizes resource efficiency, identifies redundant data and inefficient links, and reduces storage and computing costs.

[0003] Traditional lineage analysis methods primarily rely on static parsing and metadata collection. Static parsing extracts structured dependencies between data objects through syntax analysis tools, while metadata collection directly obtains data flow records from logs or systems. However, these methods have significant drawbacks: First, they lack support for complex logic and are unable to parse dynamically generated queries, nested logic, or cross-system interaction scenarios, resulting in broken lineage links. Second, they suffer from low computational efficiency, and the complexity of global dependency analysis increases exponentially with data volume, making it difficult to cope with large-scale scenarios. Therefore, efficient lineage analysis of data dependencies throughout the entire process, from source to endpoint, has become a pressing technical challenge. Summary of the Invention

[0004] In view of the above technical problems, the technical solution adopted by the present invention is:

[0005] According to a first aspect of the present application, a method for blood relationship analysis based on data dependency is provided, the method comprising the following steps:

[0006] S100, based on all table names and field names corresponding to each SQL statement, generate a SQL statement vector corresponding to each SQL statement; to obtain a SQL statement vector list A = (A1, A2, ..., A i ,…,A n ), i=1, 2,...,n; among them, A i is the SQL statement vector corresponding to the i-th SQL statement, and n is the number of SQL statements;

[0007] S200, cluster all SQL statement vectors in A to obtain a cluster list B = (B1, B2, ..., B j ,…,B m ), j = 1, 2, ..., m; where B j is the jth cluster obtained by clustering, and m is the number of clusters obtained by clustering;

[0008] S300, Analysis Bj Extract the corresponding table name, field name and operation type for each SQL statement in B j Field-level relationship diagram between table names and field names in SQL statements X j ; Among them, X j It includes several nodes and directed edges within the cluster connecting two nodes. Each node corresponds to a table name or field name. The tables or fields corresponding to the two nodes connected by the directed edge within the cluster are related.

[0009] S400: Establish a cross-cluster lineage mapping table QR corresponding to B; wherein QR records a number of clusters with dependency relationships;

[0010] S500: Based on QR, the clusters in B are merged to obtain a global blood relationship analysis graph QW; wherein QW includes a node corresponding to each table name and field name, as well as intra-cluster directed edges connecting nodes in the same cluster and inter-cluster directed edges connecting nodes in different clusters;

[0011] S600, displaying QW in a hierarchical interactive view; wherein the first-level view of QW displays the nodes corresponding to each cluster and the directed edges between clusters, and the second-level view displays the nodes corresponding to the table name and field name and the directed edges within the cluster.

[0012] According to another aspect of the present application, a non-transitory computer-readable storage medium is also provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned bloodline analysis method based on data dependency.

[0013] According to another aspect of the present application, an electronic device is provided, including a processor and the above-mentioned non-transitory computer-readable storage medium.

[0014] The present invention has at least the following beneficial effects:

[0015] The data dependency-based lineage analysis method of the present invention combines deep analysis of semantic features such as table and field information extraction, clusters SQL statements with similar business logic by learning the semantic relationship between table names and field names, and realizes the combination of structural information and semantic information. First, the intra-cluster field-level lineage relationship is constructed, and the table and field corresponding to the SQL statement are analyzed in each cluster to ensure the accuracy of the local lineage relationship. Then, the inter-cluster dependency relationship is fused, and a global lineage analysis diagram is constructed by analyzing the association relationship between clusters. Initially, only the intra-cluster lineage is displayed, which makes it easier for users to quickly understand the SQL logic of a cluster. When more detailed information is needed, users can further expand the inter-cluster lineage and view the data flow relationship across clusters. Compared with directly calculating the global lineage relationship, the intra-cluster relationship is calculated first, and then the inter-cluster relationship is integrated as needed, which greatly reduces the amount of calculation. The layered display avoids the complexity caused by directly displaying the global lineage relationship, allowing users to gradually explore the relationship between SQL statements. Through the triple innovation of semantic vectorization, layered calculation and interactive visualization, the bottlenecks of traditional lineage analysis methods in accuracy, efficiency and usability are solved, providing efficient and reliable technical support for large-scale data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flow chart of a blood relationship analysis method based on data dependency provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] It should be noted that, based on this disclosure, those skilled in the art will appreciate that an aspect described herein can be implemented independently of any other aspect, and that two or more of these aspects can be combined in various ways. For example, any number of the aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement such an apparatus and / or practice such a method.

[0020] The following will refer to Figure 1 The flowchart of the bloodline analysis method based on data dependency is shown, which introduces a bloodline analysis method based on data dependency.

[0021] The blood relationship analysis method based on data dependency may include the following steps:

[0022] S100, based on all table names and field names corresponding to each SQL statement, generate a SQL statement vector corresponding to each SQL statement; to obtain a SQL statement vector list A = (A1, A2, ..., A i ,…,A n ), i=1, 2,...,n; among them, A i is the SQL statement vector corresponding to the i-th SQL statement, and n is the number of SQL statements.

[0023] In this embodiment, the scenarios targeted are business scenarios of the same type. In business scenarios of the same type, the table names and field names involved in the SQL statements have a high degree of similarity. Each SQL statement used in a preset historical time period can be obtained, and for each SQL statement, the corresponding table name and field name can be obtained. Then, the table name and field name are vectorized to obtain A.

[0024] Furthermore, A i It can be obtained by following the steps below:

[0025] S110, obtain the i-th SQL statement S i All corresponding table names and field names to get S i Corresponding table name list C i and a list of field names D i Among them, C i =(C i，1 , C i，2 ,…,C i，r ,…,C i，f（i） ), r=1,2,…,f(i);C i，r For S i The corresponding r-th table name, f(i) is S i The number of corresponding table names; D i =(D i，1 , D i，2 ,…,D i，p ,…,D i，g（i） ), p=1, 2,…, g(i); D i，p For S i The corresponding p-th field name, g(i) is S i The number of corresponding field names.

[0026] In this embodiment, first, all SQL statements are preprocessed and standardized: all SQL statements are case-unified to avoid problems caused by inconsistent case; aliases in SQL statements are removed, and escape characters or special symbols in SQL statements are processed to provide standardized data input for subsequent analysis.

[0027] It can be understood that, since the number of table names and field names corresponding to each SQL statement in this embodiment is different, in this embodiment, f(i) does not refer to a specific function or function result value, but refers to a possible value that varies with the specific value of i. For example, when i=1, f(i)=3; when i=2, f(i)=4; when i=3, f(i)=3; the same is true for g(i).

[0028] S120, using the preset corpus model, C i Each table name and D i Each field name in C is converted into a corresponding vector to obtain i Corresponding table name vector list XC i and D i Corresponding field name vector list XD i Among them, XC i =(XC i，1 , XC i，2 ,…,XC i，r ,…,XC i，f（i） );XC i，r C i，r Corresponding table name vector; XD i =(XD i，1 , XD i，2 ,…,XD i，p ,…,XD i，g（i） );XD i，p D i，p A vector of corresponding field names.

[0029] Build a Word2Vec model: Use Word2Vec to train a corpus model of SQL statements based on business context, mapping each table name and field name in the SQL statements into a vector space; use the Skip-gram or CBOW model to generate word vectors based on the context of tables and fields in the SQL statements.

[0030] Word embedding representation: Use Word2Vec to represent table and field names in SQL as low-dimensional dense vectors. Word2Vec is a context-based word embedding model that can capture the semantic similarity between tables and fields.

[0031] S130, according to XC i and XDi ,Sure ; Among them, ω table is the preset table name weight, ω field is the preset field name weight; ω table >ω field .

[0032] In this embodiment, different weights are set for tables and fields. The table weight is greater than the field weight because the table is the core element of the SQL query (for example, the users table determines the data source) and is the main carrier that affects the query results. The field (such as name) is a specific element of the query operation, only describes the attributes of the table, and has little impact on the query structure. Giving the table name a higher weight is to highlight its semantic importance and reduce the noise caused by field redundancy.

[0033] The optimal weight distribution of table names and field names in SQL vectorization can be determined through experiments (e.g. table =0.7,ω field =0.3). A SQL corpus was constructed, a Word2Vec model was trained to generate word vectors, and the semantic relevance and lineage analysis accuracy (Acc) of the weighted vectors were calculated. A one-way variance analysis and a comprehensive scoring formula were used to determine the weights, achieving an optimal balance between semantic expression and business requirements, providing a quantitative basis for SQL semantic analysis.

[0034] Using Word2Vec, we convert the table and field names in each SQL statement into vectors for cluster analysis. We ignore the SQL operation type (such as SELECT and INSERT) to avoid misclassification and noise caused by different operation types. Each statement is converted into a low-dimensional vector representation, and the similarity of SQL statements can be determined by calculating the similarity between vectors.

[0035] S200, cluster all SQL statement vectors in A to obtain a cluster list B = (B1, B2, ..., B j ,…,B m ), j = 1, 2, ..., m; where B j is the jth cluster obtained by clustering, and m is the number of clusters obtained by clustering.

[0036] Furthermore, step S200 may include the following steps:

[0037] S210, determining the K value using the elbow rule.

[0038] In this example, the elbow rule is used to determine the K value (number of clusters). By calculating the sum of squared errors (SSE) for different K values, a K value is selected where the SSE decreases rapidly and then stabilizes. The K value is adjusted appropriately based on the cluster structure. If the differences between SQL statements within each cluster are small, more clusters may be required. Conversely, if the differences between SQL statements are large, fewer clusters may be required. The clustering effects under different K values ​​are comprehensively evaluated to determine the appropriate number of clusters.

[0039] Specifically, the sum of squared errors (SSE) corresponding to different K values ​​(such as K=1 to K=10) is calculated, that is, the sum of the squares of the distances between each data point in each cluster and the cluster center.

[0040] Draw a graph of K value and SSE, and select the "inflection point" where the SSE decreases significantly slower as the optimal K value.

[0041] Example: When K=3, SSE=1000, when K=4, SSE=800, when K=5, SSE=750, and when K=6, SSE=740.

[0042] When K=4 to K=5, the SSE drop rate decreases significantly (from 200 to 50), so K=5 is selected.

[0043] S220, clustering all SQL statement vectors in A using a K-Means clustering algorithm; wherein the similarity between different SQL statement vectors is calculated using cosine similarity.

[0044] We use Word2Vec to construct SQL statement feature vectors. Based on the semantic similarity of tables and fields, we use cosine similarity to calculate the similarity between vectors. This avoids the curse of dimensionality in high-dimensional environments and better reflects SQL semantic associations. These vectors are used as input for K-Means clustering, grouping SQL statements with similar business logic into the same cluster to facilitate subsequent relationship analysis.

[0045] S300, Analysis B j Extract the corresponding table name, field name and operation type for each SQL statement in B j Field-level relationship diagram between table names and field names in SQL statements X j ; Among them, X j It includes several nodes and directed edges within the cluster connecting two nodes; each node corresponds to a table name or field name, and the tables or fields corresponding to the two nodes connected by the directed edge within the cluster have a blood relationship.

[0046] Furthermore, step S300 may include the following steps:

[0047] S310, using the lexical analyzer and the syntax analyzer, traverse the syntax tree nodes of each SQL statement, recursively parse the field-level dependency, first determine the operation type through the syntax tree root node, determine the table name and field name of the input table and the table name and field name of the output table corresponding to each SQL statement, and obtain B j Corresponding table name and field name group list ZB j =(ZB j，1 , ZB j，2 ,…,ZB j，q ,…,ZB j，u（j） ), q = 1, 2, ..., u (j); where ZB j，q B j The table name and field name group corresponding to the qth SQL statement in B, u(j) is j The number of SQL statements in ZB j，q Contains the operation type, input table set, output table set, input field dictionary, and output field dictionary; the keys of the input field dictionary and the output field dictionary are the corresponding table names, and the values ​​are the corresponding field name lists.

[0048] In this example, ANTLR is used to parse SQL statements and generate an Abstract Syntax Tree (AST). From the AST corresponding to each SQL statement, table names, field names, field operations (such as JOIN, WHERE, SELECT, etc.), and relationships between fields are extracted to provide structured information for subsequent clustering and lineage analysis.

[0049] Input table set: Parse the FROM clause, JOIN tables, and subquery sources.

[0050] Output table set: Parse target tables such as INSERT INTO and CREATE TABLE AS.

[0051] Field dictionary: categorizes input / output fields into key-value pairs by table name (e.g. {"table1":["id","name"],"table2":["value"]}).

[0052] S320, for ZB j Any two table names and field names in the list ZB j _A and ZB j _B, if ZB j _A's output table names are MZA and ZB j _B's input table has the same name as MZB, and ZB j _A's output field dictionary and ZB jIf there is an intersection in the input field dictionary of _B, then for any field name GQ in the intersection, add j The field node corresponding to _A's GQ points to ZB j Add a field-level lineage edge to the field node corresponding to GQ of _B, and add a table-level lineage edge from the table node corresponding to MZA to the table node corresponding to MZB.

[0053] In this example, table names are first matched. That is, the table name of the upstream output table (MZA) and the table name of the downstream input table (MZB) are the same. Then, the field intersection is obtained: the output fields of MZA overlap with the input fields of MZB (for example, the id field exists at the same time). Finally, lineage edges are generated: table-level lineage edge: from MZA to MZB (table-level dependency); field-level edge: for each field in the intersection (such as id), a field-level lineage edge is generated from MZA.id to MZB.id.

[0054] S330, if ZB j _A's input table names are QSA and ZB j If the table name of the input table of _B is the same as QSB, a table-level shared edge is added from the table node corresponding to QSA to the table node corresponding to QSB; proceed to S340.

[0055] In this embodiment, the triggering condition of S330 is: the input tables of different SQL statements have the same name (for example, multiple tasks read the same original table).

[0056] Shared edge generation: QSA to QSB (table nodes have no blood relationship but a sharing relationship).

[0057] Application scenarios include:

[0058] Optimize resources: Identify frequently accessed shared tables and implement targeted caching or partitioning.

[0059] Impact analysis: When the shared table structure changes, quickly locate dependent tasks.

[0060] S340, if ZB j _A's field names QZA and ZB j _B's field name is the same as QZB, create a j _A's QZA field node points to ZB j The field-level lineage edge of the QZB field node of _B is used to reflect the direct reference of the field between different statements.

[0061] In this embodiment, QZA can be the field name of the input field or the field name of the output field, and QZB can also be the field name of the input field or the field name of the output field; fields with the same name exist in different SQL statements (such as user_id exists in both table A and table B).

[0062] Bloodline edge generation: QZA to QZB (cross-table references between field nodes).

[0063] Through the above steps, based on the strict matching of input and output tables, incorrect connections are avoided; only the SQL within the cluster needs to be traversed, and the complexity is greatly reduced.

[0064] Then, through the above steps, add directed edges within the cluster to the nodes corresponding to other tables except the input table and output table corresponding to different SQL statements; the directed edges within the cluster can clearly show how the data comes from, which processes or stages it goes through, from which tables and which fields it is calculated, indicating the source and destination of the data.

[0065] The method in the above steps has at least the following beneficial effects:

[0066] Accurate data tracing: When locating data errors, it can be accurate to the field level. For example, if a report value is abnormal, it can be traced back to the calculation logic of a field in a specific ETL step.

[0067] Impact Analysis: When a field is modified, the system automatically identifies the affected downstream fields and tasks. For example, if the orders.amount field is deleted, all queries and reports that depend on it will be prompted.

[0068] Resource optimization: Identify frequently accessed tables through shared edges and optimize storage or computing resource allocation.

[0069] Data governance: Field-level lineage supports compliance detection, such as tracking the propagation path of the sensitive field phone_number.

[0070] Logical error troubleshooting: Detect invalid lineage, such as a field being processed multiple times but ultimately not used by any task.

[0071] S400: Establish a cross-cluster lineage mapping table QR corresponding to B; wherein QR records a number of clusters with dependency relationships.

[0072] Furthermore, step S400 may include the following steps:

[0073] S410, according to B j The corresponding SQL statement input table and output table, establish B j The index of the corresponding SQL statement; the index includes the input table index and the output table index; the key of the input table index is the table name, and the value is the list of SQL statements that use the corresponding table as input; the key of the output table index is the table name, and the value is the list of SQL statements that use the corresponding table as output.

[0074] In this embodiment, after completing the intra-cluster blood relationship analysis, it is necessary to analyze the SQL statements between different clusters to identify the cross-table and field dependencies between them.

[0075] S420 , for the indexes corresponding to any two clusters, if the key of the input table index corresponding to one cluster and the key of the output table index corresponding to the other cluster have an intersection GU, it is determined that there is a cross-cluster dependency relationship between the two clusters.

[0076] S430 , determining a cluster with the keys in GU as the output table as the source cluster, and determining a cluster with the keys in the intersection as the input table as the target cluster.

[0077] S440 , construct a dictionary with the source cluster as the key and the target cluster as the value to obtain QR.

[0078] In this embodiment, a corresponding AST is generated for each SQL statement during the syntax analysis phase, and its input and output table information is extracted. An index is constructed for the input and output tables of each SQL statement. When searching for cross-cluster dependencies, the index can quickly locate the SQL statement involving a specific table, avoiding the need to traverse all SQL statements. For SQL statements in different clusters, a check is performed to see if the output table of one statement is the input table of another statement. If this is the case, a cross-cluster dependency is considered to exist between the two clusters.

[0079] Establish a cross-cluster lineage mapping based on the identified cross-cluster dependencies. This mapping records which clusters have dependencies. Specifically, a dictionary is constructed with the source cluster as the key and the list of target clusters that depend on the source cluster as the value. This dictionary is the cross-cluster lineage mapping.

[0080] The above steps have the following beneficial effects:

[0081] Fast location: Use indexes to query cross-cluster dependencies, avoiding traversing all SQL statements.

[0082] Lightweight storage: The index structure only needs to maintain the mapping relationship between table names and SQL.

[0083] S500: According to QR, clusters in B are merged to obtain a global blood relationship analysis graph QW; wherein QW includes a node corresponding to each SQL statement and directed edges between clusters connecting nodes in the same cluster and connecting nodes in different clusters.

[0084] Furthermore, step S500 may include the following steps:

[0085] S510: Set a cluster identifier corresponding to each node in each cluster in B.

[0086] S520, traverse QR, if the output table corresponding to the SQL statement in the source cluster is the same as the input table corresponding to the SQL statement in the target cluster, then add an inter-cluster directed edge from the node of the SQL statement corresponding to the source cluster to the node of the SQL statement corresponding to the target cluster to obtain QW.

[0087] In this embodiment, a corresponding node is added to the global graph for each SQL statement in the cluster, and corresponding edges are added according to the intra-cluster blood relationship graph; in order to distinguish nodes from different clusters, a cluster identifier is added to each node.

[0088] Based on the QR, we check the relationship between the input and output tables of SQL statements in different clusters. If the output table of a SQL statement in one cluster is the input table of a SQL statement in another cluster, we add a directed edge between clusters in the global graph from the corresponding SQL statement node in the source cluster to the corresponding SQL statement node in the target cluster.

[0089] The above steps have the following beneficial effects:

[0090] Global traceability: Completely display the data flow across business modules.

[0091] Hierarchical separation: Cluster identification avoids node naming conflicts and supports clear visualization.

[0092] S600, displaying QW in a hierarchical interactive view; wherein the first-level view of QW displays the nodes corresponding to each cluster and the directed edges between clusters, and the second-level view displays the nodes corresponding to the table name and field name and the directed edges within the cluster.

[0093] In this example, a graphical tool (Neo4j) is used to display a lineage analysis diagram, allowing users to view intra-cluster dependencies. This interactive view allows for detailed inter-cluster relationships, showing the comprehensive data flow and dependencies between SQL statements, and presenting detailed lineage analysis results in a visual graphical format.

[0094] Hierarchical display (cluster level → SQL level) supports interactive exploration. For example, users can click a cluster node to view internal SQL links and search for table names to highlight paths. This reduces user cognitive burden and improves troubleshooting efficiency.

[0095] The method in this embodiment has at least the following beneficial effects:

[0096] (1) Combining syntax parsing and vectorization to improve the feature expression ability of SQL statements

[0097] Traditional SQL parsing methods usually only extract structural information, while this method combines deep parsing with semantic features such as table and field information. By learning the semantic relationship between table names and field names, SQL statements with similar business logic are clustered to achieve the combination of structural and semantic information.

[0098] (2) When vectorizing SQL statements, different weights are assigned to tables and fields to enhance the SQL semantic expression capability.

[0099] Word2Vec is used to learn the semantic relationship between table names and field names through context. When constructing the vector, the table name is given a higher weight than the field name. The table is the core of the SQL query and determines the source of the data, while the field is just a data unit within the table. The field is used to describe the specific data in the table. Low weight processing can reduce unnecessary noise while retaining semantic information. Specific weights are determined through experiments to make the vector representation of SQL statements more consistent with business logic.

[0100] This solves the problem of improper SQL statement clustering caused by field redundancy, allowing different queries under the same table to be correctly clustered together, rather than being divided into different clusters due to differences in individual fields. This maintains the business consistency of SQL statements, ensuring that SQL statements for the same business function have a higher degree of similarity.

[0101] (3) Efficient SQL statement classification based on K-Means clustering

[0102] Using Word2Vec to generate vector representations of SQL statements, we clustered them using the K-Means algorithm, grouping similar SQL statements into the same cluster. Cosine similarity was used to calculate similarity during clustering, as it is unaffected by vector length and better reflects SQL semantic associations, avoiding the "curse of dimensionality" of Euclidean distance in high dimensions. The number of clusters, K, was selected using the elbow rule, combined with manual adjustments to improve clustering stability.

[0103] Compared with directly calculating the lineage relationship of all SQL statements, reducing the computational overhead of SQL statement lineage analysis, clustering first and then calculating the lineage within the cluster can significantly reduce the amount of computation.

[0104] (4) Intra-cluster blood relationship analysis and inter-cluster fusion to build a hierarchical blood relationship analysis framework

[0105] First, we build intra-cluster kinship relationships. We then perform kinship analysis on the tables and fields corresponding to SQL statements within each cluster to ensure the accuracy of local kinship relationships. We then fuse inter-cluster dependencies and build a global kinship analysis graph by analyzing inter-cluster associations.

[0106] A hierarchical display strategy initially displays only intra-cluster lineage information, allowing users to quickly understand the data flow logic between SQL statements within a cluster. For more detailed information, users can further expand inter-cluster lineage information to view cross-cluster data flow relationships.

[0107] Reduced computational overhead and improved visualization: Compared to directly calculating global kinship relationships, this approach first calculates within a cluster and then integrates inter-cluster relationships as needed, significantly reducing computational effort. Layered visualization avoids the complexity of directly displaying global kinship relationships, allowing users to gradually explore relationships between SQL statements.

[0108] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0109] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0110] The program product may utilize any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0111] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0112] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0113] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0114] An embodiment of the present invention further provides an electronic device including a processor and the aforementioned non-transitory computer-readable storage medium.

[0115] The electronic device is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0116] The electronic device is implemented as a general-purpose computing device. Components of the electronic device may include, but are not limited to, the aforementioned at least one processor, the aforementioned at least one memory, and a bus connecting different system components (including the memory and the processor).

[0117] The memory stores program codes, which can be executed by the processor, so that the processor performs the steps in various embodiments described in this specification.

[0118] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0119] The memory may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0120] The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0121] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0122] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0123] An embodiment of the present invention further provides a computer program product comprising program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0124] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A blood relationship analysis method based on data dependency, characterized in that: The method comprises the following steps: S100: Generate an SQL statement vector corresponding to each SQL statement based on all table names and field names corresponding to each SQL statement, so as to obtain an SQL statement vector list A = (A1, A2, ..., A i ,…,A n ), i=1, 2,...,n; among them, A i is the SQL statement vector corresponding to the i-th SQL statement, and n is the number of SQL statements; S200, cluster all SQL statement vectors in A to obtain a cluster list B = (B1, B2, ..., B j ,…,B m ), j = 1, 2, ..., m; where B j is the jth cluster obtained by clustering, and m is the number of clusters obtained by clustering; S300, Analysis B j Extract the corresponding table name, field name and operation type for each SQL statement in B j Field-level relationship diagram between table names and field names in SQL statements X j ; Among them, X j It includes several nodes and directed edges within the cluster connecting two nodes. Each node corresponds to a table name or field name. The tables or fields corresponding to the two nodes connected by the directed edge within the cluster are related. S400: Establish a cross-cluster lineage mapping table QR corresponding to B; wherein QR records a number of clusters with dependency relationships; S500: Based on QR, the clusters in B are merged to obtain a global blood relationship analysis graph QW; wherein QW includes a node corresponding to each table name and field name, as well as intra-cluster directed edges connecting nodes in the same cluster and inter-cluster directed edges connecting nodes in different clusters; S600, displaying QW in a hierarchical interactive view; wherein the first-level view of QW displays the nodes corresponding to each cluster and the directed edges between clusters, and the second-level view displays the nodes corresponding to the table name and field name and the directed edges within the cluster.

2. The blood relationship analysis method based on data dependency according to claim 1, characterized in that: A i Obtained through the following steps: S110, obtain the i-th SQL statement S i All corresponding table names and field names to get S i Corresponding table name list C i and a list of field names D i ; Among them, C i =(C i，1 , C i，2 ,…,C i，r ,…,C i，f（i） ), r=1,2,…,f(i);C i，r For S i The corresponding r-th table name, f(i) is S i The number of corresponding table names; D i =(D i，1 , D i，2 ,…,D i，p ,…,D i，g（i） ), p=1, 2,..., g(i); D i，p For S i The corresponding p-th field name, g(i) is S i The number of corresponding field names; S120, using the preset corpus model, C i Each table name and D i Each field name in C is converted into a corresponding vector to obtain i Corresponding table name vector list XC i and D i Corresponding field name vector list XD i Among them, XC i =(XC i，1 , XC i，2 ,…,XC i，r ,…,XC i，f（i） );XC i，r C i，r Corresponding table name vector; XD i =(XD i，1 , XD i，2 ,…,XD i，p ,…,XD i，g（i） );XD i，p D i，p The corresponding field name vector; S130, according to XC i and XD i ,Sure ; Among them, ω table is the preset table name weight, ω field is the preset field name weight; ω table >ω field .

3. The blood relationship analysis method based on data dependency according to claim 1, characterized in that: Step S300 includes the following steps: S310, using the lexical analyzer and the syntax analyzer, traverse the syntax tree nodes of each SQL statement, recursively parse the field-level dependency, first determine the operation type through the syntax tree root node, determine the table name and field name of the input table and the table name and field name of the output table corresponding to each SQL statement, and obtain B j Corresponding table name and field name group list ZB j =(ZB j，1 , ZB j，2 ,…,ZB j，q ,…,ZB j，u（j） ), q = 1, 2, ..., u (j); where ZB j，q B j The table name and field name group corresponding to the qth SQL statement in B, u(j) is j The number of SQL statements in ZB j，q Contains the operation type, input table set, output table set, input field dictionary, and output field dictionary; the keys of the input field dictionary and the output field dictionary are the corresponding table names, and the values ​​are the corresponding field name lists; S320, for ZB j Any two table names and field names in the list ZB j _A and ZB j _B, if ZB j _A's output table names are MZA and ZB j _B's input table has the same name as MZB, and ZB j _A's output field dictionary and ZB j If there is an intersection in the input field dictionary of _B, then for any field name GQ in the intersection, add j The field node corresponding to _A's GQ points to ZB j A field-level lineage edge is added to the field node corresponding to GQ of _B, and a table-level lineage edge is added from the table node corresponding to MZA to the table node corresponding to MZB; S330, if ZB j _A's input table names are QSA and ZB j If the table name of the input table of _B is the same as QSB, a table-level shared edge is added from the table node corresponding to QSA to the table node corresponding to QSB; proceed to S340; S340, if ZB j _A's field names QZA and ZB j _B's field name is the same as QZB, create a j _A's QZA field node points to ZB j The field-level lineage edge of the QZB field node of _B is used to reflect the direct reference of the field between different statements.

4. The blood relationship analysis method based on data dependency according to claim 3 is characterized in that: Step S400 includes the following steps: S410, according to B j The corresponding SQL statement input table and output table, establish B j The index of the corresponding SQL statement; the index includes the input table index and the output table index; the key of the input table index is the table name, and the value is the list of SQL statements that use the corresponding table as input; the key of the output table index is the table name, and the value is the list of SQL statements that use the corresponding table as output; S420, for the indexes corresponding to any two clusters, if the key of the input table index corresponding to one cluster and the key of the output table index corresponding to the other cluster have an intersection GU, then it is determined that there is a cross-cluster dependency relationship between the two clusters; S430, determining the cluster with the keys in GU as the output table as the source cluster, and determining the cluster with the keys in the intersection as the input table as the target cluster; S440 , construct a dictionary with the source cluster as the key and the target cluster as the value to obtain QR.

5. The blood relationship analysis method based on data dependency according to claim 4 is characterized in that: Step S500 includes the following steps: S510, setting a cluster identifier corresponding to each node in each cluster in B; S520, traverse QR, if the output table corresponding to the SQL statement in the source cluster is the same as the input table corresponding to the SQL statement in the target cluster, then add an inter-cluster directed edge from the node of the SQL statement corresponding to the source cluster to the node of the SQL statement corresponding to the target cluster to obtain QW.

6. The blood relationship analysis method based on data dependency according to claim 1, characterized in that: Step S200 includes the following steps: S210, determine the K value using the elbow rule; S220, clustering all SQL statement vectors in A using a K-Means clustering algorithm; wherein the similarity between different SQL statement vectors is calculated using cosine similarity.

7. The blood relationship analysis method based on data dependency according to claim 2, characterized in that: The preset corpus model includes: Word2Vec model.

8. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the blood relationship analysis method based on data dependency as described in any one of claims 1-7.

9. An electronic device, characterized in that: The device comprises a processor and the non-transitory computer-readable storage medium of claim 8.

Images