Data blood relationship analysis method and device, terminal equipment and computer program product

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides a data consanguinity analysis method and device, terminal equipment and a computer program product, and the method comprises the steps: obtaining a to-be-queried statement, and determining metadata information corresponding to the to-be-queried statement; based on the to-be-queried statement and the metadata information, graph structure data is constructed and obtained; the graph structure data comprises a plurality of nodes and a plurality of edges, and obtaining a data blood relationship analysis result corresponding to the to-be-queried statement based on the graph structure data and the graph attention network; the graph attention network comprises multiple attention layers, and the multiple attention layers are used for capturing different types of node relations. Compared with the prior art, blood relationship analysis is performed on the graph structure data corresponding to the to-be-queried statement in combination with the graph attention network, so that the accuracy and fine granularity of data blood relationship analysis are improved; meanwhile, the multiple attention layers can capture different types of node relationships, so that more comprehensive and abundant understanding can be obtained, and the accuracy of data blood relationship analysis is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a data lineage analysis method, apparatus, terminal equipment and computer program product. Background Art

[0002] At present, data has become the core asset of enterprises and organizations. As the amount of data grows exponentially and the data processing process becomes increasingly complex, it is crucial to accurately understand and track the source, flow and conversion process of data. This demand has given rise to the development of data lineage analysis technology, which aims to reveal the flow path and interdependencies of data in complex systems. Among them, data lineage, also known as data lineage, data origin, and data lineage, refers to a natural relationship between data in the entire life cycle of data, from data generation, processing, processing, fusion, circulation to final extinction, similar to human blood relationship.

[0003] Existing data lineage analysis methods usually adopt metadata analysis, that is, building lineage relationships based on the metadata information of the database. However, existing technologies have difficulty in discovering fine-grained data conversions in complex queries, thereby reducing the accuracy of data lineage analysis. Summary of the invention

[0004] The embodiments of the present application provide a data lineage analysis method, apparatus, terminal device and computer program product to solve the problem in the prior art that it is difficult to discover fine-grained data conversions in complex queries, thereby reducing the accuracy of data lineage analysis.

[0005] In a first aspect, an embodiment of the present application provides a data lineage analysis method, comprising:

[0006] Obtaining a query statement and determining metadata information corresponding to the query statement;

[0007] Based on the query statement and the metadata information, graph structure data is constructed; the graph structure data includes a plurality of nodes and a plurality of edges, each of the nodes is used to represent a data entity, and each of the edges is used to represent a relationship between the data entities;

[0008] Based on the graph structure data and the graph attention network, a data lineage analysis result corresponding to the query statement is obtained; the graph attention network includes a multi-head attention layer, and the multi-head attention layer is used to capture different types of node relationships.

[0009] A data lineage analysis method provided in an embodiment of the present application obtains a query statement and determines metadata information corresponding to the query statement; based on the query statement and metadata information, a graph structure data is constructed; the graph structure data includes multiple nodes and multiple edges, each node is used to represent a data entity, and each edge is used to represent the relationship between data entities; based on the graph structure data and the graph attention network, a data lineage analysis result corresponding to the query statement is obtained; the graph attention network includes a multi-head attention layer, and the multi-head attention layer is used to capture different types of node relationships. Since the graph attention network can effectively capture the complex relationship between data entities and understand the semantic structure of the query statement, compared with the prior art, the present application combines the graph attention network to perform lineage analysis on the graph structure data corresponding to the query statement, and can obtain more accurate data lineage analysis results, that is, improve the accuracy and granularity of data lineage analysis. At the same time, the multi-head attention layer can capture different types of node relationships, that is, it can focus on different aspects of the data, so that a more comprehensive and rich understanding can be obtained, further improving the accuracy of data lineage analysis.

[0010] Optionally, constructing graph structure data based on the query statement and the metadata information includes:

[0011] Parsing the query statement to obtain an abstract syntax tree corresponding to the query statement;

[0012] The graph structure data is constructed based on the abstract syntax tree and the metadata information.

[0013] In the above implementation, since the abstract syntax tree represents the grammatical structure of the programming language in a tree form, each node on the tree represents a structure in the source code. Therefore, the terminal device can combine the abstract syntax tree and metadata information to quickly obtain graph structure data, thereby improving processing efficiency.

[0014] Optionally, the data lineage analysis result is a lineage relationship graph; and obtaining the data lineage analysis result corresponding to the query statement based on the graph structure data and the graph attention network includes:

[0015] Inputting the graph structure data into the graph attention network for processing to obtain a node vector corresponding to each of the nodes in the graph structure data; the node vector includes feature information, structure information of the corresponding node and context information learned in the graph attention network;

[0016] Determine a data flow path based on each of the node vectors;

[0017] Based on the data flow path, the blood relationship graph is generated.

[0018] In the above implementation, the terminal device can accurately determine the data flow path in combination with the node vector, and then obtain a more accurate data lineage relationship in combination with the data flow path, thereby improving the accuracy of determining the data lineage analysis results.

[0019] Optionally, the graph attention network includes the input layer, the multi-head attention layer and the multi-layer graph convolution layer; the step of inputting the graph structure data into the graph attention network for processing to obtain a node vector corresponding to each node in the graph structure data includes:

[0020] Inputting the graph structure data into the input layer to obtain an initial vector corresponding to each node;

[0021] Input each of the initial vectors into the multi-head attention layer for weight calculation to obtain the attention weight corresponding to each of the initial vectors;

[0022] Each of the initial vectors and the attention weight corresponding to each of the initial vectors are input into the multi-layer graph convolution layer for processing to obtain a node vector corresponding to each of the nodes.

[0023] In the above implementation, the accuracy of determining the node vector is improved.

[0024] Optionally, the data flow path includes target flow paths at different set data granularity levels; and determining the data flow path based on each of the node vectors includes:

[0025] At any set data granularity level, the similarity between each of the nodes is calculated according to each of the node vectors;

[0026] Based on the similarities between the nodes and the graph structure data, a plurality of initial flow paths are constructed;

[0027] Scoring each of the initial flow paths in different dimensions to obtain a score corresponding to each of the initial flow paths;

[0028] The initial flow path with the largest score is determined as the target flow path at the set data granularity level.

[0029] In the above implementation, the data flow path at different data granularity levels can be obtained, so that the subsequent lineage relationship diagram can display the flow process of data from the original table to the query result, and can display data lineage at different granularities such as table level, column level and even expression level.

[0030] Optionally, the data lineage analysis result is a lineage relationship graph, the lineage relationship graph includes a data flow path, and the data flow path includes a plurality of data nodes; after obtaining the data lineage analysis result corresponding to the query statement based on the graph structure data and the graph attention network, the method further includes:

[0031] Generate explanation information corresponding to each of the data nodes; the explanation information is used to describe the relationship between the data node and the result of the data flow path.

[0032] In the above implementation, the user's understanding of the data lineage analysis results can be enhanced, thereby improving the user experience.

[0033] Optionally, the data lineage analysis result is a lineage relationship graph, the lineage relationship graph includes a data flow path, and the data flow path includes a plurality of data nodes; after obtaining the data lineage analysis result corresponding to the query statement based on the graph structure data and the graph attention network, the method further includes:

[0034] Determine a target node in the data flow path that meets a set condition; the set condition is used to indicate that the data node contains a complex operation;

[0035] Prompt information corresponding to the target node is generated; the prompt information is used to describe the impact of the complex operation on data flow.

[0036] In the above implementation, the user's understanding of the data lineage analysis results can be enhanced, thereby improving the user experience.

[0037] Optionally, the method further includes:

[0038] If it is detected that the query type corresponding to the query statement is different from the set type, or the data structure is different from the set structure, or the performance evaluation result of the graph attention network is lower than the set result, then determine the performance score of the graph attention network under different evaluation dimensions;

[0039] Determine a model optimization method based on the scores under the different evaluation dimensions;

[0040] The graph attention network is optimized based on the model optimization method to obtain the optimized graph attention network.

[0041] In the above implementation, the terminal device can ensure that the graph attention network can dynamically adapt to changes in query types and data structures, and automatically update the graph attention network, thereby optimizing the accuracy of data lineage analysis and the performance of the graph attention network.

[0042] Optionally, the different evaluation dimensions include an accuracy dimension, a recall dimension, an F1 score dimension, and a model reasoning speed dimension; and determining the model optimization method based on the scores under the different evaluation dimensions includes:

[0043] If the score of the accuracy dimension is within the first range, or the score of the recall dimension is within the first range, or the score of the F1 score dimension is within the first range, and the delay of the model reasoning speed dimension is within the second range, the incremental learning mode is determined as the model optimization mode;

[0044] If the score of the accuracy dimension is within the third range, or the score of the recall dimension is within the third range, or the score of the F1 score dimension is within the third range, or the delay of the model reasoning speed dimension is within the fourth range, the model fine-tuning mode is determined as the model optimization mode; the lower limit value of the third range is greater than or equal to the upper limit value of the first range, and the lower limit value of the fourth range is greater than or equal to the upper limit value of the second range;

[0045] If the score of the accuracy dimension is in the fifth range, or the score of the recall dimension is in the fifth range, or the score of the F1 score dimension is in the fifth range, and the delay of the model reasoning speed dimension is in the sixth range, the model retraining mode is determined as the model optimization mode; the lower limit value of the fifth range is greater than or equal to the upper limit value of the third range, and the lower limit value of the sixth range is greater than or equal to the upper limit value of the fourth range.

[0046] In the above implementation, the terminal device can flexibly determine the optimization method for the graph attention network based on the scores of different evaluation dimensions, so that the graph attention network can be optimized in a targeted manner, thereby improving the optimization accuracy.

[0047] Optionally, the method further includes:

[0048] If it is detected that the query type corresponding to the target query statement is different from the set type, a historical query statement corresponding to the target query statement is determined from a knowledge base; the knowledge base is used to store the correspondence between different historical query statements and different historical analysis results;

[0049] The historical analysis result corresponding to the historical query statement is determined as the data lineage analysis result corresponding to the target query statement.

[0050] In the above implementation, when the query type corresponding to the target query statement is different from the set type, the knowledge base can be used to search for analysis results that match it, thereby improving the efficiency of data lineage analysis.

[0051] Optionally, determining a historical query statement corresponding to the target query statement from a knowledge base includes:

[0052] Traversing and calculating the similarity between the target query statement and each of the historical query statements in different detection dimensions;

[0053] If at least one similarity among the similarities of any one of the historical query statements is greater than a set threshold, the any one of the historical query statements is determined as a historical query statement corresponding to the target query statement.

[0054] In the above implementation, the accuracy of determining historical query statements is improved.

[0055] Optionally, different detection dimensions include: a grammatical structure dimension, a table and field usage dimension, an operation type dimension, and a query purpose dimension.

[0056] In the above implementation, the accuracy of determining the historical query statements can be further improved.

[0057] In a second aspect, an embodiment of the present application provides a data lineage analysis device, comprising:

[0058] An acquisition unit, used to acquire a sentence to be queried and determine metadata information corresponding to the sentence to be queried;

[0059] A first construction unit is used to construct graph structure data based on the query statement and the metadata information; the graph structure data includes a plurality of nodes and a plurality of edges, each of the nodes is used to represent a data entity, and each of the edges is used to represent a relationship between the data entities;

[0060] The first result determination unit is used to obtain a data lineage analysis result corresponding to the query statement based on the graph structure data and the graph attention network; the graph attention network includes a multi-head attention layer, and the multi-head attention layer is used to capture different types of node relationships.

[0061] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data lineage analysis method as described in any one of the first aspects above is implemented.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data lineage analysis method as described in any one of the above-mentioned first aspects is implemented.

[0063] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the data lineage analysis method described in any one of the first aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 This is a flow chart of the implementation of the data lineage analysis method provided in one embodiment of the present application;

[0066] Figure 2 This is a blood relationship diagram provided by an embodiment of the present application;

[0067] Figure 3 is a flow chart for implementing a data lineage analysis method provided by another embodiment of the present application;

[0068] Figure 4 is a schematic diagram of the structure of a graph attention network provided by an embodiment of the present application;

[0069] Figure 5 is a flow chart of an implementation of a data lineage analysis method provided in yet another embodiment of the present application;

[0070] Figure 6 is a flow chart of an implementation of a data lineage analysis method provided by another embodiment of the present application;

[0071] Figure 7 is a flow chart of an implementation of a data lineage analysis method provided by another embodiment of the present application;

[0072] Figure 8 It is a structural diagram of a data lineage analysis system provided by an embodiment of the present application;

[0073] Fig. 9 It is a structural diagram of an AI analysis engine provided in one embodiment of the present application;

[0074] Fig.10 It is a structural schematic diagram of a data lineage analysis device provided in one embodiment of the present application;

[0075] Fig.11 It is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0076] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0077] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0078] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0079] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0080] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0081] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0082] At present, data has become the core asset of enterprises and organizations. As the amount of data grows exponentially and the data processing process becomes increasingly complex, it is crucial to accurately understand and track the source, flow and conversion process of data. This demand has given rise to the development of data lineage analysis technology, which aims to reveal the flow path and interdependencies of data in complex systems. Among them, data lineage, also known as data lineage, data origin, and data lineage, refers to a natural relationship between data in the entire life cycle of data, from data generation, processing, processing, fusion, circulation to final extinction, similar to human blood relationship.

[0083] However, existing data lineage analysis technologies have encountered severe challenges in the face of modern data environments. First, understanding complex query semantics has become a major problem. In modern data processing environments, Structured Query Language (SOL) queries often involve multi-table joins, nested subqueries, and complex aggregation operations, and even dynamically generated queries and custom functions. This complexity makes it extremely difficult to accurately parse and understand query semantics.

[0084] Existing data lineage analysis methods usually use technical means such as static code analysis, metadata analysis, rule engines, and simple machine learning models. Static code analysis infers data flow relationships by parsing SQL scripts and ETL job codes, but cannot effectively handle dynamically generated SQL or complex data conversion logic. Metadata analysis builds lineage relationships based on the metadata information of the database, but it is difficult to capture fine-grained data transformations in complex queries. Rule engines use predefined rules to infer lineage relationships, but lack flexibility and are difficult to adapt to new query patterns and data model changes. Some systems have tried to introduce simple machine learning models, but these methods find it difficult to fully capture the complex graph structure characteristics of data flows, and their ability to understand and analyze complex queries is still limited.

[0085] Therefore, in practical applications, it is urgent to develop a data lineage analysis system that can accurately understand complex query semantics, automatically adapt to changes in the data environment, and provide high performance and scalability. Graph neural network technology, due to its powerful graph structure learning ability, provides a new possibility for solving this problem. It can effectively capture the complex relationships between data entities and understand the semantic structure of queries, thereby improving the accuracy and granularity of lineage analysis.

[0086] Based on this, an embodiment of the present application proposes a data lineage analysis method, which combines graph neural network technology to implement lineage analysis of data to improve the accuracy and granularity of data lineage analysis.

[0087] See also Figure 1 , Figure 1This is a flowchart of the implementation of the data lineage analysis method provided in one embodiment of the present application. In the embodiment of the present application, the execution subject of the data lineage analysis method is a terminal device. The terminal device includes but is not limited to: a notebook, a desktop computer or a computer.

[0088] like Figure 1 As shown, the data lineage analysis method provided in the embodiment of the present application includes S101 to S103, which are described in detail as follows:

[0089] In S101, a sentence to be queried is obtained, and metadata information corresponding to the sentence to be queried is determined.

[0090] In actual applications, when a user needs to perform data lineage analysis on certain data, the user can request the data lineage analysis to the terminal device.

[0091] In an embodiment of the present application, a terminal device detecting a data lineage analysis request sent by a user may include: detecting a preset operation for the terminal device. The preset operation can be set according to actual needs and is not limited here. Exemplarily, the preset operation may be clicking a preset control of the terminal device or clicking a preset APP of the terminal device. Based on this, when the terminal device detects that its preset control is clicked, or its preset APP is clicked, it indicates that the preset operation is detected, that is, the above-mentioned data lineage analysis request is detected.

[0092] After receiving the above data lineage analysis request, the terminal device can obtain the SQL query being executed from the database connected to it wirelessly, that is, obtain the statement to be queried.

[0093] Afterwards, the terminal device can continue to obtain metadata information corresponding to the query statement from the above database, wherein the metadata information includes but is not limited to: table structure information, index information, constraint conditions, data type, and execution plan.

[0094] In S102, graph structure data is constructed based on the query statement and the metadata information; the graph structure data includes multiple nodes and multiple edges, each of the nodes is used to represent a data entity, and each of the edges is used to represent the relationship between data entities.

[0095] In the embodiment of the present application, in order to realize the lineage analysis of data in the graph attention network, the terminal device can construct the graph structure data matching the graph attention network based on the above query statement and metadata information obtained. The graph structure data includes multiple nodes and multiple edges, each node is used to represent a data entity, and each edge is used to represent the relationship between data entities.

[0096] It should be noted that since the query statements and metadata information usually contain multiple tables, fields, connection operations and other elements, and the relationship between these elements can be constructed into a directed graph, therefore, in an embodiment of the present application, the terminal device can parse and process the above-mentioned query statements and metadata information to obtain a graph structure form that can be processed by the graph attention network, that is, graph structure data.

[0097] In one embodiment of the present application, the terminal device can obtain the graph structure data through the following steps, which are described in detail as follows:

[0098] Parsing the query statement to obtain an abstract syntax tree corresponding to the query statement;

[0099] The graph structure data is constructed based on the abstract syntax tree and the metadata information.

[0100] In practical applications, since the Abstract Syntax Tree (AST) is an abstract representation of the grammatical structure of the source code, it represents the grammatical structure of the programming language in a tree form, and each node on the tree represents a structure in the source code.

[0101] Therefore, in this embodiment, in order to improve the efficiency of constructing graph structure data, the terminal device can input the above query statement into the SQL parser for parsing and processing to obtain an abstract syntax tree corresponding to the query statement.

[0102] It should be noted that the SQL parser in this embodiment can be determined based on an existing SQL parser, which will not be described in detail here.

[0103] In this embodiment, after obtaining the above-mentioned abstract syntax tree, the terminal device can map each element in the abstract syntax tree and metadata information into nodes of the graph (such as tables, fields, aggregate functions), and construct directed edges between nodes through a graph processing library or a graph database to represent the dependency relationship between tables and fields, thereby constructing graph structured data.

[0104] In practical applications, graph processing libraries mainly refer to computing frameworks or engines used to process graph data, such as graph computing engines.

[0105] Graph Database is suitable for processing complex relational networks and has key features such as nodes, edges, and property graphs. Graph databases include but are not limited to Neo4j and OrientDB.

[0106] In S103, based on the graph structure data and the graph attention network, a data lineage analysis result corresponding to the query statement is obtained; the graph attention network includes a multi-head attention layer, and the multi-head attention layer is used to capture different types of node relationships.

[0107] In an embodiment of the present application, after obtaining the graph structure data, the terminal device can input the graph structure data into the graph attention network for processing to obtain the data lineage analysis result corresponding to the query statement. The graph attention network includes a multi-head attention layer, which is used to capture different types of node relationships.

[0108] In practical applications, the Graph Attention Network (GAT) is a graph neural network that can automatically learn key nodes and relationships in data flow by assigning different weights to the neighbors of each node through the attention mechanism when processing graph structured data. The graph attention network can effectively handle complex structures such as multi-table joins and nested queries in SQL queries and identify data flow paths.

[0109] In the embodiment of the present application, the data lineage analysis result may include a lineage relationship diagram.

[0110] For example, a sales manager wants to analyze the sales in different regions and customer types in 2023, with a special focus on the sales and order quantity of the "electronic products" and "household products" categories, and exclude orders with too low discounts. The SQL query entered, i.e. the query statement to be queried, is as follows:

[0111] SELECT

[0112] c.Region AS customer region,

[0113] ct.CustomerType AS customer type,

[0114] p.Category AS product category,

[0115] SUM(o.OrderAmount)AS total order amount,

[0116] COUNT(o.OrderID)AS Order Quantity

[0117] FROM

[0118] Orders of

[0119] JOIN

[0120] Customers c ON o.CustomerID=c.CustomerID

[0121] JOIN

[0122] CustomerTypes ct ON c.CustomerTypeID=ct.CustomerTypeID

[0123] JOIN

[0124] Products p ON o.ProductID=p.ProductID

[0125] LEFT JOIN

[0126] OrderDiscounts od ON o.OrderID=od.OrderID

[0127] WHERE

[0128] o.OrderDate>='2023-01-01'AND o.OrderDate<='2023-12-31'

[0129] AND p.Category IN('Electronic products','Household items')

[0130] AND(od.DiscountAmount IS NULL OR od.DiscountAmount>10)

[0131] GROUP BY

[0132] c.Region,ct.CustomerType,p.Category

[0133] ORDER BY

[0134] c.Region,ct.CustomerType,p.Category;

[0135] Based on this, the terminal device can obtain Figure 2 Schematic diagram of blood relationship shown.

[0136] In one embodiment of the present application, in order to improve the accuracy of the determination of the data lineage analysis result, the terminal device can specifically perform the following steps: Figure 3 The steps S201 to S203 shown implement step S103, which are described in detail as follows:

[0137] In S201, the graph structure data is input into the graph attention network for processing to obtain a node vector corresponding to each node in the graph structure data; the node vector includes feature information, structure information and context information learned in the graph attention network of the corresponding node.

[0138] In this embodiment, the terminal device can input the graph structure data into the graph attention network for processing to obtain a node vector corresponding to each node in the graph structure data, wherein the node vector includes feature information, structure information of the corresponding node and context information learned in the graph attention network.

[0139] It should be noted that the graph attention network includes but is not limited to: input layer, multi-head attention layer and multi-layer graph convolution layer. Among them, the multi-head attention layer and the multi-layer graph convolution layer are collectively referred to as the graph attention layer.

[0140] The input layer is used to receive input graph structure data, including nodes such as tables, columns, and operators. The multi-head attention layer is used to capture different types of node relationships using multiple attention mechanisms to focus on different aspects of the data and achieve a more comprehensive and richer understanding of the data. The multi-layer graph convolution layer is used to aggregate neighbor information, that is, the node information adjacent to each node, to learn the contextual representation of each node, so that the graph attention network can better understand the local relationship and structure between the data.

[0141] In some possible embodiments, the graph attention network may further include an output layer.

[0142] In this embodiment, the output layer is used to generate the final representation of each node in the graph structure data to facilitate subsequent blood relationship reasoning.

[0143] Based on this, see Figure 4 , Figure 4 is a schematic diagram of the structure of a graph attention network provided by an embodiment of the present application. Figure 4 As shown, the graph attention network includes an input layer, a graph attention layer, and an output layer.

[0144] In this embodiment, the above feature information is used to describe the original features of the node, including but not limited to the node type (such as table, column, and operation, etc.) and attributes, etc.

[0145] The above structural information is used to describe the position of the node in the graph structure data and the connection relationship with other nodes.

[0146] The above contextual information is used to represent broader relational information learned by the graph attention network.

[0147] In one embodiment of the present application, in order to improve the accuracy of determining the node vector of each node, the terminal device can specifically determine the node vector of each node in the graph structure data according to the following steps, which are detailed as follows:

[0148] Inputting the graph structure data into the input layer to obtain an initial vector corresponding to each node;

[0149] Input each of the initial vectors into the multi-head attention layer for weight calculation to obtain the attention weight corresponding to each of the initial vectors;

[0150] Each of the initial vectors and the attention weight corresponding to each of the initial vectors are input into the multi-layer graph convolution layer for processing to obtain a node vector corresponding to each of the nodes.

[0151] In this embodiment, the terminal device can input the graph structure data into the input layer of the graph attention network to perform an initialization operation, that is, generate an initial feature vector corresponding to each node in the graph structure data, thereby obtaining an initial vector corresponding to each node in the graph structure data.

[0152] Exemplarily, the initial vector corresponding to the table node may include metadata of the table, the initial vector corresponding to the column node may include data type and statistical information, and the initial feature vector corresponding to the operation node (such as JOIN, WHERE) may include the encoding of the operation type.

[0153] In this embodiment, after obtaining the initial vector of each node, the terminal device can input the initial vector of each node into the multi-head attention layer for weight calculation to obtain the attention weight corresponding to the initial vector of each node.

[0154] Afterwards, the terminal device can input each initial vector and the attention weight corresponding to each initial vector into the multi-layer graph convolution layer for processing to obtain the node vector corresponding to each node.

[0155] Specifically, the terminal device can use multi-layer graph convolutional layers (GCN) to learn the features of each node and the contextual relationship of each node. Each layer of GCN can be expressed as: Among them, H^(l) is the node feature matrix of the lth layer, is the normalized adjacency matrix (including attention weights), W^(l) is the weight matrix of the lth layer, and σ is the nonlinear activation function. By stacking multiple layers of GCN, the graph attention network can capture the relationship between nodes at a longer distance.

[0156] It should be noted that in each layer of GCN, each node aggregates information from its neighboring nodes and updates its own feature vector. Considering the attention mechanism, this process can be expressed as: h_i^(l+1)=σ(Σ_jα_ij W^(l)h_j^(l)). Among them, α_ij is the attention weight of node j to node i.

[0157] It is understandable that each layer of graph convolution is essentially aggregating and propagating information. Initially, each node only contains its own features. As the graph convolution layers are superimposed, the nodes gradually obtain a wider range of contextual information.

[0158] In some possible embodiments, the multi-layer graph convolution layer may include at least three convolution layers.

[0159] In this embodiment, the first graph convolution layer is used for each node to aggregate the information of directly adjacent nodes and learn the first-order neighborhood relationship. The second graph convolution layer is used for each node to obtain the information of the second-order neighborhood (the adjacent nodes of each node's adjacent nodes, that is, the neighbors of the neighbors), and the receptive field is expanded. The third graph convolution layer is based on a deeper graph convolution layer to capture longer-distance relationships for each node, and gradually establish a global contextual understanding.

[0160] From the above, we can see that through the above layer-by-layer expansion method, the graph attention network can learn the contextual relationship of each node in the entire graph structure, not just the local features.

[0161] Based on this, the terminal device can output the node vector of each of the above nodes through the output layer of the graph attention network for subsequent blood relationship analysis.

[0162] In S202, a data flow path is determined based on each of the node vectors.

[0163] In this embodiment, after obtaining the node vector of each node, the terminal device may determine the flow path of data at different set data granularity levels based on the node vector of each node to obtain the final data flow path.

[0164] Specifically, for any set data granularity level, the terminal device can calculate the similarity between the node vectors of each node at the set data granularity level, and construct the flow path at the set data granularity level starting from the source node in the graph structure data based on the various similarities and the graph structure data.

[0165] Based on this, the terminal device can obtain the flow paths under different set data granularity levels, and determine the flow paths under different set data granularity levels as the final data flow paths.

[0166] It should be noted that different set data granularity levels include but are not limited to: table level, column level, expression level, operation level, and task level.

[0167] The table level refers to the coarsest level, showing how data flows between different tables, for example, from source table A to intermediate table B, and then to target table C.

[0168] Column Level refers to a more granular level that shows the data flow between specific columns. For example, how column x in table A is mapped to column y in table B.

[0169] The expression level refers to the most granular level, showing the detailed process of data conversion and calculation, which may include SQL expressions, function applications, etc. For example: Column x of table A becomes column y of table B after passing through the SUM function.

[0170] The operation level is used to display the various operations and transformations that the data has undergone, including JOIN, FILTER, AGGREGATE, and other operations.

[0171] The job level is used to show the flow of data between different tasks or jobs at a higher level.

[0172] The flow path is used to characterize the flow of data from the original table to the query results.

[0173] In one embodiment of the present application, when the data flow path includes target flow paths at different set data granularity levels, in order to improve the accuracy of determining each target flow path, the terminal device may specifically implement step S202 by the following steps, which are described in detail as follows:

[0174] At any set data granularity level, the similarity between each of the nodes is calculated according to each of the node vectors;

[0175] Based on the similarities between the nodes and the graph structure data, a plurality of initial flow paths are constructed;

[0176] Scoring each of the initial flow paths in different dimensions to obtain a score corresponding to each of the initial flow paths;

[0177] The initial flow path with the largest score is determined as the target flow path at the set data granularity level.

[0178] In this embodiment, for any set data granularity level, the terminal device can calculate the similarity between each node based on the node vector of each node at the set data granularity level. Afterwards, the terminal device can construct multiple possible flow paths at the set data granularity level, i.e., multiple initial flow paths, starting from the source node in the graph structure data, based on the similarity between each node and the graph structure data.

[0179] In one implementation of this embodiment, the terminal device can obtain all possible flow paths at the above-set data granularity level based on an existing graph traversal algorithm (such as a depth-first search algorithm or a breadth-first search algorithm).

[0180] In practical applications, the Depth-First-Search (DFS) process is simply to search every possible branch path until it cannot go any deeper, and each node can only be visited once.

[0181] Breadth First Search (BFS) is a blind search method that aims to systematically expand and check all nodes in the graph to find the result. It is understandable that BFS does not consider the possible location of the result, but thoroughly searches the entire graph until the result is found.

[0182] In this embodiment, after obtaining multiple initial flow paths at a set data granularity level, the terminal device can score each initial flow path in different dimensions to obtain a final score for each initial flow path, wherein the different dimensions include but are not limited to dimensions such as path length and node similarity.

[0183] In some possible embodiments, the terminal device may sequentially input each initial flow path into a trained scoring model for processing to obtain a final score for each initial flow path.

[0184] It should be noted that the scoring model includes scoring sub-models in multiple dimensions.

[0185] In this embodiment, after obtaining the scores of multiple initial flow paths at a set data granularity level, the terminal device can compare the scores to determine the initial flow path with the largest score, and determine the initial flow path with the largest score as the target flow path at the set data granularity level.

[0186] Based on this, the terminal device can obtain the target flow path under each different set data granularity level, that is, obtain the data flow path.

[0187] In S203, the blood relationship diagram is generated based on the data flow path.

[0188] In this embodiment, after obtaining the above-mentioned data flow path, the terminal device can draw a lineage relationship diagram based on the data flow path, thereby obtaining the data lineage analysis result corresponding to the query statement.

[0189] In an embodiment of the present application, after obtaining the data lineage analysis result, the terminal device may store the data lineage analysis result in a graph database to facilitate subsequent queries.

[0190] In one embodiment of the present application, in order to improve user experience, the terminal device may also visualize the data lineage analysis results so that the user can intuitively see the data lineage analysis results corresponding to the query statement.

[0191] From the above, it can be seen that the data lineage analysis method provided by the embodiment of the present application obtains the query statement and determines the metadata information corresponding to the query statement; based on the query statement and the metadata information, the graph structure data is constructed; the graph structure data includes multiple nodes and multiple edges, each node is used to represent a data entity, and each edge is used to represent the relationship between data entities; based on the graph structure data and the graph attention network, the data lineage analysis result corresponding to the query statement is obtained; the graph attention network includes a multi-head attention layer, and the multi-head attention layer is used to capture different types of node relationships. Since the graph attention network can effectively capture the complex relationship between data entities and understand the semantic structure of the query statement, compared with the prior art, the present application combines the graph attention network to perform lineage analysis on the graph structure data corresponding to the query statement, and can obtain more accurate data lineage analysis results, that is, improve the accuracy and granularity of the data lineage analysis. At the same time, the multi-head attention layer can capture different types of node relationships, that is, it can focus on different aspects of the data, so that a more comprehensive and rich understanding can be obtained, further improving the accuracy of the data lineage analysis.

[0192] In order to improve the interpretability of data lineage analysis results and enhance users' understanding of data lineage analysis results, when the data lineage analysis result is a lineage relationship diagram, and the lineage relationship diagram includes a data flow path, and the data flow path includes multiple data nodes, the terminal device can generate a detailed description for each data flow path.

[0193] Based on this, see Figure 5 , Figure 5 FIG. 1 is a flowchart of a data lineage analysis method according to another embodiment of the present application. Figure 5 As shown, relative to Figure 1 Corresponding embodiment, in this embodiment, after step S103, S301 and / or steps S302 to S303 may also be included, which are described in detail as follows:

[0194] In S301, explanation information corresponding to each of the data nodes is generated; the explanation information is used to describe the relationship between the data node and the result of the data flow path.

[0195] In this embodiment, the terminal device can generate relevant explanations of how each data node participates in the generation of final data through steps such as table connection and field conversion according to the connection relationship between each data node in the data flow path, that is, generate explanation information corresponding to each data node. The explanation information of each data node is used to describe the relationship between the data node and the result of the data flow path.

[0196] In one embodiment of the present application, in order to further improve the user experience and help the user understand how complex operations (such as aggregate functions and calculation expressions, etc.) affect the data flow in the data flow path, the terminal device may further perform the following steps:

[0197] In S302, a target node in the data flow path that meets a set condition is determined; the set condition is used to characterize that the data node contains a complex operation.

[0198] In S303, prompt information corresponding to the target node is generated; the prompt information is used to describe the impact of the complex operation on the data flow.

[0199] In this embodiment, the setting conditions may include: a certain data node includes complex operations such as aggregation functions or calculation expressions.

[0200] Based on this, when the terminal device detects that a data node in the data flow path has complex operations such as aggregation functions or calculation expressions, it can determine that the data node is a target node that meets the set conditions.

[0201] Afterwards, the terminal device may generate prompt information corresponding to the target node, wherein the prompt information is used to describe the impact of the complex operation on the data flow.

[0202] It can be seen from the above that the data lineage analysis method provided in this embodiment can enhance the user's understanding of the data lineage analysis results and improve the user experience.

[0203] In one embodiment of the present application, in order to automatically adapt to changes in the data environment and implement an adaptive learning mechanism so that the graph attention network in the present application can continuously adapt to new data structures and query types, the terminal device can Figure 6 Steps S401 to S403 shown automatically update the graph attention network to ensure that the graph attention network in this application can dynamically adapt to changes in query types and data structures, thereby optimizing the accuracy of data lineage analysis and the performance of the graph attention network.

[0204] Based on this, see Figure 6 , Figure 6 FIG. 1 is a flowchart of a method for analyzing data lineage provided in another embodiment of the present application. Figure 6 As shown, relative to Figure 1 Corresponding to the embodiment, the data lineage analysis method provided in this embodiment may also include S401 to S403, which are described in detail as follows:

[0205] In S401, if it is detected that the query type corresponding to the query statement is different from the set type, or the data structure is different from the set structure, or the performance evaluation result of the graph attention network is lower than the set result, then the performance score of the graph attention network under different evaluation dimensions is determined.

[0206] In this embodiment, in order to achieve timely update of the graph attention network in this application, the terminal device can detect in real time whether the query type corresponding to the query statement that requires data lineage analysis is the same as the set type, and whether the data structure of the query statement is the same as the set structure, or perform real-time evaluation on the real-time acquired data lineage analysis results to obtain the performance evaluation results of the graph attention network in real time. Among them, the set type can be determined based on the query type already in the database, and the set structure can be determined based on the data structure already in the database. The performance evaluation result is used to describe whether the performance of the graph attention network has declined. The setting result is used to describe that the performance of the graph attention network has declined.

[0207] It is understandable that the data structure may be the structure of a table in a database, the definition of a field, the setting of an index, etc.

[0208] It should be noted that the query statement can be the query statement in step S101, or any query statement after the query statement. The data lineage analysis result obtained in real time can be the data lineage analysis result corresponding to the query statement in step S104, or any query statement after the query statement.

[0209] In this embodiment, when the terminal device detects that the query type corresponding to the query statement is different from the set type, it indicates that a new query pattern has appeared in the database. Therefore, in order to clarify whether the graph attention network at this time is suitable for data lineage analysis of query statements under the above new pattern, the terminal device can determine the performance of the graph attention network in performing data lineage analysis on query statements under the above new pattern under different evaluation dimensions.

[0210] When the terminal device detects that the data structure of the query statement is different from the set structure, it means that the data structure in the database has changed. Therefore, in order to clarify whether the graph attention network is applicable to the new data structure at this time, the terminal device can determine the performance of the graph attention network when performing data lineage analysis on the query statement with the above new data structure. The score under different evaluation dimensions.

[0211] When the terminal device detects that the performance evaluation result of the graph attention network is lower than the set result, it means that the performance of the graph attention network has declined. Therefore, in order to clarify the degree of performance degradation of the graph attention network at this time, the terminal device can determine the performance score of the graph attention network under different evaluation dimensions.

[0212] It should be noted that the above-mentioned different evaluation dimensions include but are not limited to accuracy dimension, recall dimension, F1 score dimension and model inference speed dimension.

[0213] Among them, the accuracy refers to the accuracy of the graph attention network in parsing the structure and data flow of the query statement.

[0214] Recall refers to the effective tracking rate of the graph attention network in lineage tracking.

[0215] The F1 score takes into account both the precision and recall of the graph attention network. The F1 score can be regarded as a harmonic average of the precision and recall of the graph attention network, with a maximum value of 1 and a minimum value of 0.

[0216] Model inference speed refers to the time it takes for the graph attention network to parse query statements and generate kinship relationships.

[0217] In this embodiment, the score of the accuracy dimension is used to describe the degree of decrease in accuracy, the score of the recall dimension is used to describe the degree of decrease in recall, the score of the F1 score dimension is used to describe the degree of decrease in F1 score, and the score of the model inference speed dimension is used to describe the degree of delay in inference speed.

[0218] In some possible embodiments, when the terminal device detects that the query type corresponding to the query statement is different from the set type, the terminal device can compare the current evaluation scores of the performance of the graph attention network in performing data lineage analysis on the query statement in the above new mode under different evaluation dimensions with the standard evaluation scores under the corresponding evaluation dimensions, so as to obtain the scores of the performance of the graph attention network under different evaluation dimensions. Among them, the standard evaluation scores of different evaluation dimensions can be set according to actual needs and are not limited here.

[0219] In other possible embodiments, when the terminal device detects that the data structure of a query statement is different from the set structure, the terminal device can compare the current evaluation scores of the performance of the graph attention network in performing data lineage analysis on the query statement with the new data structure under different evaluation dimensions with the standard evaluation scores under the corresponding evaluation dimensions to obtain the performance scores of the graph attention network under different evaluation dimensions.

[0220] In some further possible embodiments, when the terminal device detects that the performance of the graph attention network has declined, the terminal device may compare the current evaluation scores of the graph attention network under different evaluation dimensions with the standard evaluation scores under the corresponding evaluation dimensions to obtain the performance scores of the graph attention network under different evaluation dimensions.

[0221] In S402, a model optimization method is determined based on the scores under the different evaluation dimensions.

[0222] In S403, the graph attention network is optimized based on the model optimization method to obtain the optimized graph attention network.

[0223] In this embodiment, the terminal device can determine the performance degradation of the graph attention network according to the scores of the graph attention network under different evaluation dimensions, and thus determine the optimization method for the graph attention network, that is, the model optimization method, according to the performance degradation degree.

[0224] It should be noted that the terminal device pre-stores the corresponding relationship between different performance degradation degrees and different model optimization methods. Among them, different performance degradation degrees include but are not limited to: insignificant performance degradation, significant performance degradation, and severe performance degradation. Different model optimization methods include but are not limited to: incremental learning method, model fine-tuning method, and model retraining method.

[0225] Incremental learning refers to periodically updating model parameters with new samples without completely retraining.

[0226] Model fine-tuning refers to the targeted optimization of the pre-trained model through data in a specific field to improve its performance on specific tasks.

[0227] Model retraining refers to retraining the entire model.

[0228] In this embodiment, the terminal device can set the model optimization method corresponding to insignificant performance degradation to incremental learning; the terminal device can set the model optimization method corresponding to significant performance degradation to model fine-tuning; the terminal device can set the model optimization method corresponding to severe performance degradation to model retraining.

[0229] In one embodiment of the present application, the terminal device may specifically determine the model optimization method according to the following steps, which are described in detail as follows:

[0230] If the score of the accuracy dimension is within the first range, or the score of the recall dimension is within the first range, or the score of the F1 score dimension is within the first range, and the delay of the model reasoning speed dimension is within the second range, the incremental learning mode is determined as the model optimization mode;

[0231] If the score of the accuracy dimension is within the third range, or the score of the recall dimension is within the third range, or the score of the F1 score dimension is within the third range, or the delay of the model reasoning speed dimension is within the fourth range, the model fine-tuning mode is determined as the model optimization mode; the lower limit value of the third range is greater than or equal to the upper limit value of the first range, and the lower limit value of the fourth range is greater than or equal to the upper limit value of the second range;

[0232] If the score of the accuracy dimension is in the fifth range, or the score of the recall dimension is in the fifth range, or the score of the F1 score dimension is in the fifth range, and the delay of the model reasoning speed dimension is in the sixth range, the model retraining mode is determined as the model optimization mode; the lower limit value of the fifth range is greater than or equal to the upper limit value of the third range, and the lower limit value of the sixth range is greater than or equal to the upper limit value of the fourth range.

[0233] In this embodiment, when the terminal device detects that the score of the accuracy dimension is in the first range, that is, the decrease in the accuracy dimension is in the first range, or the score of the recall dimension is in the first range, that is, the decrease in the recall dimension is in the first range, or the score of the F1 score dimension is in the first range, that is, the decrease in the F1 score dimension is in the first range, and the delay of the model reasoning speed dimension is in the second range, that is, the delay amplitude of the model reasoning speed dimension is in the second range, it indicates that the performance degradation of the graph attention network at this time is not significant, so the terminal device can determine the incremental learning method as the model optimization method. Among them, the first range and the second range can be set according to actual needs, and there is no limitation here.

[0234] Exemplarily, the first range may be [2%, 3%), and the second range may be [0, 5%).

[0235] In this embodiment, when the terminal device detects that the score of the accuracy dimension is in the third range, that is, the decrease in the accuracy dimension is in the third range, or the score of the recall dimension is in the third range, that is, the decrease in the recall dimension is in the third range, or the score of the F1 score dimension is in the third range, that is, the decrease in the F1 score dimension is in the third range, and the delay of the model reasoning speed dimension is in the fourth range, that is, the delay amplitude of the model reasoning speed dimension is in the fourth range, it indicates that the performance degradation of the graph attention network at this time is significant, so the terminal device can determine the model fine-tuning method as the model optimization method. Among them, the third range and the fourth range can be set according to actual needs, and there is no limitation here.

[0236] Illustratively, the third range may be [3%, 5%], and the fourth range may be [5%, 15%].

[0237] In this embodiment, when the terminal device detects that the score of the accuracy dimension is in the fifth range, that is, the decrease in the accuracy dimension is in the fifth range, or the score of the recall dimension is in the fifth range, that is, the decrease in the recall dimension is in the fifth range, or the score of the F1 score dimension is in the fifth range, that is, the decrease in the F1 score dimension is in the fifth range, and the delay of the model reasoning speed dimension is in the sixth range, that is, the delay amplitude of the model reasoning speed dimension is in the sixth range, it indicates that the performance degradation of the graph attention network at this time is a serious performance degradation. Therefore, the terminal device can determine the model retraining method as the model optimization method. Among them, the fifth range and the sixth range can be set according to actual needs and are not limited here.

[0238] Illustratively, the fifth range may be (5%, +∞), and the sixth range may be (15%, +∞).

[0239] In this embodiment, after determining the model optimization method for the graph attention network, the terminal device can optimize the graph attention network according to the model optimization method to achieve automatic updating of the graph attention network.

[0240] From the above, it can be seen that the data lineage analysis method provided in this embodiment, the terminal device can ensure that the graph attention network can dynamically adapt to changes in query types and data structures, and automatically update the graph attention network, thereby optimizing the accuracy of data lineage analysis and the performance of the graph attention network.

[0241] In another embodiment of the present application, when the query type corresponding to the target query statement is different from the set type, combined with steps S401 to S403, since the optimization of the graph attention network takes a lot of time and the process is cumbersome, in order to improve the efficiency of the data lineage analysis of the target query statement, the terminal device can specifically perform the following steps: Figure 7 Steps S501 to S502 shown determine the data lineage analysis result of the target query statement.

[0242] It should be noted that the target query statement may be the query statement to be queried, or may be a query statement after the query statement to be queried.

[0243] Based on this, see Figure 7 , Figure 7 FIG. 1 is a flowchart of a method for analyzing data lineage provided by another embodiment of the present application. Figure 7 As shown, relative to Figure 1 Corresponding to the embodiment, the data lineage analysis method provided in this embodiment may also include S501 to S502, which are described in detail as follows:

[0244] In S501, if it is detected that the query type corresponding to the target query statement is different from the set type, a historical query statement corresponding to the target query statement is determined from a knowledge base; the knowledge base is used to store the correspondence between different historical query statements and different historical analysis results.

[0245] In S502, the historical analysis result corresponding to the historical query statement is determined as the data lineage analysis result corresponding to the target query statement.

[0246] It should be noted that the terminal device is pre-set with a knowledge base, and the knowledge base is used to store the correspondence between different historical query statements and different historical analysis results, wherein the historical analysis results refer to the data lineage analysis results corresponding to the historical query statements.

[0247] Based on this, in this embodiment, when the terminal device detects that the query type corresponding to the target query statement is different from the set type, in order to improve the work efficiency of data lineage analysis of the target query statement, the terminal device can determine the historical query statement corresponding to the target query statement from the knowledge base.

[0248] In one embodiment of the present application, the terminal device may compare the target query statement with each historical query statement in the knowledge base one by one to determine the historical query statement that is most similar to the target query statement.

[0249] In another embodiment of the present application, in order to improve the accuracy of determining the historical query statement corresponding to the target query statement, the terminal device can specifically implement step S501 according to the following steps, which are described in detail as follows:

[0250] Traversing and calculating the similarity between the target query statement and each of the historical query statements in different detection dimensions;

[0251] If at least one similarity among the similarities of any one of the historical query statements is greater than a set threshold, the any one of the historical query statements is determined as a historical query statement corresponding to the target query statement.

[0252] In this embodiment, the terminal device may traverse and calculate the similarity between the target query statement and each historical query statement in the knowledge base in different detection dimensions.

[0253] Among them, different detection dimensions include but are not limited to: grammatical structure dimension, table and field usage dimension, operation type dimension and query purpose dimension.

[0254] The similarity of the grammatical structure dimension specifically refers to comparing the similarity of the grammatical structures of query statements, such as the composition of SELECT, FROM, and WHERE clauses.

[0255] The similarity of table and field usage dimensions specifically refers to the similarity of tables and fields involved in the query statement.

[0256] The similarity of the operation type dimension specifically refers to the similarity of the set operation types (such as selection, connection, and aggregation, etc.) of the query statements.

[0257] The similarity of the query purpose dimension specifically refers to the similarity of the overall purpose of the query statement (such as data extraction and statistical analysis).

[0258] In this embodiment, after obtaining the similarity between the target query statement and each historical query statement under different detection dimensions, the terminal device can compare the similarity between each historical query statement under different detection dimensions with a set threshold value one by one. The set threshold value can be determined according to actual needs and is not limited here. For example, the set threshold value can be 90%.

[0259] When the terminal device detects that among the similarities of a certain historical query statement under different detection dimensions, there is at least one similarity greater than a set threshold, the terminal device can determine the historical query statement as a historical query statement corresponding to the target query statement.

[0260] In one embodiment of the present application, when the terminal device detects that among the similarities of multiple historical query statements under different detection dimensions, there is at least one similarity greater than a set threshold, the terminal device can determine the historical query statement corresponding to the largest number of similarities greater than the set threshold among the above-mentioned multiple historical query statements as the historical query statement corresponding to the target query statement.

[0261] In this embodiment, after the terminal device determines the historical query statement corresponding to the target query statement in the knowledge base, the terminal device can directly determine the historical analysis result corresponding to the historical query statement as the data lineage analysis result corresponding to the target query statement.

[0262] It can be seen from the above that the data lineage analysis method provided in this embodiment can search for analysis results matching the target query statement through the knowledge base when the query type corresponding to the target query statement is different from the set type, thereby improving the efficiency of data lineage analysis.

[0263] It should be noted that the data lineage analysis methods provided in the above embodiments can be applied to Figure 8 Based on this, please refer to Figure 8 , Figure 8 Schematic diagram of the structure of the data lineage analysis system provided by an embodiment of the present application. Figure 8 As shown, the data lineage analysis system 1 includes: a query acquisition module 10, a query parsing module 20, an AI analysis engine 30, an adaptive learning module 40, a lineage storage module 50, a visualization module 60 and an API interface module 70. The query acquisition module 10 is connected to the query parsing module 20 in communication, the query parsing module 20 is connected to the AI ​​analysis engine 30 in communication, the AI ​​analysis engine 30 is connected to the adaptive learning module 40 and the lineage storage module 50 in communication, and the lineage storage module 50 is connected to the visualization module 60 and the API interface module 70 in communication.

[0264] In this embodiment, the query capture module 10 is used to obtain the SQL query (ie, the query statement) and related metadata (ie, metadata information) executed in the database in real time.

[0265] The query parsing module 20 is used to parse the above SQL query and generate its corresponding abstract syntax tree (AST).

[0266] The AI ​​analysis engine 30 is used to use a graph attention network to analyze the query structure and data flow of the graph structure data constructed via an abstract syntax tree and metadata information to construct a blood relationship, that is, to obtain a data blood relationship analysis result.

[0267] The adaptive learning module 40 is used to continuously learn from new data and user feedback to update the graph attention network.

[0268] The lineage storage module 50 is used to store and manage the generated data lineage analysis results based on the graph database.

[0269] The visualization module 60 is used to visualize and display an interactive blood relationship diagram.

[0270] The API interface module 70 is used to provide a RESTful API so that other systems can access the bloodline data.

[0271] In one embodiment of the present application, see Fig. 9 , Fig. 9 Schematic diagram of the structure of the AI ​​analysis engine provided by an embodiment of the present application. Fig. 9 As shown, the AI ​​analysis engine 30 includes: a query representation learning unit 31, a graph attention network 32, a multi-level lineage reasoning unit 33, an explainability enhancement unit 34 and an adaptive learning unit 35.

[0272] In this embodiment, the query representation learning unit 31 is used to convert SQL queries and metadata information into graph structure representations (such as Figure 1 Step S102 shown).

[0273] The graph attention network 32 is used to analyze the query structure and data flow pattern of the graph structure data to determine the node vector (such as Figure 2 Step S201 shown).

[0274] The multi-level blood relationship reasoning unit 33 is used to construct a blood relationship graph (such as Figure 2 Steps S202 to S203 shown).

[0275] The interpretability enhancement unit 34 is used to generate explanations and evidences corresponding to the blood relationship graph (such as Figure 5 Steps S301 to S303 shown).

[0276] The adaptive learning control unit 35 is used to coordinate the update of the graph attention network (such as Figure 6 Steps S401 to S403 shown).

[0277] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0278] Corresponding to a data lineage analysis method described in the above embodiment, Fig.10 The structure diagram of a data lineage analysis device provided by an embodiment of the present application is shown. For the sake of convenience, only the part related to the embodiment of the present application is shown. Fig.10 The data lineage analysis device 600 includes: an acquisition unit 61, a first construction unit 62 and a first result determination unit 63. Among them:

[0279] The acquisition unit 61 is used to acquire the sentence to be queried and determine metadata information corresponding to the sentence to be queried.

[0280] The first construction unit 62 is used to construct graph structure data based on the query statement and the metadata information; the graph structure data includes multiple nodes and multiple edges, each of the nodes is used to represent a data entity, and each of the edges is used to represent the relationship between data entities.

[0281] The first result determination unit 63 is used to obtain a data lineage analysis result corresponding to the query statement based on the graph structure data and the graph attention network; the graph attention network includes a multi-head attention layer, and the multi-head attention layer is used to capture different types of node relationships.

[0282] In one embodiment of the present application, the first construction unit 62 specifically includes: a parsing unit and a second construction unit.

[0283] The parsing unit is used to parse the sentence to be queried to obtain an abstract syntax tree corresponding to the sentence to be queried.

[0284] The second construction unit is used to construct the graph structure data based on the abstract syntax tree and the metadata information.

[0285] In one embodiment of the present application, the data lineage analysis result is a lineage relationship diagram; the first result determination unit 63 specifically includes: a first processing unit, a first path determination unit and a first generation unit.

[0286] in:

[0287] The first processing unit is used to input the graph structure data into the graph attention network for processing to obtain a node vector corresponding to each node in the graph structure data; the node vector includes feature information, structure information and context information learned in the graph attention network of the corresponding node.

[0288] The first path determination unit is used to determine a data flow path based on each of the node vectors.

[0289] The first generating unit is used to generate the blood relationship diagram based on the data flow path.

[0290] In one embodiment of the present application, the graph attention network includes the input layer, the multi-head attention layer and the multi-layer graph convolution layer; the first processing unit specifically includes: an input unit, a first computing unit and a second processing unit. Among them:

[0291] The input unit is used to input the graph structure data into the input layer to obtain an initial vector corresponding to each of the nodes.

[0292] The first calculation unit is used to input each of the initial vectors into the multi-head attention layer for weight calculation to obtain the attention weight corresponding to each of the initial vectors.

[0293] The second processing unit is used to input each of the initial vectors and the attention weight corresponding to each of the initial vectors into the multi-layer graph convolution layer for processing to obtain a node vector corresponding to each of the nodes.

[0294] In one embodiment of the present application, the data flow path includes target flow paths at different set data granularity levels; the first path determination unit specifically includes: a second calculation unit, a third construction unit, a scoring unit and a second path determination unit. Among them:

[0295] The second calculation unit is used to calculate the similarity between each of the nodes according to each of the node vectors at any set data granularity level.

[0296] The third construction unit is used to construct a plurality of initial flow paths based on the similarities between the nodes and the graph structure data.

[0297] The scoring unit is used to score each of the initial flow paths in different dimensions to obtain a score corresponding to each of the initial flow paths.

[0298] The second path determination unit is used to determine the initial flow path with the largest score as the target flow path at the set data granularity level.

[0299] In one embodiment of the present application, the data lineage analysis result is a lineage relationship diagram, the lineage relationship diagram includes a data flow path, and the data flow path includes multiple data nodes; the data lineage analysis device 600 also includes: a second generation unit.

[0300] The second generating unit is used to generate explanation information corresponding to each of the data nodes; the explanation information is used to describe the relationship between the data node and the result of the data flow path.

[0301] In one embodiment of the present application, the data lineage analysis result is a lineage relationship diagram, the lineage relationship diagram includes a data flow path, and the data flow path includes multiple data nodes; the data lineage analysis device 600 also includes: a node determination unit and a third generation unit. Among them:

[0302] The node determination unit is used to determine a target node in the data flow path that meets a set condition; the set condition is used to characterize that the data node contains a complex operation.

[0303] The third generating unit is used to generate prompt information corresponding to the target node; the prompt information is used to describe the impact of the complex operation on the data flow.

[0304] In one embodiment of the present application, the data lineage analysis device 600 further includes: a score determination unit, a first method determination unit, and an optimization unit.

[0305] The score determination unit is used to determine the performance scores of the graph attention network under different evaluation dimensions if it is detected that the query type corresponding to the query statement is different from the set type, or the data structure is different from the set structure, or the performance evaluation result of the graph attention network is lower than the set result.

[0306] The first mode determination unit is used to determine the model optimization mode based on the scores under the different evaluation dimensions.

[0307] The optimization unit is used to optimize the graph attention network based on the model optimization method to obtain the optimized graph attention network.

[0308] In one embodiment of the present application, the different evaluation dimensions include accuracy dimension, recall dimension, F1 score dimension and model reasoning speed dimension; the first mode determination unit specifically includes: a second mode determination unit, a third mode determination unit and a fourth mode determination unit. Among them:

[0309] The second mode determination unit is used to determine the incremental learning mode as the model optimization mode if the score of the accuracy dimension is within the first range, or the score of the recall dimension is within the first range, or the score of the F1 score dimension is within the first range, and the delay of the model reasoning speed dimension is within the second range.

[0310] The third mode determination unit is used to determine the model fine-tuning mode as the model optimization mode if the score of the accuracy dimension is within the third range, or the score of the recall dimension is within the third range, or the score of the F1 score dimension is within the third range, or the delay of the model reasoning speed dimension is within the fourth range; the lower limit value of the third range is greater than or equal to the upper limit value of the first range, and the lower limit value of the fourth range is greater than or equal to the upper limit value of the second range.

[0311] The fourth mode determination unit is used to determine the model retraining mode as the model optimization mode if the score of the accuracy dimension is in the fifth range, or the score of the recall dimension is in the fifth range, or the score of the F1 score dimension is in the fifth range, and the delay of the model reasoning speed dimension is in the sixth range; the lower limit value of the fifth range is greater than or equal to the upper limit value of the third range, and the lower limit value of the sixth range is greater than or equal to the upper limit value of the fourth range.

[0312] In one embodiment of the present application, the data lineage analysis device 600 further includes: a first statement determination unit and a second result determination unit.

[0313] The first statement determination unit is used to determine the historical query statement corresponding to the target query statement from the knowledge base if it is detected that the query type corresponding to the target query statement is different from the set type; the knowledge base is used to store the correspondence between different historical query statements and different historical analysis results.

[0314] The second result determination unit is used to determine the historical analysis result corresponding to the historical query statement as the data lineage analysis result corresponding to the target query statement.

[0315] In one embodiment of the present application, the first sentence determination unit specifically includes: a third calculation unit and a second sentence determination unit.

[0316] The third calculation unit is used to traverse and calculate the similarity between the target query statement and each of the historical query statements in different detection dimensions.

[0317] The second statement determination unit is used to determine any one of the historical query statements as a historical query statement corresponding to the target query statement if at least one of the similarities of any one of the historical query statements is greater than a set threshold.

[0318] In one embodiment of the present application, the different detection dimensions include: a grammatical structure dimension, a table and field usage dimension, an operation type dimension, and a query purpose dimension.

[0319] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0320] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0321] Fig.11 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Fig.11 As shown, the terminal device 7 of this embodiment includes: at least one processor 70 ( Fig.11 Only one is shown in the figure), a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, wherein the processor 70 implements the steps of any of the above-mentioned data lineage analysis method embodiments when executing the computer program 72.

[0322] The terminal device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Fig.11 It is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0323] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0324] In some embodiments, the memory 71 may be an internal storage unit of the terminal device 7, such as the memory of the terminal device 7. In other embodiments, the memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the terminal device 7. Further, the memory 71 may also include both the internal storage unit of the terminal device 7 and an external storage device. The memory 71 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0325] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0326] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0327] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0328] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0329] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data lineage analysis method, characterized in that: include: Obtaining a query statement and determining metadata information corresponding to the query statement; Based on the query statement and the metadata information, graph structure data is constructed; The graph structure data includes a plurality of nodes and a plurality of edges, each of the nodes is used to represent a data entity, and each of the edges is used to represent a relationship between data entities; Based on the graph structure data and the graph attention network, obtaining a data lineage analysis result corresponding to the query statement; The graph attention network includes a multi-head attention layer, which is used to capture different types of node relationships.

2. The data lineage analysis method according to claim 1, characterized in that: The constructing graph structure data based on the query statement and the metadata information includes: Parsing the query statement to obtain an abstract syntax tree corresponding to the query statement; The graph structure data is constructed based on the abstract syntax tree and the metadata information.

3. The data lineage analysis method according to claim 1, characterized in that: The data lineage analysis result is a lineage relationship graph; the data lineage analysis result corresponding to the query statement is obtained based on the graph structure data and the graph attention network, including: Input the graph structure data into the graph attention network for processing to obtain a node vector corresponding to each of the nodes in the graph structure data; the node vector includes feature information, structure information of the corresponding node and context information learned in the graph attention network; Determine a data flow path based on each of the node vectors; Based on the data flow path, the blood relationship graph is generated.

4. The data lineage analysis method according to claim 3, characterized in that: The graph attention network includes the input layer, the multi-head attention layer and the multi-layer graph convolution layer; the graph structure data is input into the graph attention network for processing to obtain the node vector corresponding to each node in the graph structure data, including: Inputting the graph structure data into the input layer to obtain an initial vector corresponding to each node; Input each of the initial vectors into the multi-head attention layer for weight calculation to obtain the attention weight corresponding to each of the initial vectors; Each of the initial vectors and the attention weight corresponding to each of the initial vectors are input into the multi-layer graph convolution layer for processing to obtain a node vector corresponding to each of the nodes.

5. The data lineage analysis method according to claim 3, characterized in that: The data flow path includes target flow paths at different set data granularity levels; the determining of the data flow path based on each of the node vectors includes: At any set data granularity level, the similarity between each of the nodes is calculated according to each of the node vectors; Based on the similarities between the nodes and the graph structure data, a plurality of initial flow paths are constructed; Scoring each of the initial flow paths in different dimensions to obtain a score corresponding to each of the initial flow paths; The initial flow path with the largest score is determined as the target flow path at the set data granularity level.

6. The data lineage analysis method according to any one of claims 1 to 5, characterized in that: The data lineage analysis result is a lineage relationship diagram, the lineage relationship diagram includes a data flow path, and the data flow path includes a plurality of data nodes; After obtaining the data lineage analysis result corresponding to the query statement based on the graph structure data and the graph attention network, the method further includes: Generate explanation information corresponding to each of the data nodes; the explanation information is used to describe the relationship between the data node and the result of the data flow path; and / or, Determine a target node in the data flow path that meets a set condition; the set condition is used to indicate that the data node contains a complex operation; Prompt information corresponding to the target node is generated; the prompt information is used to describe the impact of the complex operation on data flow.

7. The data lineage analysis method according to any one of claims 1 to 5, characterized in that: The data lineage analysis method further includes: If it is detected that the query type corresponding to the query statement is different from the set type, or the data structure is different from the set structure, or the performance evaluation result of the graph attention network is lower than the set result, then determine the performance score of the graph attention network under different evaluation dimensions; Determine a model optimization method based on the scores under the different evaluation dimensions; The graph attention network is optimized based on the model optimization method to obtain the optimized graph attention network.

8. The data lineage analysis method according to claim 7, characterized in that: The different evaluation dimensions include an accuracy dimension, a recall dimension, an F1 score dimension, and a model reasoning speed dimension; and determining the model optimization method based on the scores under the different evaluation dimensions includes: If the score of the accuracy dimension is within the first range, or the score of the recall dimension is within the first range, or the score of the F1 score dimension is within the first range, and the delay of the model reasoning speed dimension is within the second range, the incremental learning mode is determined as the model optimization mode; If the score of the accuracy dimension is within the third range, or the score of the recall dimension is within the third range, or the score of the F1 score dimension is within the third range, or the delay of the model reasoning speed dimension is within the fourth range, the model fine-tuning mode is determined as the model optimization mode; the lower limit value of the third range is greater than or equal to the upper limit value of the first range, and the lower limit value of the fourth range is greater than or equal to the upper limit value of the second range; If the score of the accuracy dimension is in the fifth range, or the score of the recall dimension is in the fifth range, or the score of the F1 score dimension is in the fifth range, and the delay of the model reasoning speed dimension is in the sixth range, the model retraining mode is determined as the model optimization mode; the lower limit value of the fifth range is greater than or equal to the upper limit value of the third range, and the lower limit value of the sixth range is greater than or equal to the upper limit value of the fourth range.

9. The data lineage analysis method according to any one of claims 1 to 5, characterized in that: The data lineage analysis method further includes: If it is detected that the query type corresponding to the target query statement is different from the set type, a historical query statement corresponding to the target query statement is determined from a knowledge base; the knowledge base is used to store the correspondence between different historical query statements and different historical analysis results; The historical analysis result corresponding to the historical query statement is determined as the data lineage analysis result corresponding to the target query statement.

10. The data lineage analysis method according to claim 9, characterized in that: The determining, from the knowledge base, a historical query statement corresponding to the target query statement includes: Traversing and calculating the similarity between the target query statement and each of the historical query statements in different detection dimensions; If at least one similarity among the similarities of any one of the historical query statements is greater than a set threshold, the any one of the historical query statements is determined as a historical query statement corresponding to the target query statement.

11. The data lineage analysis method according to claim 10, characterized in that: The different detection dimensions include: a grammatical structure dimension, a table and field usage dimension, an operation type dimension, and a query purpose dimension.

12. A data lineage analysis device, characterized in that: include: An acquisition unit, used to acquire a sentence to be queried and determine metadata information corresponding to the sentence to be queried; A first construction unit is used to construct graph structure data based on the query statement and the metadata information; the graph structure data includes a plurality of nodes and a plurality of edges, each of the nodes is used to represent a data entity, and each of the edges is used to represent a relationship between the data entities; A first result determination unit, configured to obtain a data lineage analysis result corresponding to the query statement based on the graph structure data and the graph attention network; The graph attention network includes a multi-head attention layer, which is used to capture different types of node relationships.

13. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data lineage analysis method as described in any one of claims 1 to 11 is implemented.

14. A computer program product, characterized in that It comprises a computer program, which, when executed, implements the data lineage analysis method as claimed in any one of claims 1 to 11.

Images