Method and device for determining blood relationship of data and electronic equipment

Abstract

The invention relates to the technical field of data management, and discloses a method for determining a data blood relationship, which comprises the following steps: acquiring multi-source heterogeneous data; obtaining target metadata corresponding to the multi-source heterogeneous data and a dependency relationship between the target metadata; constructing an initial data blood relationship map by taking the target metadata as a node and the dependency relationship as an edge; obtaining node embedding of each node in the initial data consanguinity map; determining a dependency relationship condition between the candidate node pairs according to node embedding of each node; and obtaining a target data blood relationship map according to the initial data blood relationship map and the dependency relationship condition. According to the method, hidden or unclearly defined hidden data relationships can be found and captured, so that unrecognized dependencies in the initial data blood relationship map are recognized, and the accuracy of determining the data blood relationship is improved. The invention further discloses a device for determining the data blood relationship and electronic equipment.

Description

Technical Field

[0001] The present application relates to the field of data management technology, for example, to a method and device, and electronic equipment for determining data lineage relationships. Background Art

[0002] Data lineage is a key technical approach for describing the entire data process, from generation to processing and application, and its inherent logical connections. It helps enterprises clearly understand data flow paths, thereby achieving efficient data governance, data quality management, anomaly monitoring, and problem tracing. With the widespread adoption of big data, cloud computing, and artificial intelligence, business processes across various industries are becoming increasingly complex, the amount of data involved is growing exponentially, and the flow and processing relationships between data are becoming increasingly intricate. Effectively understanding, accurately retrieving, and monitoring and reviewing data lineage in real time have become crucial technical requirements for ensuring effective data governance and data quality.

[0003] Currently, the industry is widely adopting graph data structures to express data lineage relationships. This approach offers significant advantages in data representation and analysis efficiency over traditional tables or linear structures. Especially when data lineage relationships are extremely complex, graph data structures can more efficiently display data dependency paths and internal logic, providing a clear and intuitive way to express them.

[0004] However, related technologies for identifying and constructing data lineage relationships mostly rely on static definitions and manual maintenance. This approach makes it difficult to discover and capture hidden or unclearly defined implicit data relationships. There are a large number of unidentified lineage paths, resulting in a low accuracy rate in determining data lineage relationships.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0007] The embodiments of the present disclosure provide a method, apparatus, and electronic device for determining data lineage relationships, which can improve the accuracy of determining data lineage relationships.

[0008] In some embodiments, the method includes: obtaining multi-source heterogeneous data; obtaining target metadata corresponding to the multi-source heterogeneous data, and the dependency relationships between the target metadata; constructing an initial data lineage graph using the target metadata as nodes and the dependency relationships as edges; obtaining node embeddings of each node in the initial data lineage graph; determining the dependency relationships between candidate node pairs based on the node embeddings of each node, wherein the candidate node pairs are node pairs without connecting edges in the initial data lineage graph; obtaining a target data lineage graph based on the initial data lineage graph and the dependency relationships.

[0009] In some embodiments, the device includes: a first acquisition module, configured to acquire multi-source heterogeneous data; a second acquisition module, configured to acquire target metadata corresponding to the multi-source heterogeneous data, and the dependency relationship between each target metadata; an initial data lineage map construction module, configured to use the target metadata as nodes and the dependency relationship as edges to construct an initial data lineage map; a node embedding acquisition module, configured to acquire the node embedding of each node in the initial data lineage map; a dependency determination module, configured to determine the dependency relationship between candidate node pairs based on the node embedding of each node, wherein the candidate node pair is a node pair without a connecting edge in the initial data lineage map; a target data lineage map construction module, configured to obtain a target data lineage map based on the initial data lineage map and the dependency relationship.

[0010] In some embodiments, the electronic device includes: a processor and a memory storing program instructions, and the processor is configured to execute the above-mentioned method for determining data lineage when running the program instructions.

[0011] The method, device, and electronic device for determining data lineage relationships provided by the embodiments of the present disclosure can achieve the following technical effects: By using the metadata of multi-source heterogeneous data as nodes and the initially acquired dependencies as edges, an initial data lineage graph is constructed. Based on the node embeddings of each node in the initial data lineage graph, the dependencies between pairs of nodes that do not have edges connecting them in the initial data lineage graph are determined. This allows the discovery and capture of hidden or poorly defined implicit data relationships, identifying dependencies that were not identified in the initial data lineage graph, and thus improving the accuracy of determining data lineage relationships.

[0012] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] One or more embodiments are exemplarily described by corresponding drawings. These exemplary descriptions and drawings do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation. In addition, Figure 1 is a schematic diagram of a method for determining data lineage relationships provided by an embodiment of the present disclosure; Figure 2 is a schematic diagram of a device for determining data lineage relationship provided by an embodiment of the present disclosure; Figure 3 It is a structural diagram of an electronic product provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0015] In the description and claims of the embodiments of the present disclosure, as well as in the accompanying drawings, the terms "first," "second," and the like are used to distinguish similar items and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate to describe the embodiments of the present disclosure herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

[0016] Unless otherwise stated, the term "plurality" means two or more.

[0017] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0018] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0019] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0020] Aiming at the problem of incomplete and inaccurate identification of data lineage relationships. The disclosed embodiment provides a method for determining data lineage relationships. The execution subject is an electronic device, which includes a computer or server, etc. The electronic device constructs an initial data lineage graph by preliminarily extracting the metadata of multi-source heterogeneous data and the dependency relationships between the metadata. A neural network that integrates graph convolution and graph attention mechanism is used to obtain the node embedding of each node in the initial data lineage graph, and then determine the dependency relationship between node pairs that have no connecting edges in the initial data lineage graph. In this way, it is possible to identify unrecognized dependency relationships in the initial data lineage graph, thereby improving the accuracy of determining data lineage relationships.

[0021] Combine Figure 1 As shown, the embodiment of the present disclosure provides a method for determining data lineage relationship, including: Step S101: Acquire multi-source heterogeneous data, wherein the multi-source heterogeneous data is data whose lineage relationship is to be determined.

[0022] Step S102 : acquiring target metadata corresponding to multi-source heterogeneous data and the dependency relationships between target metadata.

[0023] Step S103: construct an initial data lineage graph using the target metadata as nodes and the dependency relationships as edges.

[0024] Step S104: Obtain the node embedding of each node in the initial data bloodline graph.

[0025] Step S105 , determining the dependency relationship between candidate node pairs based on the node embedding of each node, wherein the candidate node pairs are node pairs that have no connecting edges in the initial data lineage graph.

[0026] Step S106: Obtain the target data lineage map based on the initial data lineage map and dependency relationships.

[0027] The method for determining data lineage relationships provided by the embodiments of the present disclosure uses the metadata of multi-source heterogeneous data as nodes and the initially acquired dependency relationships as edges to construct an initial data lineage graph. The method then determines the dependency relationships between pairs of nodes in the initial data lineage graph that do not have edges connecting them, based on the node embeddings of each node in the initial data lineage graph. This method can discover and capture hidden or unclearly defined implicit data relationships, thereby identifying dependencies that were not identified in the initial data lineage graph and improving the accuracy of determining data lineage relationships.

[0028] Optionally, acquiring multi-source heterogeneous data includes collecting text information from multiple data sources and determining the collected text information as multi-source heterogeneous data. The data sources include multiple sources such as SQL data, Python code, log information, and configuration files.

[0029] In some embodiments, SQL data includes one or more of database DDL (Data Definition Language) statements, stored procedures, view definitions, and query logs. DDL statements include CREATE and ALTER statements. SQL data is periodically extracted using database management tools or custom scripts, such as Python's pymysql or cx_Oracle libraries. SQL data is collected using SQL Server system views or Oracle data dictionary tables.

[0030] In some embodiments, the Python code includes one or more of: ETL scripts, database operation code, and data processing flow code. The Python code is obtained through a Git repository interface, log files, or file system scans. The scripts are version controlled to ensure timely updates.

[0031] In some embodiments, log information includes one or more of system logs, ETL logs, and application logs, including operation records, error information, and data transmission records. Log information is captured in real time using log collection tools such as ELK Stack and Fluentd. Regular expressions are used to parse log entries.

[0032] In some embodiments, the configuration file includes one or more of: data source connection information, task dependencies, and scheduling policies defined in formats such as YAML, JSON, and INI. The configuration file is read through a file system API. A version control system, such as Git, is integrated to monitor file changes. A unified character encoding, such as UTF-8, is used for the configuration file to ensure consistency.

[0033] Optionally, after acquiring the multi-source heterogeneous data, the method further includes: preprocessing the multi-source heterogeneous data to obtain preprocessed multi-source heterogeneous data to filter out noise data.

[0034] Specifically, preprocessing multi-source heterogeneous data involves performing text cleaning on the data and converting the cleaned data into a pre-set format to obtain preprocessed data. This cleans the collected text data to filter out noise and ensure that the preprocessed data is free of special characters. The data is then converted into a unified format, making it suitable for model parsing.

[0035] In some embodiments, regular expressions (Regex) are used to perform text cleaning operations on multi-source heterogeneous data. For collected SQL data, Python code, log information, configuration files, and other multi-source heterogeneous data, irrelevant whitespace and invalid information in comments are removed. Keywords such as "CREATE TABLE," "INSERT INTO," "def," and "class" are retained, as are symbols such as brackets and quotation marks. For structured data such as JSON and INI, pre-set parsing libraries, such as Python's json and configparser, are used to convert the data into a dictionary format. Semi-structured text information is segmented, for example, by SQL statement or code function / class.

[0036] In some embodiments, the cleaned multi-source heterogeneous data is uniformly converted into a standard UTF-8 encoded string. For multi-line text information in the multi-source heterogeneous data, a unified line break character, such as: \n, is used to facilitate subsequent large model parsing.

[0037] Furthermore, obtaining target metadata corresponding to multi-source heterogeneous data includes: inputting the multi-source heterogeneous data into a preset large language model to obtain first metadata corresponding to the multi-source heterogeneous data; parsing the multi-source heterogeneous data using a preset parsing strategy to obtain second metadata corresponding to the multi-source heterogeneous data; and fusing the first metadata and second metadata to obtain target metadata corresponding to the multi-source heterogeneous data. The preset parsing strategy is a regular expression and an abstract syntax tree (AST). By fusing the parsing results of the large language model with those of traditional parsing, a more accurate and robust metadata set can be obtained.

[0038] In some embodiments, regular expressions and abstract syntax trees are used to parse and process multi-source heterogeneous data to obtain second metadata corresponding to the multi-source heterogeneous data and dependency relationships between the second metadata.

[0039] Optionally, inputting the multi-source heterogeneous data into a preset large language model includes: concatenating the pre-processed multi-source heterogeneous data to obtain input text information; segmenting the input text information according to an input length threshold of the large language model; and inputting the segmented input text information into the preset large language model.

[0040] In some embodiments, the pre-processed multi-source heterogeneous data is spliced, including: when splicing text information from different data sources, adding preset marking symbols between different types of data sources to distinguish the data sources. The preset marking symbols are, for example:<SQL_START> ","<LOG_START> If the text information of the same data source is long, for example, it exceeds the input length threshold, it is segmented by statement or preset function.

[0041] In some embodiments, the large language model includes a GPT series or a BERT variant. Select a pre-trained large language model , fine-tuning within the domain or utilizing zero-shot prompting strategies to adapt it to data parsing tasks. Large language models utilize chain-of-thought reasoning for step-by-step parsing, improving accuracy in complex scenarios. They also output structured data in a predefined format.

[0042] pass Obtain structured data output by a large language model. Structured data output by large language models, To input text information into the large language model, is the large language model parameter.

[0043] The structured data output by the large language model includes first metadata corresponding to multi-source heterogeneous data and the dependency relationships between the first metadata.

[0044] For example, to use a large language model to parse multi-source heterogeneous data, input the following: Please extract all tables, their field lists and data dependencies from the following SQL data code: [SQL data code] The large language model outputs structured data in a predefined format: {"table name": "xxx", "field": ["a", "b", ...], "dependency": [{"source table": "xxx", "target table": "yyy", "relationship": "insert / select"}]} Among them, "Table Name" and "Field" are the first metadata, and "Dependency" represents the dependency relationship between the first metadata.

[0045] Furthermore, the first metadata and the second metadata are fused to obtain target metadata corresponding to the multi-source heterogeneous data, including: Get the metadata of the i-th target. is the i-th target metadata, is the first metadata of the i-th item, is the i-th second metadata. is the first metadata weight, is the second metadata weight. and The value of is determined by cross-validation, for example, and The best resolution accuracy is achieved when . It is a multidimensional feature vector that contains the semantics, structure, logic and other information of the data entity.

[0046] Furthermore, obtaining the dependency relationship between the target metadata includes: determining the dependency relationship between the first metadata as the dependency relationship between the corresponding target metadata. Alternatively, determining the dependency relationship between the second metadata as the dependency relationship between the corresponding target metadata. Alternatively, determining the dependency relationship between the corresponding target metadata based on the dependency relationship between the first metadata and the dependency relationship between the second metadata.

[0047] For example, the target metadata is a data table or field, and the dependency relationship includes Insert / Select dependency, view dependency, stored procedure dependency, ETL task dependency, operation frequency dependency, task scheduling dependency, or logging dependency.

[0048] In some embodiments, an initial data lineage graph is constructed using target metadata as nodes and dependencies between target metadata as directed edges. Target metadata, obtained by fusing first metadata obtained through a large language model with second metadata obtained through a traditional parsing strategy, serves as a node in the initial data lineage graph. Its node attribute vector incorporates the fusion characteristics of the large language model and the traditional parsing strategy, ensuring that each node has a unique identifier and sufficient attribute information to facilitate subsequent model learning. Furthermore, dependencies between target metadata are used as directed edges to establish associations between data entities.

[0049] Optionally, after constructing the initial data lineage map, the process further includes normalizing the attributes of each node to ensure numerical stability, for example, by using z-score normalization or min-max normalization.

[0050] Optionally, after constructing the initial data lineage graph, the process further includes obtaining the weight of each edge in the initial data lineage graph. Edges with weights less than a set threshold are removed. This optimizes the initial data lineage graph, removes noise, and enhances graph quality. Low-confidence edges are removed to filter out potentially false positives in dependencies.

[0051] Optionally, the weight of each edge in the initial data lineage graph is obtained, including: calculating Get the weight of the edge between the i-th node and the j-th node. is the weight of the edge between the i-th node and the j-th node. The operation frequency is the number of operations performed by one data on another data within a set time. is the total number of operations performed by the kth node on the lth node; The maximum value of the operation frequency. For example, the number of times one data item is inserted into or updated by another data item in a day. is the number of operations performed by the i-th node on the j-th node within the set duration.

[0052] Optionally, obtaining the weight of each edge in the initial data bloodline map includes: normalizing the operation frequency between each node, and determining the normalized operation frequency as the weight of the corresponding edge.

[0053] Optionally, the weight of each edge in the initial data lineage graph is obtained, including: calculating Get the weight of the edge between the i-th node and the j-th node .make is the initial data bloodline map, where A collection of nodes representing data entities, such as tables, files, fields, etc. is a set of directed edges, indicating that “data from node Flow Node "blood relationship. is any node index, indicating the Nodes" to " nodes". Used to enumerate dummy indices of node pairs across the entire graph to calculate the maximum value across the entire graph. The duration is set to a fixed time window for statistical operation frequency and data volume, such as one day or several hours. : The total number of operations performed by the kth node on the lth node within a set duration, such as insert operations, update operations, etc. : The amount of data transmitted from the kth node to the lth node within a set time, for example, the number of records or bytes. Indicates the maximum operation frequency between all node pairs in the graph, used for Perform normalization. Indicates the maximum amount of data between all pairs of nodes in the graph, used for Normalize. α is the operation frequency weight parameter, which is used to control the relative contribution of the “operation frequency” component in the final edge weight, α∈[0,1]. β is the data volume weight parameter, which is used to control the relative contribution of the “data volume” component in the final edge weight, β∈[0,1].

[0054] In this way, the two items are first normalized to , then press Linear weighting is used to reflect the relative contribution of operation frequency and data volume in edge weights and eliminate the impact of dimensional differences on the results.

[0055] Optionally, after constructing the initial data lineage map, the process further includes: further correcting node attributes, detecting abnormal nodes using a clustering algorithm, and adjusting the clustering algorithm based on the mean of similar nodes. For example, the clustering algorithm is K-means.

[0056] Optionally, after constructing the initial data lineage graph, the method further includes: using a graph regularization method to smooth the distribution of nodes and edges, thereby ensuring that the entire initial data lineage graph has better stability and robustness.

[0057] Furthermore, obtaining a node embedding for each node in the initial data lineage graph involves obtaining numerical attribute features, discrete attribute features, and semantic features of each node in the initial data lineage graph. Based on the numerical attribute features, discrete attribute features, and semantic features of each node, an initial feature vector for each node is obtained. Based on the initial feature vector for each node, a node embedding for each node is obtained. This allows for the extraction of high-quality, semantically rich feature representations for each node.

[0058] Furthermore, the numerical attribute characteristics of each node in the initial data lineage map are obtained, including: calculation Get the numerical attribute characteristics of the i-th node. Among them, is the numerical attribute feature of the i-th node, μ is the mean value of all nodes on the corresponding numerical attribute, σ is the standard deviation of all nodes on the corresponding numerical attribute, is the original numerical attribute vector of the i-th node, , is the dimension of the original numerical attribute vector. For example, the original numerical attributes include one or more of data volume, update time, and access frequency. Furthermore, the discrete attribute features of each node in the initial data lineage graph are obtained, including: for the discrete attributes of the i-th node, using one-hot encoding or pre-trained embedding methods, converting them into a continuous vector of a preset fixed dimension: .in, is the discrete attribute feature of the i-th node. The dimension of the discrete attribute feature. In some embodiments, the discrete attribute includes: data table category or business domain code, etc.

[0059] Furthermore, the semantic features of each node in the initial data lineage graph are obtained, including: the structured description information of the i-th node is mapped into the vector space using a large language model or a preset text encoding method: .in, is the semantic feature of the i-th node. This is the dimension of semantic features. For example, the structured description information extracted by the large language model includes "table name", "field list", "dependency", and other structured descriptions.

[0060] Furthermore, based on the numerical attribute features, discrete attribute features, and semantic features of each node, an initial feature vector of each node is obtained, including: concatenating the numerical attribute features, discrete attribute features, and semantic features of each node to obtain an intermediate concatenated vector, and performing dimensionality reduction on the intermediate concatenated vector to obtain the initial feature vector.

[0061] Specifically, by calculating Get the intermediate splicing vector. Among them, is the intermediate splicing vector of the i-th node, [;] represents vector splicing. The dimension of the intermediate splicing vector is .

[0062] Optionally, the intermediate concatenated vector is subjected to dimensionality reduction, including: performing dimensionality reduction on the intermediate concatenated vector through a fully connected layer and introducing a nonlinear activation function, such as a ReLU (Rectified Linear Unit).

[0063] In this dimensionality reduction operation, the intermediate concatenation vector of the i-th node is Apply a layer of full connection + ReLU to get the initial feature vector :

[0064] in, : A trainable weight matrix with dimensions The concatenation vector Linear mapping to dimension Matrix elements Indicates "the input vector Dimension" for "output vector The mapping weight of the "dimension". : A trainable bias vector that adds a bias term to each output dimension. : Non-linear activation function, defined as , used to increase the model's expressiveness and introduce sparsity. That group of " dimensional splicing features are projected onto dimensional" trainable parameter matrix.

[0065] Optionally, after obtaining the initial feature vector, the method further includes normalizing the initial feature vector of each node, so as to ensure the numerical stability of the fused initial feature vector features in the subsequent deep model.

[0066] In some embodiments, the initial feature vector is L2 normalized by calculating Get the normalized initial eigenvector of the i-th node .

[0067] In this way, the numerical, discrete and semantic information can be directly integrated to ensure that the initial feature vector of the node contains both entity information and additional fine-grained features, which can provide sufficient and accurate input for obtaining the node embedding of the node.

[0068] Furthermore, based on the initial feature vector of each node, a node embedding for each node is obtained, including: updating the initial feature vector of each node using a preset hybrid graph neural network encoder to obtain the node embedding of each node. The hybrid graph neural network encoder includes GCN (Graph Convolution Neural Networks) and GAT (Graph Attention Network). Specifically, the initial feature vector of each node is first updated using GCN, followed by GAT. This simultaneously leverages the local structure capture capabilities of GCN and the multi-head attention advantage of GAT. It can capture the local structural information between nodes in the graph while highlighting key dependencies using the multi-head attention mechanism, resulting in high-quality node embeddings.

[0069] In some embodiments, during the update process using GCN, the update formula of the graph convolutional neural network is: .in, In the GCN update process, the i-th node is l +1 layer of feature representation, l ≥0, For the i-th node in l The feature representation of the layer, N(i) is the neighbor set of the i-th node, and does not include the i-th node, is the normalization constant, , is the weight matrix of the lth layer; is a non-linear activation function. Use Xavier or He initialization. The nonlinear activation function is ReLU function.

[0070] The feature representation of the i-th node at layer 0 is: .in, is the feature representation of the i-th node at layer 0, is the initial eigenvector of the i-th node, is the dimension of the initial feature vector of the i-th node.

[0071] The hybrid graph neural network encoder first performs GCN update and then GAT update. Specifically, by calculating , get the node embedding of the i-th node. is the node embedding of the i-th node.

[0072] Among them, during the GAT update process, in the l The layer uses K attention heads, , calculate for each head: ,in, For the l The feature representation of the i-th node in the k-th head of the layer, is the weight matrix of the k-th head in the l-th layer, is the attention coefficient of the i-th node in the k-th head of the l-th layer to its j-th neighbor node.

[0073] The final output of each node in the GAT update part is the concatenation of all headers: ,in, is the feature representation of the i-th node in the l+1 layer during the GAT update process.

[0074] Among them, by calculating Get the attention coefficient of the i-th node in the k-th head of the l-th layer to its j-th neighbor node, where: is the attention weight vector of the k-th head in the l-th layer, ∥ represents vector concatenation, is the LeakyReLU activation function, whose negative semi-axis slope is 0.2.

[0075] In some embodiments, in order to prevent gradient disappearance and overfitting, Dropout and residual connections are introduced in the update of each layer. l The output of the layer is randomly deactivated (Dropout). The Dropout rate is set to 0.5. That is, .

[0076] like and If the dimensions are the same, add them together: Otherwise, it can be adjusted by linear mapping Dimensions are then added together.

[0077] Among them, all weight matrices, such as 、 Both use Xavier or He initialization method. Taking Xavier initialization as an example, its initialization scheme is: the weights obey the mean of 0 and the variance of The distribution of is the input dimension, is the output dimension.

[0078] This hybrid graph neural network encoder design leverages GCN to capture local structural information, leverages GAT multi-head attention to highlight key dependencies, and introduces dropout and residual connections to enhance training stability. The resulting node embeddings provide high-quality input for subsequent link prediction and anomaly detection tasks.

[0079] Furthermore, the dependency relationships between candidate node pairs are determined based on the node embeddings of each node, including: calculating the similarity between candidate node pairs based on the node embeddings of each node. Based on the similarity, determining whether a dependency relationship exists between the corresponding candidate node pairs. In this way, the similarity between candidate node pairs represents the probability that a dependency relationship exists between the candidate node pairs, and can more accurately reflect whether a dependency relationship exists between the candidate node pairs, thereby enabling the timely discovery of hidden data relationships between the candidate node pairs. This allows the ultimately generated target data lineage map to more accurately and comprehensively reflect the actual dependency relationships between the data.

[0080] Optionally, calculating the similarity between candidate node pairs based on the node embeddings of each node includes: calculating the node embeddings of each node using a preset link prediction model to obtain the similarity between the candidate node pairs. The preset link prediction model includes a simple inner product model or a multi-layer perceptron (MLP) model.

[0081] In some embodiments, when the link prediction model is a simple inner product model, by calculating Get the similarity between candidate node pairs. is the similarity between the p-th candidate node and the q-th candidate node. is the node embedding of the p-th candidate node, is the node embedding of the qth candidate node, and T represents the transpose operation. The sigmoid function is used to map the similarity between candidate node pairs to the interval [0, 1] as the probability that there is a dependency relationship between the candidate node pairs.

[0082] In some embodiments, when the link prediction model is a multi-layer perceptron model, by calculating Get the similarity between candidate node pairs. For the p candidate nodes and q The similarity between candidate nodes. Indicates that the p candidate nodes andq The node embedding vectors of candidate nodes are concatenated into a 2D vector. is the trainable weight vector, is the bias term.

[0083] Optionally, determining whether a dependency relationship exists between the corresponding candidate node pairs based on the similarity between the candidate node pairs includes: determining that a dependency relationship exists between the corresponding candidate node pairs when the similarity between the candidate node pairs is greater than a similarity threshold. For example, the similarity threshold is 0.8. When the similarity between the candidate node pairs is less than or equal to the similarity threshold, determining that no dependency relationship exists between the corresponding candidate node pairs.

[0084] In some embodiments, the parameters of the link prediction model are optimized. Specifically, positive sample data and negative sample data are obtained. By constructing positive and negative sample data and training with a loss function, the link prediction model parameters are optimized so that the link prediction model can accurately predict missing dependencies. Positive sample data are node pairs with connected edges in the initial data lineage graph, and negative sample data are node pairs without connected edges in the initial data lineage graph. In this way, by adopting a negative sampling strategy, the number of positive and negative samples is balanced to prevent training instability.

[0085] Using the binary cross entropy loss function, by calculating Obtaining the loss function .in, For the s The connection node and r The similarity between connected nodes is the similarity between connected nodes, where connected nodes are nodes with connected edges in the initial data lineage graph. E is the set of nodes with connected edges in the initial data lineage graph. For the p candidate nodes and q The similarity between candidate nodes. It is the set of nodes without connecting edges in the initial data bloodline graph.

[0086] Use Adam optimizer for back propagation training according to the loss function Adjust the parameters of the link prediction model. And, perform backpropagation training using the Adam optimizer according to the loss function Adjust the parameters of the hybrid graph neural network encoder. The initial learning rate of the Adam optimizer is set to 0.001.

[0087] Furthermore, based on the initial data lineage graph and the dependency relationships, a target data lineage graph is obtained. This includes: if dependencies exist between candidate node pairs, the dependencies between the candidate node pairs are used as edges corresponding to the candidate node pairs in the initial data lineage graph to obtain the target data lineage graph. The updated target data lineage graph can more accurately and comprehensively reflect the actual dependencies between the data.

[0088] Furthermore, after obtaining the target data lineage map, it also includes: reconstructing the connection relationship between the nodes in the target data lineage map, and obtaining the reconstruction error of the target node pair. The target node pair is all the node pairs in the target data lineage map. According to the reconstruction error of the target node pair, the abnormality score of the target node pair is obtained. According to the abnormality score of the target node pair, it is determined whether the corresponding target node pair is abnormal. In this way, using the updated target data lineage map to perform abnormality detection on the target node can make the detection of whether the target node is abnormal more accurate.

[0089] In some embodiments, a graph autoencoder (GAE) is used to reconstruct the connection relationship between nodes in the target data lineage graph.

[0090] Using the preset hybrid graph neural network encoder as the encoder, the input feature vector of each target node passes through a multi-layer GCN / GAT network to generate the final target node embedding vector: .in, is the embedding vector of the i-th target node. L is the number of layers of the hybrid graph neural network encoder.

[0091] In the decoder process, according to the node embedding Reconstruct the adjacency matrix of the target data's lineage graph. Using the inner product decoder: .in, is the probability of connection between the i-th target node and the j-th target after reconstruction, As the activation function, the sigmoid function is used to map the inner product result to the interval [0,1].

[0092] The mean square error (MSE) is used as the reconstruction loss: .in, is the probability of connection between the i-th target node and the j-th target in the target data lineage graph before reconstruction. This loss reflects the difference between the reconstructed matrix and the true adjacency matrix. The goal of model training is to minimize this reconstruction error so that the final target node embedding vector can fully capture the structural information of the graph.

[0093] Furthermore, the reconstruction error of the target node pair is obtained by calculating Obtain the reconstruction error of the target node pair. is the reconstruction error between the i-th target node and the j-th target node.

[0094] Furthermore, according to the reconstruction error of the target node pair, the abnormality score of the target node pair is obtained, including: by calculating Get the anomaly score of the target node pair. is the anomaly score between the i-th target node and the j-th target node, is the mean reconstruction error of the target node pair, is the standard deviation of the reconstruction error for the target node pair.

[0095] Furthermore, determining whether the corresponding target node pair is abnormal based on the anomaly score of the target node pair includes: if the anomaly score of the target node pair is greater than a first anomaly threshold, determining that the corresponding target node pair is abnormal. That is, determining that both corresponding target nodes are abnormal. For example, the first anomaly threshold is 0.8.

[0096] The reconstruction error is used to perform an anomaly score on each target node pair. When the score exceeds the first anomaly threshold, the corresponding target node is judged to be an anomaly. This enables the detection of abnormal relationships in the target data lineage map.

[0097] Furthermore, after obtaining the target data lineage map, the method further includes: reconstructing the edges in the target data lineage map to obtain the reconstruction error of the target edges. The target edges are all the edges in the target data lineage map. Based on the reconstruction error of the target edge, the anomaly score of the target edge is obtained. Based on the anomaly score of the target edge, it is determined whether the corresponding target edge is abnormal. In this way, using the reconstruction error to perform anomaly detection on the target edge of the updated target data lineage map can make the detection of whether the dependency relationship between the target nodes is abnormal more accurate.

[0098] In some embodiments, a graph autoencoder (GAE) is used to reconstruct target edges in the target data lineage graph. The node embeddings of each target node are used to reconstruct the edge weights using a simple inner product model or an MLP model.

[0099] Furthermore, the reconstruction error of the target edge is obtained by calculating Obtain the reconstruction error of the target edge between the i-th target node and the j-th target node. is the reconstruction error of the target edge between the i-th target node and the j-th target node, is the weight of the target edge between the i-th target node and the j-th target in the target data lineage graph before reconstruction, is the weight of the target edge between the i-th target node and the j-th target after reconstruction.

[0100] Furthermore, according to the reconstruction error of the target edge, the abnormality score of the target edge is obtained, including: by calculating Get the anomaly score of the target edge. is the anomaly score of the target edge between the i-th target node and the j-th target node. is the mean reconstruction error of the target edge, The standard deviation of the reconstruction error for the target edge.

[0101] Further, according to the abnormality score of the target edge, determining whether the corresponding target edge is abnormal includes: if the abnormality score of the target edge is greater than the second abnormality threshold, determining that the corresponding target edge is abnormal. For example, the second abnormality threshold is 0.8. When >0.8, it is determined that there is an abnormality in the dependency relationship between the i-th target node and the j-th target node.

[0102] In this way, by calculating and standardizing the reconstruction error of the edge, we can effectively identify abnormal edges that deviate significantly from the overall data flow pattern, thereby clarifying the propagation path and impact range of the anomaly in the target data lineage map.

[0103] The method for determining data lineage relationships provided by the disclosed embodiments enables real-time and accurate monitoring and identification of abnormal nodes or relationships within a data lineage graph. Compared to methods that rely on manual inspection or simple rule matching, the method for detecting abnormal nodes or relationships in the disclosed embodiments is more intelligent, enabling real-time perception and automatic analysis of abnormalities in data lineage networks.

[0104] Furthermore, after determining whether the corresponding target node pair is abnormal based on the anomaly score of the target node pair, the method further includes: extracting the local neighborhood subgraph of the abnormal node in the target data lineage map. The abnormal node is an abnormal target node in the target data lineage map. The betweenness centrality of the abnormal node is obtained. Based on the betweenness centrality of the abnormal node, the influence of the abnormal node in the local neighborhood subgraph is determined. This makes it easier for users to clearly identify the position of the abnormal node in the target data lineage map and the possible impact range.

[0105] In some embodiments, a preset local subgraph extraction algorithm is used to extract the local neighborhood subgraph of the abnormal node in the target data lineage graph. The local subgraph extraction algorithm includes a depth-first search (DFS) or breadth-first search (BFS) algorithm. For example, the local neighborhood subgraph of the abnormal node in the target data lineage graph is: Among them, the neighborhood radius is set to 1-hop or 2-hop), ensuring that the extraction range covers the direct neighbors of the abnormal node and retains sufficient context information for subsequent analysis.

[0106] Optionally, obtain the betweenness centrality of the abnormal node, including: by calculating Get the betweenness centrality of abnormal nodes, where is the betweenness centrality of the Vth abnormal node in the local neighborhood subgraph, is the number of all shortest paths from the sth node to the tth node in the local neighborhood subgraph, is the number of shortest paths passing through the vth abnormal node. The betweenness centrality of an abnormal node can be used to evaluate the importance of the node as an information transmission bridge in the local neighborhood subgraph.

[0107] Furthermore, based on the betweenness centrality of the abnormal node, the influence of the abnormal node in the local neighborhood subgraph is determined, including: when the betweenness centrality of the abnormal node is greater than or equal to the influence threshold, the influence of the abnormal node in the local neighborhood subgraph is determined to be relatively large. When the betweenness centrality of the abnormal node is less than the influence threshold, the influence of the abnormal node in the local neighborhood subgraph is determined to be relatively small. In this way, the relative importance of the abnormal node in the local neighborhood subgraph can be analyzed. For example, a high betweenness centrality of the abnormal node may indicate that the abnormal node plays a key role in the data flow transmission process, and its anomaly may cause a large-scale impact.

[0108] In some embodiments, after obtaining the betweenness centrality of the abnormal node, the method further includes calculating a network indicator clustering coefficient. Alternatively, the number of direct connections of the abnormal node or the closeness of the distance between the abnormal node and other nodes is obtained. This measures the ability of the abnormal node to form a close group in the local network.

[0109] Optionally, after determining the influence of the abnormal node in the local neighborhood subgraph, the method further includes outputting the analysis results in the form of a structured report. The analysis results include the local neighborhood subgraph of the abnormal node and the betweenness centrality of the abnormal node. The analysis results also include calculating network indicators such as the clustering coefficient, the number of direct connections of the abnormal node, and the closeness of the abnormal node to other nodes.

[0110] Furthermore, after determining whether the target edge is anomalous based on its anomaly score, the process also includes extracting the local neighborhood subgraph of the anomalous edge in the target data lineage graph. The anomalous edge is an edge that is anomalous in the target data lineage graph. The betweenness centrality of the anomalous edge is obtained. Based on the betweenness centrality of the anomalous edge, the influence of the anomalous edge in the local neighborhood subgraph is determined. This allows the user to easily identify the location of the anomalous edge in the target data lineage graph and its potential impact range.

[0111] In some embodiments, a preset local subgraph extraction algorithm is used to extract the local neighborhood subgraph of the abnormal edge in the target data lineage map. When the betweenness centrality of the abnormal edge is greater than or equal to the influence threshold, the influence of the abnormal edge in the local neighborhood subgraph is determined to be relatively large. When the betweenness centrality of the abnormal edge is less than the influence threshold, the influence of the abnormal edge in the local neighborhood subgraph is determined to be relatively small. In this way, the relative importance of the abnormal edge in the local neighborhood subgraph can be analyzed. For example, a higher betweenness centrality on the abnormal edge may indicate that the abnormal edge plays a key role in the data flow transmission process, and its abnormality may cause a large-scale impact.

[0112] Optionally, after determining the influence of the abnormal edge in the local neighborhood subgraph, the method further includes: outputting the analysis results in the form of a structured report, wherein the analysis results include the local neighborhood subgraph of the abnormal edge and the betweenness centrality of the abnormal edge.

[0113] Combine Figure 2 As shown, an embodiment of the present disclosure provides a device for determining data lineage relationships, including: a first acquisition module 201, a second acquisition module 202, an initial data lineage graph construction module 203, a node embedding acquisition module 204, a dependency determination module 205, and a target data lineage graph construction module 206. The first acquisition module 201 is configured to acquire multi-source heterogeneous data. The second acquisition module 202 is configured to acquire target metadata corresponding to the multi-source heterogeneous data, and the dependency relationships between the target metadata. The initial data lineage graph construction module 203 is configured to construct an initial data lineage graph using the target metadata as nodes and the dependency relationships as edges. The node embedding acquisition module 204 is configured to acquire the node embeddings of each node in the initial data lineage graph. The dependency determination module 205 is configured to determine the dependency relationships between candidate node pairs based on the node embeddings of each node, wherein the candidate node pairs are node pairs without connecting edges in the initial data lineage graph. The target data lineage map construction module 206 is configured to obtain a target data lineage map according to the initial data lineage map and the dependency relationship.

[0114] Optionally, the apparatus for determining data lineage relationship further includes: a detection module 207 and an impact determination module 208 .

[0115] Optionally, the detection module 207 is configured to reconstruct the connection relationship between the nodes in the target data lineage map, obtain the reconstruction error of the target node pair, wherein the target node pair is all the node pairs in the target data lineage map; obtain the abnormality score of the target node pair based on the reconstruction error of the target node pair; and determine whether the corresponding target node pair is abnormal based on the abnormality score of the target node pair.

[0116] Optionally, the detection module 207 is configured to reconstruct the edges in the target data lineage map and obtain the reconstruction error of the target edge, wherein the target edge is all the edges in the target data lineage map; obtain the anomaly score of the target edge based on the reconstruction error of the target edge; and determine whether the corresponding target edge is abnormal based on the anomaly score of the target edge.

[0117] Optionally, the impact determination module 208 is configured to extract a local neighborhood subgraph of the abnormal node in the target data lineage map, wherein the abnormal node is an abnormal target node in the target data lineage map; obtain the betweenness centrality of the abnormal node; and determine the impact of the abnormal node in the local neighborhood subgraph based on the betweenness centrality of the abnormal node.

[0118] Optionally, the impact determination module 208 is configured to extract a local neighborhood subgraph of the anomalous edge in the target data lineage graph. The anomalous edge is an edge that is abnormal in the target data lineage graph. The betweenness centrality of the anomalous edge is obtained. Based on the betweenness centrality of the anomalous edge, the impact of the anomalous edge in the local neighborhood subgraph is determined.

[0119] Combine Figure 3 As shown, an embodiment of the present disclosure provides an electronic device 300, including a processor 304 and a memory 301 storing program instructions. Optionally, the device may also include a communication interface 302 and a bus 303. The processor 304, the communication interface 302, and the memory 301 may communicate with each other via the bus 303. The communication interface 302 may be used for information transmission. The processor 304 may call the program instructions in the memory 301 to execute the method for determining data lineage relationships of the above embodiment.

[0120] In addition, the logic instructions in the memory 301 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0121] Memory 301, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present disclosure. Processor 304 executes the program instructions / modules stored in memory 301 to perform functional applications and data processing, thereby implementing the method for determining data lineage relationships in the above-mentioned embodiments.

[0122] The memory 301 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal device. Furthermore, the memory 301 may include high-speed random access memory and non-volatile memory.

[0123] The technical solutions of the embodiments of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, and other media that can store program code, or a transient storage medium.

[0124] The above description and the accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible variations. Unless expressly required, individual components and functions are optional, and the order of operations may vary. Portions and features of some embodiments may be included in or replace portions and features of other embodiments. Moreover, the terms used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, the singular forms "a", "an" and "the" are intended to also include the plural forms unless the context clearly indicates otherwise. Similarly, the term "and / or" as used in this application means any and all possible combinations of one or more of the associated listings. In addition, when used in this application, the term "comprise" and its variations "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, an element defined by the sentence "comprising a..." does not exclude the presence of other identical elements in the process, method or device that includes the element. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments can be referenced to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can be referred to the description of the method part.

[0125] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0126] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units may be merely a logical functional division. In actual implementation, other divisions may be used, such as combining or integrating multiple units or components into another system, or omitting or disabling some features. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be through interfaces, indirect couplings or communication connections between devices or units, and may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of these units may be selected to implement the embodiments according to actual needs. Furthermore, the functional units in the disclosed embodiments may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0127] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for determining data lineage, characterized in that: include: Acquire multi-source heterogeneous data; Obtaining target metadata corresponding to the multi-source heterogeneous data and dependencies between target metadata; Using the target metadata as nodes and the dependency relationships as edges, an initial data lineage graph is constructed; Obtaining node embeddings of each node in the initial data lineage graph; Determine the dependency relationship between candidate node pairs based on the node embedding of each node, wherein the candidate node pairs are node pairs that have no connecting edges in the initial data lineage graph; According to the initial data lineage map and the dependency relationship, a target data lineage map is obtained.

2. The method according to claim 1, characterized in that The acquiring target metadata corresponding to the multi-source heterogeneous data includes: Inputting the multi-source heterogeneous data into a preset large language model to obtain first metadata corresponding to the multi-source heterogeneous data; Parsing the multi-source heterogeneous data using a preset parsing strategy to obtain second metadata corresponding to the multi-source heterogeneous data; The first metadata and the second metadata are fused to obtain target metadata corresponding to the multi-source heterogeneous data.

3. The method according to claim 1, characterized in that The obtaining of node embeddings of each node in the initial data bloodline graph includes: Obtaining numerical attribute features, discrete attribute features, and semantic features of each node in the initial data lineage graph; Obtaining an initial feature vector of each node according to the numerical attribute feature, discrete attribute feature and semantic feature of each node; A node embedding of each node is obtained according to the initial feature vector of each node.

4. The method according to claim 1, wherein The step of determining the dependency relationship between candidate node pairs based on the node embedding of each node includes: Based on the node embedding of each node, the similarity between candidate node pairs is calculated; According to the similarity, it is determined whether there is a dependency relationship between corresponding candidate node pairs.

5. The method according to claim 1, wherein Obtaining a target data lineage map according to the initial data lineage map and the dependency relationship includes: In the case where there is a dependency relationship between candidate node pairs, in the initial data lineage graph, the dependency relationship between the candidate node pairs is used as the edge corresponding to the candidate node pairs to obtain the target data lineage graph.

6. The method according to any one of claims 1 to 5, characterized in that After obtaining the target data bloodline map, it also includes: Reconstructing the connection relationship between nodes in the target data lineage graph to obtain the reconstruction error of the target node pair, wherein the target node pair is all the node pairs in the target data lineage graph; Obtaining an anomaly score of the target node pair according to a reconstruction error of the target node pair; According to the abnormality score of the target node pair, it is determined whether the corresponding target node pair is abnormal.

7. The method according to any one of claims 1 to 5, characterized in that After obtaining the target data bloodline map, it also includes: Reconstructing edges in the target data lineage graph to obtain reconstruction errors of target edges, wherein the target edges are all edges in the target data lineage graph; Obtaining an anomaly score of the target edge according to a reconstruction error of the target edge; According to the anomaly score of the target edge, determine whether the corresponding target edge is abnormal.

8. The method according to claim 6, characterized in that After determining whether the corresponding target node pair is abnormal according to the abnormality score of the target node pair, the method further includes: Extracting a local neighborhood subgraph of an abnormal node in a target data lineage graph, wherein the abnormal node is an abnormal target node in the target data lineage graph; Obtaining the betweenness centrality of the abnormal node; According to the betweenness centrality of the abnormal node, the influence of the abnormal node in the local neighborhood subgraph is determined.

9. A device for determining data lineage relationship, characterized in that: include: A first acquisition module is configured to acquire multi-source heterogeneous data; A second acquisition module is configured to acquire target metadata corresponding to the multi-source heterogeneous data and dependency relationships between target metadata; An initial data lineage graph construction module is configured to construct an initial data lineage graph using target metadata as nodes and the dependency relationships as edges; a node embedding acquisition module configured to acquire the node embedding of each node in the initial data lineage graph; A dependency determination module is configured to determine the dependency between candidate node pairs based on the node embeddings of each node, wherein the candidate node pairs are node pairs that have no connecting edges in the initial data lineage graph; The target data lineage map construction module is configured to obtain the target data lineage map according to the initial data lineage map and the dependency relationship.

10. An electronic device comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the method for determining data lineage relationship according to any one of claims 1 to 8 when running the program instructions.

Images