Data asset-oriented data lineage tracking method and system

Abstract

This invention discloses a method and system for tracing data lineage relationships for data assets. The method includes the following steps: acquiring static metadata, operation logs, and runtime data slices of the data assets; parsing the operation logs to construct an explicit lineage relationship set and calculating a first confidence score based on log completeness; for asset pairs without an explicit lineage relationship, using a multimodal Siamese neural network to integrate semantic features and temporal features of runtime data slices to infer a implicit lineage relationship set and output a second confidence score; finally, fusing explicit and implicit lineage relationships to construct a dynamic lineage graph. This invention achieves low-cost, high-precision lineage relationship tracing in heterogeneous data environments through multimodal feature verification and an adaptive scheduling strategy based on asset value.

Description

Technical Field

[0001] This invention belongs to the field of big data governance and data asset management technology, and specifically relates to a method and system for tracing the data lineage of data assets. Background Technology

[0002] As enterprises deepen their digital transformation, data assets have become a core strategic resource. Data lineage technology aims to record the entire lifecycle of data, from its generation, processing, flow, to its eventual disappearance, forming a visual network similar to a family tree. Precise lineage relationships are the foundation for data quality monitoring, sensitive data tracing, change impact analysis, and compliance auditing.

[0003] Currently, there are two main technical approaches in the field of data lineage analysis: The first type is explicit tracing technology based on metadata and log parsing. For example, Chinese invention patent application CN121456063A primarily deploys a data collection agent to obtain database metadata, ETL (Extract-Transform-Load) scripts, and SQL operation logs. By parsing the logs, table-level or field-level dependencies are extracted, and natural language processing is used to semantically understand the metadata, constructing a static lineage graph. Furthermore, some technologies incorporate algorithms such as PageRank to evaluate node importance.

[0004] The drawback of this approach is its heavy reliance on standardized, structured data processing workflows. In real-world production environments, there are numerous implicit data flows, such as direct copying of underlying files, in-memory computations within applications, implicit API calls, or exporting data from Excel and then importing it into the database. These operations often fail to generate standard SQL logs, resulting in gaps in the lineage graph based on log parsing and an inability to reflect the true overall data picture.

[0005] The second type is implicit data mining techniques based on data content features. For example, Chinese invention patent application CN120410106A does not rely on operation logs but directly analyzes the runtime data stored in the database. The common practice is to extract statistical features or time-series fluctuation features of the data and use neural networks to calculate the similarity or fitting relationship between different data columns. If the fluctuation trends of two data columns are highly consistent, it is inferred that they are related.

[0006] The drawbacks of this method are as follows: For massive data assets, performing deep learning calculations on every pair of fields in the entire database is computationally expensive for enterprises, making real-time updates of the entire dataset difficult. Relying solely on numerical fluctuations can easily lead to spurious correlations. Simple similarity calculations are insufficient to determine data flow or identify derived relationships after complex logical transformations. Summary of the Invention

[0007] This invention provides a method and system for tracking data lineage relationships for data assets. By integrating multi-source operation log parsing and multimodal twin neural network inference technology, and introducing a confidence-based dynamic graph fusion and asset value adaptive verification mechanism, it aims to solve the problems in existing technologies, such as the inability to identify implicit lineages due to missing logs or non-standardized operations, the huge computational resource consumption and susceptibility to false correlations caused by relying solely on data content similarity analysis, and the difficulty of static lineage graphs in reflecting the real-time dynamic flow of data.

[0008] To address the aforementioned technical problems, this invention proposes a method for tracing the lineage of data assets, comprising the following steps: Obtain static metadata, historical operation logs, and runtime data slices of the data assets to be analyzed; The historical operation logs are parsed to identify the operational dependencies between data assets, construct an explicit lineage set, and calculate a first confidence level for each explicit lineage based on the completeness of the log parsing. For data asset pairs without an established explicit blood relationship, candidate association pairs are screened based on the static metadata. The runtime data slices and semantic features of the candidate association pairs are input into a pre-trained neural network to calculate the data behavior similarity, construct a set of implicit blood relationships, and output a second confidence score. The dominant blood relationship set and the recessive blood relationship set are fused to construct a blood relationship graph; the blood relationship edge weights in the graph are dynamically generated based on the first confidence level and the second confidence level, and the blood relationship origin attribute is marked.

[0009] Preferably, the method further includes an adaptive verification strategy based on asset value: The topological importance and business value score of each data asset are calculated based on the aforementioned lineage map. For high-value assets with a business value score higher than the first threshold, a high-frequency deep verification mode is configured, the second confidence calculation process is triggered according to a preset period, and the runtime data slices are sampled at high density before being input into the neural network. For low-value assets with a business value score below the second threshold, configure a low-frequency log trigger mode, which will only trigger the first confidence calculation process when changes are detected in historical operation logs.

[0010] Preferably, the static metadata includes table structure and business terminology descriptions, and the runtime data slice is a set of data samples with time-series characteristics.

[0011] Preferably, the method for calculating the first confidence level is as follows: Initialize the first confidence baseline value to 1.0; If a syntax error occurs during the parsing of the SQL statement in the operation log, but the table name can be extracted through regular expression matching, it is determined to be a downgraded parsing, and the first penalty coefficient is deducted from the baseline value. If the parsed data asset name is ambiguous in the metadata, fuzzy matching is required, and the second penalty coefficient is deducted from the benchmark value. If the parsing results can only determine the dependencies at the data table level, but not the mapping relationships at the specific field level, then the third penalty coefficient is deducted from the baseline value; The baseline value after deducting various penalty coefficients is used as the first confidence level.

[0012] Preferably, the neural network is a multimodal Siamese neural network, and the construction and inference process is as follows: Construct a twin architecture containing two parameter-shared sub-networks, each of which includes a semantic feature extraction channel and a temporal behavior extraction channel; Input the source asset data and target asset data of the candidate association pairs into two sub-networks respectively; The semantic feature extraction channel uses a pre-trained language model to transform business term descriptions into semantic vectors. The temporal behavior extraction channel uses a gated recurrent unit or a long short-term memory network to extract the fluctuation trend fingerprint of runtime data slices; The semantic vector and the fluctuation trend fingerprint are concatenated by the fusion layer. The Euclidean distance or cosine similarity between the source asset and the target asset in the feature space is calculated. If the similarity exceeds the preset threshold, it is determined that there is a hidden blood relationship.

[0013] Preferably, the multimodal twin neural network incorporates a transformation logic recognition mechanism during the training phase: The training sample set contains positive sample pairs and negative sample pairs, where the positive sample pairs contain the original data and the derived data after linear or nonlinear transformation; The neural network is trained to identify source asset data sequences. With target asset data sequence Does a functional relationship exist between them? ,in This includes logic for four arithmetic operations, aggregate calculations, and unit conversions; The second confidence level and its functional relationship The fitting residuals are inversely proportional.

[0014] Preferably, the calculation of the bloodline edge weight incorporates access popularity and bloodline credibility: Set bloodline edge weights ;in, The first or second confidence level is used to characterize the reliability of blood relations; This is an activity metric calculated based on the frequency of runtime data access. These are the preset weighting coefficients; When the bloodline attribute is dominant. The value range is [0.8, 1.0]; when the bloodline origin attribute is recessive, The value is determined by the probability output by the neural network.

[0015] Preferably, the method for screening the candidate association pairs is as follows: Determine if the data types of the fields of the source asset and the target asset are compatible; Calculate the text similarity between field names or annotations of the source asset and the target asset; The correlation between the trend of changes in the number of records of source assets and target assets within a preset time period; Asset pairs with incompatible data types or text similarity and relevance below the initial screening threshold are removed, and the remaining asset pairs are retained as candidate association pairs.

[0016] Preferably, in the process of constructing the kinship map, the method for fusing the dominant kinship set and the recessive kinship set is as follows: Create a complete list containing all undetermined bloodline edges; Traverse the entire set of blood relations. If a blood relation edge exists in both the dominant and recessive blood relation sets, calculate the final weight after merging. , The first and second confidence levels are respectively used; and the source attribute of the bloodline edge is marked as double-confirmed. If a bloodline boundary exists only in the dominant bloodline set, then it is directly taken. Assign the final weight and mark the source attribute as log inference; If a bloodline boundary exists only in the set of recessive bloodlines, then it is directly taken. The final weight is determined, and the source attribute is labeled as behavior inference. If there is an edge in the dominant blood relationship cluster pointing from asset A to asset B, but the behavioral similarity between asset A and asset B inferred from the implicit blood relationship is lower than a preset negation threshold, then the weight of the dominant blood relationship edge is reduced by introducing a decay factor.

[0017] A second aspect of the present invention also proposes a data lineage tracking system for data assets, the system being used to implement the method as described in the first aspect of the present invention, comprising: The asset awareness module is configured to collect static metadata, operation logs, and runtime data slices from heterogeneous data sources. The explicit resolution module is configured to parse operation logs to extract explicit dependency paths and generate explicit lineage relationships with a first confidence level. The implicit reasoning module, with a built-in multimodal twin neural network, is configured to receive semantic features and data slices, infer potential data flow paths by calculating feature spatial distance, and generate implicit kinship relationships with a second confidence level. The genealogy management module is configured to integrate explicit and implicit kinship relationships, construct and store a kinship genealogy with dynamic weights, and dynamically schedule the running frequency of the explicit parsing module and the implicit inference module according to the value level of the data assets.

[0018] Compared with the prior art, the present invention has the following technical effects: 1. The method proposed in this invention combines historical operation logs and runtime data slices to construct explicit and implicit kinship sets respectively; it directly parses the historical operation logs; it introduces a multimodal Siamese neural network for runtime data slices to directly compare the features of the content and behavior of runtime data slices, which can uncover implicit kinship relationships without log records; finally, it integrates and calculates to construct a kinship map, thereby improving the completeness of the kinship map.

[0019] 2. The method proposed in this invention adds a semantic feature extraction channel to the neural network and uses business terms in the metadata to semantically constrain data association. Only under the dual conditions of similar business semantics and consistent data behavior is a blood relationship determined, which significantly reduces the false alarm rate.

[0020] 3. The method proposed in this invention calculates the final weight using a first confidence level and a second confidence level, which not only solves the arbitration problem when there is conflict among multiple sources of evidence, but also provides a precise quantitative basis for subsequent data asset valuation and risk propagation analysis.

[0021] 4. The method proposed in this invention uses an intelligent scheduling computing mode based on the topological importance and business value of assets: high-frequency deep verification is used for core high-value assets, and low-frequency log triggering is used for edge low-value assets. While ensuring real-time monitoring of core links, the overall computing power consumption of the system is significantly reduced. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the data kinship tracing method described in this invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present application and with reference to the accompanying drawings.

[0024] Example 1 This embodiment describes a method for tracing the data lineage of data assets, such as... Figure 1 As shown, it includes the following steps one through four: Step 1: Obtain the static metadata, historical operation logs, and runtime data slices of the data asset to be analyzed. The static metadata includes table structure and business terminology descriptions, while the runtime data slices are collections of data samples with time-series characteristics.

[0025] In practice, this step relies on distributed data collection probes deployed on various data source nodes, such as relational databases like MySQL / Oracle, big data warehouses like Hive / MaxCompute, and file systems like HDFS. The specific implementation process includes the following sub-steps S11–S13: S11: Multi-source heterogeneous connectivity and metadata collection First, establish a connection to the target data source via JDBC, ODBC, or RESTful API interfaces. The data collection probe periodically scans the system tables or metadata management center of the data source to extract static metadata.

[0026] The static metadata specifically includes: Physical schema information: database name, table name, field name, field data type, primary and foreign key constraints, and index information.

[0027] Business semantic description: This includes Chinese annotations for fields, table-level business descriptions, and associated business term tags. For example, the field name "amt" is associated with the business term "transaction amount". This textual information will be used in subsequent steps to generate semantic vectors.

[0028] S12: Capture and Cleaning of Full Operation Logs The system accesses the operation log stream of the data source in real time or near real time.

[0029] The sources of the historical operation logs include, but are not limited to: general query logs and binary logs of the database, as well as execution logs of ETL scheduling tools. After collection, the raw logs are cleaned and filtered to remove operations without actual business meaning, such as heartbeat detection statements and internal system maintenance statements. Only valid log entries containing DDL (Data Definition Language) and DML (Data Manipulation Language) are retained, and uniformly formatted as a five-tuple structure of {timestamp, operating user, source IP, SQL statement / script content, execution time}.

[0030] S13: Sampling and Timing of Runtime Data Slices To support the subsequent neural network inference of latent kinship, this step also needs to acquire the dynamic change features of the data content, i.e., runtime data slices. Since scanning the entire dataset is too costly, this embodiment adopts a head sampling strategy based on time windows. The specific operation is as follows: Select the most recent A time period (e.g., the past 30 days) is used as the observation window. For numerical fields, aggregation is performed at the time granularity to form a time series vector. For non-numeric fields, label encoding or hash encoding is used to convert them into numeric sequences.

[0031] To eliminate the differences in dimensions between different data assets and ensure the convergence efficiency of the neural network, the collected data slices are subjected to min-max normalization. The normalization formula is as follows:

[0032] in: This represents the normalized data point value, with a range of [0,1]. Indicates the original data slice at the 1st Observations at each time point; This represents the minimum value of the data sequence within that time window; This represents the maximum value of the data sequence within that time window.

[0033] Step 2: Parse the historical operation logs, identify the operation dependencies between data assets, construct an explicit lineage set, and calculate the first confidence level for each explicit lineage based on the completeness of the log parsing.

[0034] In this embodiment, the method for calculating the first confidence level is as follows: Initialize the first confidence baseline value to 1.0; If a syntax error occurs during the parsing of the SQL statement in the operation log, but the table name can be extracted through regular expression matching, it is determined to be a downgraded parsing, and the first penalty coefficient is deducted from the baseline value. If the parsed data asset name is ambiguous in the metadata, fuzzy matching is required, and the second penalty coefficient is deducted from the benchmark value. If the parsing results can only determine the dependencies at the data table level, but not the mapping relationships at the specific field level, then the third penalty coefficient is deducted from the baseline value; The baseline value after deducting various penalty coefficients is used as the first confidence level.

[0035] In practice, step two is implemented through the following logical flow: S21: First, try to use an SQL parsing engine to perform lexical and syntactic analysis on the cleaned SQL statements to construct an Abstract Syntax Tree (AST). Parsing engines such as Apache Calcite or ANTLR4 are suitable.

[0036] If the AST is successfully constructed, continue traversing the syntax tree nodes to accurately locate operation commands such as INSERT OVERWRITE and CREATETABLE AS SELECT, extract the source table, target table, and specific field projection transformation relationships, and establish a fine field-level lineage.

[0037] If AST construction fails due to SQL incompatibility or log truncation, predefined regular expressions are used to scan the log text to attempt to extract table names and establish a coarse table-level dependency relationship. In this embodiment, an example of a predefined regular expression is shown below: FROM\s+([a-zA-Z0-9_]+), or JOIN\s+([a-zA-Z0-9_]+).

[0038] S22: After parsing the table name, it needs to be mapped to a unique asset ID in the metadata. If the database name is explicitly specified in the SQL, an exact match is performed directly; if the SQL only contains the table name, and multiple tables with the same name exist in the metadata, it is considered ambiguous. In this case, fuzzy matching is performed based on contextual information in the logs, such as the executing user and the default database for the session.

[0039] S23: In order to scientifically assess the reliability of dominant kinship, this embodiment uses a linear deduction model to calculate the first confidence level. The calculation formula is as follows:

[0040] in, This represents the first confidence level benchmark value, which is set to 1.0 in this embodiment; This indicates a degradation parsing indicator factor. When SQL parsing fails and is converted to regular expression matching, this value is 1; otherwise, it is 0. This represents the first penalty coefficient, corresponding to the risk of downgraded parsing. Since regular expression matching cannot understand SQL logic, the risk is high. Therefore, this embodiment sets... ; This indicates a polysemy indicator. When an asset name is ambiguous and requires fuzzy matching, this value is 1; otherwise, it is 0. This represents the second penalty coefficient, corresponding to the risk introduced by fuzzy matching. In this embodiment, it is set as follows: ; This indicates the granularity indicator. When the parsing result only shows table-level dependencies and lacks field-level mappings, this value is 1; otherwise, it is 0. This represents the third penalty coefficient, corresponding to the uncertainty introduced by coarse-grained processing. In this embodiment, it is set as follows: .

[0041] The following is an example calculation from this embodiment: Suppose the system captures a log entry that fails to be parsed due to its special syntax. The system extracted the table name using regular expressions, but the table name is not unique in the database. Furthermore, regular expressions cannot analyze specific fields. The first confidence level for this dominant kinship is calculated as follows:

[0042] In subsequent steps, a lower first confidence level will indicate that the system is more inclined to rely on the implicit inference results in step three, or trigger an alarm requiring manual confirmation.

[0043] Step 3: For data asset pairs without established explicit kinship, candidate association pairs are screened based on the static metadata. The runtime data slices and semantic features of the candidate association pairs are input into a pre-trained neural network to calculate data behavior similarity, construct a set of implicit kinship relationships, and output a second confidence score.

[0044] The method for screening candidate association pairs in this step is as follows: Determine if the data types of the fields of the source asset and the target asset are compatible; Calculate the text similarity between field names or annotations of the source asset and the target asset; The correlation between the trend of changes in the number of records of source assets and target assets within a preset time period; Asset pairs with incompatible data types or text similarity and relevance below the initial screening threshold are removed, and the remaining asset pairs are retained as candidate association pairs.

[0045] To avoid performing Cartesian product-style deep calculations on the entire data asset, this embodiment first performs a lightweight coarse screening process: Establish a type mapping table that only allows matching of types within the same family. For example, if the source field is of type DECIMAL, the target field can be DECIMAL, DOUBLE, or FLOAT, but not BLOB or DATE. If incompatible, the field should be discarded.

[0046] Extract field names and Chinese annotations, and calculate similarity scores using the edit distance algorithm. .

[0047] Count the total number of rows recorded each day for the past 30 days in two tables, and generate two sequences. and Calculate the Pearson correlation coefficient. .like This indicates that the changes in the data volume of the two tables are highly synchronized.

[0048] If the condition is met: (Type incompatibility) OR ( AND If the condition is met, the asset pair is removed; otherwise, it is added to the candidate associated pair list.

[0049] The neural network is a multimodal twin neural network, and its construction and inference process are as follows: Construct a twin architecture containing two parameter-shared sub-networks, each of which includes a semantic feature extraction channel and a temporal behavior extraction channel; Input the source asset data and target asset data of the candidate association pairs into two sub-networks respectively; The semantic feature extraction channel uses a pre-trained language model to transform business term descriptions into semantic vectors. The temporal behavior extraction channel uses a gated recurrent unit or a long short-term memory network to extract the fluctuation trend fingerprint of runtime data slices; The semantic vector and the fluctuation trend fingerprint are concatenated by the fusion layer. The Euclidean distance or cosine similarity between the source asset and the target asset in the feature space is calculated. If the similarity exceeds the preset threshold, it is determined that there is a hidden blood relationship.

[0050] In practice, the details of this neural network architecture are as follows: Each subnetwork accepts two inputs. Input A is a segmented sequence of business description text, such as transformer, rated, voltage, etc.; Input B is a normalized sequence of runtime data slices, with a shape of T×1, for example, the numerical changes over 30 days, with a shape of 30×1.

[0051] The semantic channel employs the lightweight BERT-Tiny model to extract text features and outputs a semantic vector with a dimension of 1×128. .

[0052] The time-series channel employs a bidirectional GRU (Bi-GRU, Bidirectional Gate Recurrent Unit) network to extract data fluctuation features. The GRU hidden layer has 64 units, and the bidirectional superposition outputs a trend fingerprint with a dimension of 1×128. .

[0053] Feature fusion layer will and The vectors are concatenated to obtain a 1×256 fused vector, which is then compressed into a final 1×64 embedding vector through a fully connected layer.

[0054] In the distance calculation step, let the embedding vector of the source asset be... The embedding vector of the target asset is Calculate the Euclidean distance between the two. The smaller the distance, the more likely the two are to be related by blood. Set a threshold for this determination. ,like If so, then a latent blood relationship is determined to exist; In another embodiment of the invention, the distance calculation step calculates the embedding vector of the source asset. With target asset embedding vector The cosine similarity is then compared with the judgment threshold, and the existence of hidden blood relations is determined based on the comparison result.

[0055] The multimodal twin neural network incorporates a transformation logic recognition mechanism during the training phase: The training sample set contains positive sample pairs and negative sample pairs, where the positive sample pairs contain the original data and the derived data after linear or nonlinear transformation; The neural network is trained to identify source asset data sequences. With target asset data sequence Does a functional relationship exist between them? ,in This includes logic for four arithmetic operations, aggregate calculations, and unit conversions; The second confidence level and its functional relationship The fitting residuals are inversely proportional.

[0056] To enable the model to recognize manipulated lineage relationships, for example: Y=X×1000 represents the unit changing from kilowatt to watt, or In summary, the multimodal twin neural network introduces a transformation logic recognition mechanism during the training phase.

[0057] In the positive sample construction phase, a real baseline data column X is selected, and Y is generated using a pre-defined transformation function library. The function library includes: linear scaling, noise addition, and moving average. It is marked as a positive sample, with Label=1.

[0058] The loss function is a contrastive loss function, and the optimization objective is to minimize the distance between positive sample pairs in the feature space while maximizing the distance between negative sample pairs.

[0059] Euclidean distance output by the network Essentially, it reflects the residuals after the model maps X and Y to the same kinship feature space; the smaller the distance, the better the fit. Therefore, the formula for the second confidence score is defined as:

[0060] Where k is a scaling factor, for example, k=2, and D is the distance output by the network. In another embodiment of the present invention, cosine similarity is used instead of Euclidean distance D in the second confidence calculation formula.

[0061] The calculation of the bloodline edge weight incorporates access popularity and bloodline credibility: Set bloodline edge weights ;in, The first or second confidence level is used to characterize the reliability of blood relations; This is an activity metric calculated based on the frequency of runtime data access. These are the preset weighting coefficients; When the bloodline attribute is dominant. The value range is [0.8, 1.0]; when the bloodline origin attribute is recessive, The value is determined by the probability output by the neural network.

[0062] In practice, each edge in the final kinship graph carries a weight W for subsequent impact analysis.

[0063] Among them, activity index A counts the number of queries on the target table connected to this blood relationship in the past week. Normalization is performed using the Sigmoid function:

[0064] In the formula, and These represent the system's historical average number of visits and standard deviation, respectively. The closer A is to 1, the more popular the data is, and the higher the importance of its lineage.

[0065] For confidence level C, if it is a dominant kinship, C is the value calculated above. The range is 0.8-1.0, depending on the downgrading situation. If it is a recessive lineage, C is the value calculated above. This refers to the confidence level of the neural network output, ranging from 0.0 to 1.0.

[0066] For the weighting coefficients, this embodiment sets , The goal is to balance the accuracy of bloodline records with business activity.

[0067] Step 4: Merge the set of explicit kinship relationships and the set of implicit kinship relationships to construct a kinship graph; the kinship edge weights in the graph are dynamically generated based on the first confidence level and the second confidence level, and the kinship origin attribute is marked.

[0068] The method for fusing the dominant and recessive kinship sets in this step is as follows: Create a complete list containing all undetermined bloodline edges; Traverse the entire set of blood relations. If a blood relation edge exists in both the dominant and recessive blood relation sets, calculate the final weight after merging. , The first and second confidence levels are respectively used; and the source attribute of the bloodline edge is marked as double-confirmed. If a bloodline boundary exists only in the dominant bloodline set, then it is directly taken. Assign the final weight and mark the source attribute as log inference; If a bloodline boundary exists only in the set of recessive bloodlines, then it is directly taken. The final weight is determined, and the source attribute is labeled as behavior inference. If there is an edge in the dominant blood relationship cluster pointing from asset A to asset B, but the behavioral similarity between asset A and asset B inferred from the implicit blood relationship is lower than a preset negation threshold, then the weight of the dominant blood relationship edge is reduced by introducing a decay factor.

[0069] In practice, this step involves the construction and purification of the graph through the following logical flow: First, the dominant kinship set generated in step two (denoted as...) ) and the set of implicit kinship relationships generated in step three (denoted as ) Physical merging is performed. For efficient processing, this embodiment constructs a hash mapping table with the source asset ID and target asset ID as the unique key.

[0070] This complete list For each item in the list, check whether its metadata contains both the log parsing record from step two and the neural network inference record from step three.

[0071] For scenarios involving double confirmation—where logs show blood ties and data behavior is highly similar—this embodiment treats them as two independent sources of evidence verifying the same fact. Therefore, the final weight W is calculated using the following formula:

[0072] This formula ensures that the fused weight W is always greater than or equal to any confidence level. or And the maximum value is no more than 1.0. For example, if the log confidence is 0.8 and the neural network confidence is 0.7, then the fusion weight W = 0.8 + 0.7 - 0.56 = 0.94.

[0073] For scenarios involving explicit and implicit conflicts, where the logs record operations but the data does not actually undergo the expected changes, this often occurs when a zombie script runs without writing data, or when log parsing errors match irrelevant tables. The specific handling logic is as follows: for Each edge in The neural network in step three of the retrieval process The original similarity score. If the original similarity score is less than the preset negation threshold... (In this embodiment, a negative threshold is preset) If so, a weighting reduction is triggered, and attenuation calculations are performed:

[0074] in The decay factor is set to 0.5 in this embodiment. At this point, the source attribute of the edge is corrected and marked as a questionable log entry. This means that although there are logs, the data behavior does not support this lineage, prompting administrators to conduct manual verification.

[0075] After the above calculations, the final lineage relationships are written into the graph database. Nodes in the database store data assets, with attributes including metadata information and business value scores. Edges store lineage relationships, with attributes including: the final calculated lineage... Enumerated values ​​(log derivation, behavior inference, double confirmation or questionable logs), as well as specific SQL statement summaries or similarity fingerprints.

[0076] On the visualization side, the system colors the edges according to the enumeration values. For example, log inference is displayed as a solid blue line, behavior inference is displayed as a dashed green line, double confirmation is displayed as a bold dark line, and questionable logs are displayed as a red warning line, thus intuitively presenting the panoramic link of data assets and their credibility distribution.

[0077] In a preferred embodiment of the present invention, the method further includes an adaptive verification strategy based on asset value: The topological importance and business value score of each data asset are calculated based on the aforementioned lineage map. For high-value assets with a business value score higher than the first threshold, a high-frequency deep verification mode is configured, the second confidence calculation process is triggered according to a preset period, and the runtime data slices are sampled at high density before being input into the neural network. For low-value assets with a business value score below the second threshold, configure a low-frequency log trigger mode, which will only trigger the first confidence calculation process when changes are detected in historical operation logs.

[0078] Specifically, the implementation process begins with grading the value of all data assets and calculating a comprehensive score. The score is based on topological importance. Business value score Weighted average yields:

[0079] Topological importance is calculated iteratively using the PageRank algorithm based on the weighted lineage graph constructed in step four. The higher the in-degree of a node (asset) and the greater the weight W (confidence score after fusion) of the connecting edges, the higher its importance. The higher the value, the more centrally located the asset is in the data link; an error in this location would impact numerous downstream applications. In this embodiment, the weighted calculation... .

[0080] Business value scores are assigned based on asset level tags (such as core, general, or temporary) in static metadata. For example, a core transaction table is assigned a value of 1.0, and a temporary log table is assigned a value of 0.2.

[0081] Next, based on the distribution of the overall scores, thresholds are set respectively. (For example, the percentile value of the top 20% of the total score) and (For example, the percentile ranking of the bottom 50% of the total score). Targeting For high-value assets, ignore log triggers and enforce a time-driven strategy, setting a preset cycle of 1 hour, meaning the implicit inference in step three is executed once per hour. Before executing step three, adjust the runtime data slicing sampling rate in step one. Change the default daily aggregation point to an aggregation point every 5 minutes. High-density data allows neural networks to capture subtle fluctuations, thus enabling them to detect abnormal lineages or data tampering even without operation logs.

[0082] against For low-value assets, time-driven processing is disabled, relying entirely on event-driven processing. The first confidence level is only calculated when step two detects a DDL (e.g., ALTER TABLE) or DML (e.g., INSERT) operation log related to the asset. If no logs are generated, the system does not allocate any computing resources for neural network computation, nor does it sample data slices. For edge assets, this significantly reduces storage and computing overhead, avoiding resource waste.

[0083] For medium-value assets that fall between these two categories, a standard approach is adopted, such as performing a full verification once a day with sampling granularity at the hourly level.

[0084] Example 2 This embodiment is a data lineage tracking system for data assets. The system is used to implement the method described in Embodiment 1, including: The asset awareness module is configured to collect static metadata, operation logs, and runtime data slices from heterogeneous data sources. The explicit resolution module is configured to parse operation logs to extract explicit dependency paths and generate explicit lineage relationships with a first confidence level. The implicit reasoning module, with a built-in multimodal twin neural network, is configured to receive semantic features and data slices, infer potential data flow paths by calculating feature spatial distance, and generate implicit kinship relationships with a second confidence level. The genealogy management module is configured to integrate explicit and implicit kinship relationships, construct and store a kinship genealogy with dynamic weights, and dynamically schedule the running frequency of the explicit parsing module and the implicit inference module according to the value level of the data assets.

[0085] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A method for tracing the lineage of data assets, characterized in that, Includes the following steps: Obtain static metadata, historical operation logs, and runtime data slices of the data assets to be analyzed; The historical operation logs are parsed to identify the operational dependencies between data assets, construct an explicit lineage set, and calculate a first confidence level for each explicit lineage based on the completeness of the log parsing. For data asset pairs without an established explicit blood relationship, candidate association pairs are screened based on the static metadata. The runtime data slices and semantic features of the candidate association pairs are input into a pre-trained neural network to calculate the data behavior similarity, construct a set of implicit blood relationships, and output a second confidence score. The dominant blood relationship set and the recessive blood relationship set are fused to construct a blood relationship graph; the blood relationship edge weights in the graph are dynamically generated based on the first confidence level and the second confidence level, and the blood relationship origin attribute is marked.

2. The method according to claim 1, characterized in that, The method also includes an adaptive verification strategy based on asset value: The topological importance and business value score of each data asset are calculated based on the aforementioned lineage map. For high-value assets with a business value score higher than the first threshold, a high-frequency deep verification mode is configured, the second confidence calculation process is triggered according to a preset period, and the runtime data slices are sampled at high density before being input into the neural network. For low-value assets with a business value score below the second threshold, configure a low-frequency log trigger mode, which will only trigger the first confidence calculation process when changes are detected in historical operation logs.

3. The method according to claim 1, characterized in that, The static metadata includes table structure and business terminology descriptions, and the runtime data slice is a set of data samples with time-series characteristics.

4. The method according to claim 1, characterized in that, The specific method for calculating the first confidence level is as follows: Initialize the first confidence baseline value to 1.0; If a syntax error occurs during the parsing of the SQL statement in the operation log, but the table name can be extracted through regular expression matching, it is determined to be a downgraded parsing, and the first penalty coefficient is deducted from the baseline value. If the parsed data asset name is ambiguous in the metadata, fuzzy matching is required, and the second penalty coefficient is deducted from the benchmark value. If the parsing results can only determine the dependencies at the data table level, but not the mapping relationships at the specific field level, then the third penalty coefficient is deducted from the baseline value; The baseline value after deducting various penalty coefficients is used as the first confidence level.

5. The method according to claim 1, characterized in that, The neural network is a multimodal twin neural network, and its construction and inference process are as follows: Construct a twin architecture containing two parameter-shared sub-networks, each of which includes a semantic feature extraction channel and a temporal behavior extraction channel; Input the source asset data and target asset data of the candidate association pairs into two sub-networks respectively; The semantic feature extraction channel uses a pre-trained language model to transform business term descriptions into semantic vectors. The temporal behavior extraction channel uses a gated recurrent unit or a long short-term memory network to extract the fluctuation trend fingerprint of runtime data slices; The semantic vector and the fluctuation trend fingerprint are concatenated by the fusion layer. The Euclidean distance or cosine similarity between the source asset and the target asset in the feature space is calculated. If the similarity exceeds the preset threshold, it is determined that there is a hidden blood relationship.

6. The method according to claim 5, characterized in that, The multimodal twin neural network incorporates a transformation logic recognition mechanism during the training phase: The training sample set contains positive sample pairs and negative sample pairs, where the positive sample pairs contain the original data and the derived data after linear or nonlinear transformation; The neural network is trained to identify source asset data sequences. With target asset data sequence Does a functional relationship exist between them? ,in This includes logic for four arithmetic operations, aggregate calculations, and unit conversions; The second confidence level and its functional relationship The fitting residuals are inversely proportional.

7. The method according to claim 1, characterized in that, The calculation of the bloodline edge weight incorporates access popularity and bloodline credibility: Set bloodline edge weights ;in, The first or second confidence level is used to characterize the reliability of blood relations; This is an activity metric calculated based on the frequency of runtime data access. These are the preset weighting coefficients; When the bloodline attribute is dominant. The value range is [0.8, 1.0]; when the bloodline origin attribute is recessive, The value is determined by the probability output by the neural network.

8. The method according to claim 1, characterized in that, The method for selecting the candidate association pairs is as follows: Determine if the data types of the fields of the source asset and the target asset are compatible; Calculate the text similarity between field names or annotations of the source asset and the target asset; The correlation between the trend of changes in the number of records of source assets and target assets within a preset time period; Asset pairs with incompatible data types or text similarity and relevance below the initial screening threshold are removed, and the remaining asset pairs are retained as candidate association pairs.

9. The method according to claim 1, characterized in that, In the process of constructing the kinship map, the method for fusing the dominant kinship relationship set and the recessive kinship relationship set is as follows: Create a complete list containing all undetermined bloodline edges; Traverse the entire set of blood relations. If a blood relation edge exists in both the dominant and recessive blood relation sets, calculate the final weight after merging. , The first and second confidence levels are respectively used; and the source attribute of the bloodline edge is marked as double-confirmed. If a bloodline boundary exists only in the dominant bloodline set, then it is directly taken. Assign the final weight and mark the source attribute as log inference; If a bloodline boundary exists only in the set of recessive bloodlines, then it is directly taken. The final weight is determined, and the source attribute is labeled as behavior inference. If there is an edge in the dominant blood relationship cluster pointing from asset A to asset B, but the behavioral similarity between asset A and asset B inferred from the implicit blood relationship is lower than a preset negation threshold, then the weight of the dominant blood relationship edge is reduced by introducing a decay factor.

10. A data lineage tracking system for data assets, characterized in that, The system is used to implement the method as described in any one of claims 1-9, comprising: The asset awareness module is configured to collect static metadata, operation logs, and runtime data slices from heterogeneous data sources. The explicit resolution module is configured to parse operation logs to extract explicit dependency paths and generate explicit lineage relationships with a first confidence level. The implicit reasoning module, with a built-in multimodal twin neural network, is configured to receive semantic features and data slices, infer potential data flow paths by calculating feature spatial distance, and generate implicit kinship relationships with a second confidence level. The genealogy management module is configured to integrate explicit and implicit kinship relationships, construct and store a kinship genealogy with dynamic weights, and dynamically schedule the running frequency of the explicit parsing module and the implicit inference module according to the value level of the data assets.

Images