A data bloodline confidence checking method

Abstract

The present application relates to the technical field of data governance and big data processing, and discloses a data blood relationship confidence checking method, comprising: asynchronously collecting metadata of multi-source heterogeneous systems, and hierarchically analyzing and constructing an initial blood relationship graph; calculating the comprehensive confidence score of each blood relationship edge based on a multi-dimensional feature index and a confidence evaluation model, screening low-confidence edges to form a list to be checked; performing at least two kinds of collaborative checking means, such as cross-source active detection, AI semantic reasoning and manual visual labeling, on the blood relationship edges to be checked, and triggering them in sequence; writing the checking results back to the graph, updating the topological structure and the confidence state, and iteratively performing scoring, screening and checking until the preset closed-loop termination condition is met. The method can realize high-precision, interpretable and self-optimized blood relationship governance.

Description

Technical Field

[0001] This invention relates to the field of data governance and big data processing technology, and more specifically, to a data lineage confidence verification method. Background Technology

[0002] In industrial internet platforms, data lineage management is a key technology for achieving full data lifecycle tracking, quality traceability, impact analysis, and compliance auditing. However, existing data lineage management solutions generally adopt a unidirectional open-loop architecture of "collection—parsing—storage—display," which presents significant technical bottlenecks.

[0003] First, due to the complexity of ETL logic in industrial scenarios (such as dynamic SQL and cross-language scripts), the accuracy of mainstream lineage resolution tools is not high, resulting in a large number of erroneous, missing, or redundant lineage relationships in the initially constructed lineage graph. Second, existing solutions lack an automatic verification mechanism and cannot dynamically correct the lineage relationships already written into the graph. When the source data structure changes, the ETL script is modified, or the initial parsing is incorrect, the system relies entirely on manual verification of each relationship, which is costly and impractical on medium-sized platforms with hundreds of thousands of lineage edges. Finally, due to the lack of quantitative assessment of lineage reliability, the data aggregation engine cannot dynamically adjust its strategy based on the lineage status, causing erroneous data to continue to spread downstream.

[0004] Therefore, there is an urgent need for a closed-loop mechanism that can automatically detect low-confidence kinship relationships and trigger verification and correction to ensure that the kinship map remains in a high-confidence state throughout its entire life cycle. Summary of the Invention

[0005] This application proposes a data lineage confidence verification method to solve one of the above-mentioned technical problems.

[0006] The technical solution adopted in this application is as follows:

[0007] This application provides a data lineage confidence verification method, including:

[0008] Asynchronously collect metadata from multi-source heterogeneous data systems, extract field-level lineage mappings through hierarchical parsing, and construct an initial lineage relationship graph containing lineage edges and their attribute information;

[0009] Multidimensional feature indicators are extracted from each bloodline edge in the initial bloodline relationship map, and a comprehensive confidence score for each bloodline edge is calculated based on a multidimensional confidence assessment model. Bloodline edges with a comprehensive confidence score lower than the application threshold are filtered into the bloodline list to be verified.

[0010] Perform multi-method collaborative verification on the questionable bloodline edges in the bloodline list to be verified. The multi-method collaborative verification includes at least two of the following: cross-source active detection verification, AI semantic reasoning completion combined with graph context, and manual visual annotation feedback. Selectively trigger the next verification method based on the judgment result of the previous verification method.

[0011] The judgment result of the multi-means collaborative verification is written back to the initial blood relationship graph, the topology, confidence status and verification audit information of the corresponding blood relationship edges are updated, and the comprehensive confidence score calculation, blood relationship list screening and multi-means collaborative verification are re-executed based on the updated blood relationship graph until the preset closed-loop termination condition is met.

[0012] Preferably, the field-level lineage mapping is extracted using a hierarchical parsing method, including:

[0013] Prioritize structured semantic parsing to extract field-level source-target mappings from SQL class processing tasks with available source code;

[0014] When structured semantic parsing coverage is insufficient, cross-language script data flow analysis is performed to recover hidden lineage links from script code;

[0015] When information gaps still exist, perform fallback inference based on task topology dependencies to infer field-level lineage mapping by combining task dependencies, field naming similarity, and data type compatibility.

[0016] When different parsing methods produce conflicting lineage mappings for the same field, conflict arbitration is performed based on the basic score of the parsing credibility of each parsing method. The lineage mapping with the highest basic score is retained as the main lineage edge in the active graph, and the remaining lineage mappings are written into the alternative attributes.

[0017] If the difference in the baseline scores of different analytical methods is less than the preset conflict threshold, then multiple conflicting bloodline edges will be written into the graph simultaneously, the conflict status field of the corresponding bloodline edge will be marked as a conflict pending state, and an initial low confidence value will be assigned for subsequent comprehensive confidence assessment and verification screening.

[0018] Preferably, the multidimensional feature indicators include parsing credibility, metadata integrity, data consistency verification, timeliness, and link integrity;

[0019] The comprehensive confidence score is obtained by weighted fusion of the scores of each dimension and their corresponding weight coefficients in the multidimensional feature indicators.

[0020] The weight coefficients are updated online through a logistic regression model. Specifically, after each closed-loop iteration, the bloodline edges that are confirmed to be real through the multi-method collaborative verification are used as positive samples, and the bloodline edges that are confirmed to be erroneous or invalid through the multi-method collaborative verification are used as negative samples. The scores of each dimension are used as the model input for training, and the feature coefficients of each dimension obtained from the training are normalized to obtain the updated weight coefficients.

[0021] Preferably, the application threshold is an adaptive application threshold, which is determined according to the following method:

[0022] Adaptive application threshold = clamp (base threshold + business domain adjustment amount + data level adjustment amount + verification effectiveness adjustment amount, lower threshold, upper threshold).

[0023] The adjustment amount of the business domain is determined based on the business domain labels involved in the bloodline edge;

[0024] The data level adjustment amount is determined based on the logical layering of the data warehouse spanned by the bloodline edge;

[0025] The adjustment amount of the verification effectiveness is determined based on the proportion of the bloodline edges that were confirmed to be incorrect or invalid in the bloodline list to be verified in the previous closed-loop iteration to the bloodline edges that have been verified.

[0026] The clamp is used to limit the adaptive application threshold to a preset threshold range defined by the lower threshold and the upper threshold.

[0027] Preferably, the multi-method collaborative verification includes cross-source active detection verification, which includes:

[0028] Establish a memory computing sandbox independent of the underlying heterogeneous data source;

[0029] The data source adapter extracts sample data of upstream and downstream fields of the bloodline edge within the same association key range, loads the sample data into the memory computing sandbox and establishes a hash index;

[0030] The probe query template is executed within the memory computing sandbox to calculate the consistency index of field values, the consistency index of null value propagation, and the synchronization index of update time.

[0031] If the field value propagation consistency index, null value propagation consistency index, and update time synchronization index meet the corresponding verification conditions, the comprehensive confidence score of the corresponding bloodline edge will be raised to the qualified range.

[0032] If the above indicators meet the corresponding discard criteria, the corresponding bloodline edge will be marked as discarded or invalid.

[0033] If the above indicators fail to reach a definite judgment, the corresponding bloodline edge remains in a pending verification state, and the latter verification method in the multi-means collaborative verification is triggered.

[0034] Preferably, the multi-method collaborative verification includes AI semantic reasoning completion combined with graph context, wherein the AI ​​semantic reasoning completion combined with graph context includes:

[0035] Encode the metadata information of each field in the kinship graph into semantic vectors and construct a vector index;

[0036] Extract the table structure information, field semantic information, script fragments, and sampling features of the bloodline edges to be verified as the graph context;

[0037] Based on the vector index, a candidate matching list is recalled, and the candidate matching list is concatenated with the graph context to form a structured prompt word input into the large language model, outputting a comprehensive semantic matching score at the name level, data type level, and semantic role level;

[0038] When a graph topology breakpoint is detected, a large language model is used in conjunction with the graph context to recommend candidate connections, and the probability of the candidate connections is calculated using a graph probability model to complete the broken links.

[0039] When implicit conversion logic is detected, the conversion function is inferred using a large language model, and an independent conversion node is inserted into the kinship graph after verification.

[0040] Preferably, the multi-method collaborative verification includes manual visual annotation feedback, which includes:

[0041] Bloodline boundaries that cannot be determined by previous verification methods will be pushed to the data governance workbench;

[0042] The data governance workbench displays the upstream and downstream table structures, field semantics, original scripts, detection and verification results, and AI inference suggestions for questionable bloodline edges;

[0043] Receive manual annotation results and correct the confidence state and topological relationship of the corresponding bloodline edge based on the manual annotation results;

[0044] The manually labeled results are fed back to the AI ​​semantic reasoning model as supervision samples and to the multidimensional confidence evaluation model as calibration samples, so as to update the inference parameters of the AI ​​semantic reasoning model and the weight coefficients of the multidimensional confidence evaluation model.

[0045] Preferably, the preset closed-loop termination condition includes any one of the following conditions:

[0046] The proportion of bloodline edges in the entire graph whose overall confidence score is lower than the application threshold is less than or equal to a first preset ratio;

[0047] In two consecutive closed-loop iterations, the absolute value of the decrease in the percentage of bloodline edges with a comprehensive confidence score lower than the application threshold is less than the second preset ratio.

[0048] The closed-loop iteration round counter reaches its maximum limit.

[0049] When the closed-loop iteration round counter reaches the maximum upper limit round, the bloodline edge that has not yet been verified and confirmed is marked as a residual questionable state.

[0050] Preferably, after the preset closed-loop termination condition is met, the method further includes:

[0051] Continuous monitoring of the kinship graph based on time-series trajectories is performed, and the comprehensive confidence score of each kinship edge is recorded periodically;

[0052] Linear fitting is performed on the confidence of the time series of a single bloodline edge within a set sliding window. If the fitting slope is less than zero and the cumulative decrease exceeds the set trend threshold, it is judged as an abnormal continuous downward trend.

[0053] If the difference in confidence level of a single bloodline edge exceeds the set threshold for sudden drop in adjacent evaluation periods, it is judged as a sudden drop anomaly.

[0054] When an abnormal continuous downward trend or a sudden drop is detected, the evaluation dimension with the largest drop in location reliability score is identified. Combined with metadata collection logs, structural change events, task modification events, or data source interruption events within the abnormal window period, a diagnostic report containing the root cause type and the affected scope is output, and the corresponding lineage edge is re-included in the lineage list to be verified.

[0055] Preferably, after the preset closed-loop termination condition is met, the method further includes:

[0056] A state change event is generated based on the updated bloodline edge confidence state, and the state change event is published to a distributed message queue;

[0057] The data aggregation engine asynchronously consumes the state change events and sends corresponding linkage control commands through API callbacks.

[0058] For verified lineage links, issue a command to restore full aggregation;

[0059] For related links where the confidence level is within the preset rate limiting range or the status is pending verification, a rate reduction and rate limiting command is issued.

[0060] For confirmed abandoned, ineffective, or broken bloodline links, issue a pause and block command to interrupt convergence.

[0061] Before executing a speed reduction and current limiting instruction or an aggregation suspension and blocking instruction, generate a parameter configuration snapshot of the corresponding aggregation task;

[0062] After the re-evaluation of the confidence status of the restricted blood relationship link is qualified, load the parameter configuration snapshot to restore the state of the aggregation task.

[0063] Embodiments of the present application disclose a method for data lineage confidence verification. By asynchronously collecting multi-source heterogeneous metadata and constructing an initial lineage relationship graph, then extracting multi-dimensional feature indicators (such as analysis reliability, data consistency, etc.) for each lineage edge in the graph and calculating a comprehensive confidence score, the system can accurately screen lineage edges whose confidence is lower than the application threshold into the list to be verified. This solves the problem in the prior art that lineage reliability cannot be quantified, which leads to an excessively large manual verification scope, enables verification resources to focus on the few truly suspicious lineage edges, and greatly improves recognition efficiency.

[0064] For suspicious lineage edges in the list to be verified, the solution innovatively adopts a "multi-means collaborative verification" strategy, that is, cross-source active detection and verification, AI semantic reasoning completion, and manual visual annotation feedback are performed sequentially or selectively. This layered and progressive verification mechanism enables most suspicious lineages to be adjudicated through fully automatic active detection, complex or ambiguous situations can be assisted by AI models for reasoning, and only a very small number of extreme cases require manual intervention. More importantly, the verification results will be written back to the initial lineage relationship graph to update its topology and confidence status, and trigger a new round of evaluation-verification cycle until the preset closed-loop termination condition is met. This iterative closed loop of "evaluation-verification-update-re-evaluation" fundamentally solves the problem of continuous attenuation of lineage quality under the existing open-loop architecture, and endows the lineage graph with the ability of self-healing and increasing accuracy.

[0065] The above closed-loop mechanism continuously maintains a high-confidence lineage graph, which provides a reliable basis for downstream data aggregation tasks. The system can dynamically issue linkage control instructions such as recovery, current limiting or blocking according to the latest confidence status of lineage edges, effectively preventing the spread of wrong data caused by invalid lineage relationships, and improving the robustness and reliability of the entire data governance system. Description of Drawings

[0066] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, throughout the drawings, the same reference numerals are used to refer to the same components. In the drawings:

[0067] Figure 1This is a flowchart illustrating a data lineage confidence verification method provided in an embodiment of the present invention. Detailed Implementation

[0068] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0069] like Figure 1 As shown, in some embodiments of this application, this embodiment provides a data lineage confidence verification method. Specifically, the method includes the following steps:

[0070] Step S101: Asynchronously collect metadata from the multi-source heterogeneous data system, and extract field-level lineage mappings through hierarchical parsing to construct an initial lineage relationship graph containing lineage edges and their attribute information.

[0071] The core objective of this step is to efficiently and comprehensively construct an initial, fine-grained kinship map. This process can be divided into two closely linked phases: asynchronous acquisition of multi-source heterogeneous metadata and hierarchical parsing and map construction.

[0072] Asynchronous collection of multi-source heterogeneous metadata is specifically implemented as follows: To address the issue of diverse data sources and formats in industrial internet platforms, this solution employs a distributed architecture of "centralized scheduling + edge execution" to deploy metadata collection agents. These lightweight agents are deployed on various data nodes (such as database servers, ETL engines, etc.) and can collect metadata from three core objects at differentiated frequencies (e.g., full collection every 2 hours for ODS layer tables with high-frequency changes and real-time monitoring of incremental changes, and full collection weekly for report layers with low-frequency changes): (1) technical metadata such as schema definitions and field comments of databases / data warehouses; (2) processing logic metadata such as SQL scripts and Python / Java code of ETL tasks; and (3) operational metadata such as data change events in platform operation and maintenance logs. This asynchronous and distributed collection strategy ensures the comprehensiveness and timeliness of metadata acquisition while avoiding performance impact on the production system.

[0073] Hierarchical parsing and graph construction specifically involve: the collected raw metadata needs to undergo deep parsing to be transformed into usable lineage relationships. This solution abandons the limitations of single parsing methods and innovatively adopts a hierarchical parsing fusion strategy of "precision priority, complementary coverage, and conflict arbitration":

[0074] The first layer, high precision: prioritizes structured semantic parsing of SQL-type tasks with available source code (such as Hive SQL and Spark SQL), and accurately extracts field-level source-target mappings by constructing an Abstract Syntax Tree (AST).

[0075] The second layer, supplementary coverage: When structured semantic parsing coverage is insufficient, cross-language script data flow analysis is performed to recover hidden lineage links from the script code. For example, the specific device-executable rules are as follows: First, an Abstract Syntax Tree (AST) analysis tool is invoked to parse non-SQL scripts such as Python, Java, or Shell into control flow graphs and data flow graphs; second, in the data flow graphs, input variables corresponding to external data source read operations (such as read_csv, fetch_data) and output variables corresponding to external data write operations (such as to_sql, save) are identified; finally, taint tracking is performed along the assignment statements, function return values, and variable propagation paths in the data flow graph, from output variables to input variables. If the data flow path of an output variable can connect to its corresponding input variable, its underlying physical field mapping relationship is extracted as a hidden lineage link and written into the graph.

[0076] The third layer is fallback inference: when the above methods still have information gaps, heuristic inference is performed based on the topological dependencies of the ETL task, combined with field naming similarity and data type compatibility, to ensure maximum coverage of lineage relationships.

[0077] Ultimately, all the parsed field-level mappings are constructed into an initial lineage graph. Each "lineage edge" in the graph not only records the upstream and downstream entity and field information, but also includes rich attribute information, such as the conversion type, the parsing method identifier used, the parsing timestamp, and a marker reflecting its initial credibility.

[0078] For example, suppose there is a data processing link in a manufacturing company's data platform, from ods_sensor_raw (raw sensor table in the ODS layer) to dwd_sensor_clean (cleaned table in the DWD layer).

[0079] During the data collection phase: The data collection agent deployed on the MySQL server periodically pulls the table structure of ods_sensor_raw (containing the fields sensor_id, raw_value, and timestamp); at the same time, the agent deployed on the Airflow scheduling server obtains the Hive SQL script responsible for this processing task via API.

[0080] Analysis phase:

[0081] First-level parsing: The system first performs AST parsing on the obtained Hive SQL script. The script content is "INSERT INTO dwd_sensor_clean SELECT sensor_id, raw_value * 0.1 AS temperature_celsius, timestamp FROM ods_sensor_raw;". Through AST traversal, the system accurately identifies that ods_sensor_raw.raw_value, after being multiplied by 0.1, is mapped to dwd_sensor_clean.temperature_celsius.

[0082] Graph Construction: The system creates a lineage edge in the initial lineage graph, with its upstream being ods_sensor_raw.raw_value and its downstream being dwd_sensor_clean.temperature_celsius. The transformation type is "arithmetic operation", the parsing method is identified as "AST_SQL", and a high initial confidence level (e.g., 0.90) is assigned.

[0083] Let's consider a more complex scenario: a data cleaning logic is implemented by a Python script. This script reads a CSV file, performs a unit conversion on the `pressure_psi` field (multiplying it by 6.895), and then writes it to the `pressure_kpa` field in a Hive table. Since this is not standard SQL, the first-level parsing will fail. At this point, the system will initiate a second-level parsing, using a Python AST analysis tool to analyze the script. It successfully traces the data flow from `pressure_psi` to `pressure_kpa` and creates the corresponding lineage edge in the graph. The parsing method is identified as "CODE_DFA," with an initial confidence level slightly lower than the AST parsing (e.g., 0.75).

[0084] It should be noted that, in specific implementation scenarios, an expanded approach can be adopted based on the above solution. That is, in addition to the databases, ETL tasks, and operation logs explicitly listed in the document, this solution is also applicable to collecting other types of metadata sources, such as the topic schema of a message queue (Kafka), the document structure definition of a NoSQL database (such as MongoDB), and even data source connection information from BI tools (such as Tableau). As long as relevant information about data entities, fields, and processing logic can be extracted, it can be included in the category of "multi-source heterogeneous data systems" and connected to this collection framework via the adapter pattern.

[0085] Step S102: Extract multidimensional feature indicators from each bloodline edge in the initial bloodline relationship map, calculate the comprehensive confidence score of each bloodline edge based on the multidimensional confidence assessment model, and filter the bloodline edges with the comprehensive confidence score lower than the application threshold into the bloodline list to be verified.

[0086] The purpose of this step is to assign a quantifiable credibility label to each blood relationship, thereby providing a basis for decision-making in subsequent accurate verification.

[0087] The first stage, the quantification process for extracting multidimensional feature indicators, is as follows:

[0088] Each feature dimension is not merely used as a qualitative label, but is converted into a quantized score within the [0,1] interval. The system iterates through each kinship edge in the initial kinship graph and extracts feature scores S1 to S5 for the five dimensions using a pre-defined quantization algorithm model. All scores are normalized to the [0,1] interval.

[0089] The quantification of the parsing credibility score S1 is as follows: A base score is determined based on the parsing method used during the construction of the lineage edge, and then adjusted using context factors. Specifically, the system pre-defines the base score mapping rules: 0.95 for operator-level AST semantic parsing; 0.80 for cross-language script analysis; 0.60 for heuristic inference; 1.00 for manually confirmed sources; and 0.45 for conflict-pending sources. The system further extracts parsing context richness factors (+0.05 for complete field annotations, -0.05 for missing ones) and parser version factors for adjustment, and uses the clamp function to limit the final adjusted score to the [0,1] interval, thus obtaining the parsing credibility score S1.

[0090] Quantification of Metadata Integrity S2: The metadata of upstream and downstream data entities involved in the lineage boundary is divided into three subcategories: technical metadata, business metadata, and operational metadata. Several check items are preset for each subcategory (e.g., business metadata includes Chinese field names, comments, and the meaning of enumeration values). The system uses a binary judgment method to evaluate each check item, recording a passing item as 1 and a failing item as 0. The passing percentage score for each of the three subcategories is calculated, and the scores of the three subcategories are then averaged with equal weights to obtain the metadata integrity score S2.

[0091] Quantification of data consistency verification score S3: A fixed number of sample data are extracted from the upstream and downstream fields of the bloodline boundary, and S3 is obtained by quantifying and averaging them equally from the following three sub-dimensions:

[0092] Statistical feature consistency: Extract K statistical features (such as mean, deduplication count, etc.) from upstream and downstream fields, and calculate the score using the absolute deviation normalization formula.

[0093] ,

[0094] Among them, stat upj and stat downj These are the j-th statistical feature values ​​of the upstream and downstream fields, respectively.

[0095] Distribution consistency: The same binning strategy is used to calculate the distribution of upstream and downstream data, and the difference is quantified by calculating the Jensen-Shannon divergence (JSD) between the two. If the Jensen-Shannon divergence is calculated using the logarithm to the base 2, its value range is [0,1], and the distribution consistency score Sdist = 1 - JSD; if the natural logarithm is used, it is first normalized according to JSDnorm = JSD / ln2, and then Sdist = 1 - min(1, JSDnorm) is calculated.

[0096] Business logic consistency: Match the corresponding transformation rule base (such as direct mapping, aggregation operation, etc.) and calculate the proportion of sample data that passes the verification rule as the score.

[0097] Quantification of Timeliness S4: The timeliness of metadata is assessed using an exponential decay function. The quantification formula is as follows:

[0098] ,

[0099] Where ΔT is the difference (in days) between the current system time and the most recent collection time of the metadata upon which the lineage relationship is based; λ is the attenuation coefficient. The system dynamically determines the value of λ based on the historical change frequency of the data source. For example, the system first calculates the average update interval Tavg of the most recent M updates of the data source, and then determines the attenuation coefficient according to λ=ln2 / Tavg, so that when the time lag ΔT equals the average update interval Tavg, the timeliness score is attenuated to 0.5. If there are insufficient historical update records, a default Tavg is set according to the data level: 2 hours for ODS layer, 24 hours for DWD / DWS layer, 7 days for ADS layer, and 30 days for archive layer. Thus, the timeliness score S4 can be quantified according to different data update frequencies.

[0100] The link integrity score S5 is quantified by combining upstream connectivity, downstream connectivity, the proportion of abnormal adjacencies, and the rationality of the link hierarchy. For a lineage edge e=(u,v), where u is the upstream node and v is the downstream node, the system first determines whether the upstream node u has a traceable predecessor node or whether it has been declared as a legitimate source node; if so, the upstream connectivity identifier Cu is set to 1, otherwise it is set to 0. The system further determines whether the downstream node v has a reachable subsequent consumer node or whether it has been declared as a legitimate terminal node; if so, the downstream connectivity identifier Cd is set to 1, otherwise it is set to 0.

[0101] The system calculates the adjacency anomaly ratio Ba of the bloodline edge e. The adjacency anomaly ratio Ba is the ratio of the number of upstream and downstream adjacent edges of the bloodline edge e that are in a state of abandonment, failure, pending conflict, or long-term doubt to the total number of adjacent edges. If there are no adjacent edges, the system determines whether the node is declared as a source node or a terminal node; if it is not declared as a source node or a terminal node, Ba is 1.

[0102] The system also calculates the link hierarchy hop penalty factor Dp. Specifically, the hierarchy span ΔL is determined based on the data hierarchy to which the upstream and downstream nodes of the bloodline edge belong. If ΔL belongs to a preset allowed span set, then Dp is 0; if ΔL does not belong to the preset allowed span set, then Dp is 0. p Take min(1,|ΔL-ΔL) exp | / D max ), where: ΔL=∣L v -L u |, L u L represents the data level number to which the upstream node u of the bloodline edge belongs. v ΔL represents the data level number to which the downstream node v of the bloodline edge belongs; ΔL represents the actual level span of the bloodline edge e=(u,v); ΔL exp D represents the expected hierarchical span corresponding to this type of bloodline edge; max This indicates the preset maximum tolerance span, and Dmax >0;D p This represents the link-level hop penalty factor, and its value is restricted to the interval [0,1].

[0103] The link integrity score S5 is calculated using the following formula:

[0104] S5=γu×Cu+γd×Cd+γa×(1-Ba)+γp×(1-Dp);

[0105] Where γu, γd, γa, and γp are the weights of upstream connectivity, downstream connectivity, adjacency health, and hierarchical rationality, respectively, and γu + γd + γa + γp = 1. For example, γu = 0.30, γd = 0.30, γa = 0.20, and γp = 0.20. For an upstream node declared as a legitimate source node, Cu is not reduced due to the lack of a preceding node; for a downstream node declared as a legitimate terminal node, Cd is not reduced due to the lack of a subsequent node.

[0106] The second stage involves comprehensive confidence scoring and screening.

[0107] The extracted multidimensional feature metrics are input into a multidimensional confidence assessment model. This model is not a simple rule aggregation, but a weighted fusion model that can learn online (such as a logistic regression model). The feature score of each dimension is multiplied by its corresponding weight coefficient, and finally summed to obtain the comprehensive confidence score of the lineage edge. This score is a value between 0 and 1, with the closer to 1 indicating higher confidence. Subsequently, the system compares the scores of all lineage edges with an "application threshold." All lineage edges below this threshold are considered "questionable" or "low confidence" and are uniformly collected into a "lineage to be verified list" as clear targets for the next step of collaborative verification.

[0108] For example, let's continue with the bloodline edge "ods_sensor_raw.raw_value -> dwd_sensor_clean.temperature_celsius":

[0109] Feature extraction:

[0110] Reliability of the parsing: Since it is obtained through AST_SQL parsing, the score is 0.95.

[0111] Metadata completeness: The raw_value field lacks business comments, but temperature_celsius has complete comments, resulting in an overall score of 0.80.

[0112] Data consistency verification: The system extracted 1000 samples and found that the values ​​of raw_value and temperature_celsius strictly satisfy the relationship temperature_celsius = raw_value * 0.1, with a score of 0.98.

[0113] Timeliness: This ETL task is executed daily at midnight, with the most up-to-date data, and a score of 0.99.

[0114] Link integrity: Its upstream ods_sensor_raw table itself has no upstream lineage (is the source), and the downstream dwd_sensor_clean table is referenced by multiple downstream tasks. The link is complete, and the score is 0.90.

[0115] Overall Score: Assuming the current weight coefficients for each dimension are [0.3, 0.1, 0.3, 0.1, 0.2], the overall score is: 0.95×0.3+0.80×0.1+0.98×0.3+0.99×0.1+0.90×0.2=0.932. If the current application threshold is 0.85, then this bloodline edge does not need to be included in the verification list because its score is higher than the threshold.

[0116] Consider another lineage edge derived through fallback inference, such as inferring the mapping from the `user_id` field in the `app_log` table to the `id` field in the `user_profile` table. Its parsing reliability might only be 0.60, and data consistency verification might score low (e.g., 0.70) due to non-standard log formats. Even if other dimensions score reasonably well, its overall score is likely to be below 0.85, thus accurately filtering it into the verification list, awaiting further proactive probing or AI inference to confirm its authenticity.

[0117] It should be noted that, in specific implementation scenarios, an expanded approach based on the above scheme can be adopted, incorporating additional indicator dimensions. That is, in addition to the five core dimensions mentioned above, this scheme can also include more indicators that reflect lineage quality. For example, a "business importance" dimension can be added, assigning higher quality requirements based on the data domain of the lineage edge (such as core business domains like finance and risk control); or a "historical stability" dimension can be added to examine whether the lineage relationship has frequently changed or failed over a period of time. Any new indicator that can quantify the reliability of lineage relationships can be integrated into this evaluation system as a new characteristic dimension.

[0118] An extended approach to the evaluation model is adopted. Although the embodiment uses logistic regression as an example, the scope of protection of this solution covers all machine learning or statistical models capable of achieving multi-dimensional feature weighted fusion and outputting a confidence score. For example, more complex ensemble learning models (such as XGBoost, random forest) can be used to capture non-linear relationships between features, or deep neural networks can be used to process higher-dimensional feature vectors. As long as the model's input is a multi-dimensional feature index and the output is a comprehensive confidence score, it falls within the scope of protection of this solution. All of the above optional solutions are within the scope of protection of this application.

[0119] Step S103: Perform multi-means collaborative verification on the questionable bloodline edges in the bloodline list to be verified. The multi-means collaborative verification includes at least two of the following: cross-source active detection verification, AI semantic reasoning completion combined with graph context, and manual visual annotation feedback. Then, the next verification method is selectively triggered based on the judgment result of the previous verification method.

[0120] This step aims to efficiently and accurately resolve questionable lineage issues in the verification list through an intelligent, hierarchical, and collaborative verification mechanism. Its core is to simulate the screening logic of human experts: first, use automated, highly deterministic methods for rapid verification; then, for ambiguous or complex cases, introduce more advanced intelligent reasoning; and finally, hand over the very few difficult cases that cannot be automatically determined to human staff.

[0121] The three verification methods and their collaborative logic are as follows:

[0122] Cross-source proactive probing verification: The system constructs an independent, secure in-memory computing sandbox for the lineage edges to be verified. Through the data source adapter, sample data with consistent association keys is extracted from the upstream and downstream data sources of the lineage edge and loaded into the sandbox. Then, a preset probing query template is executed within the sandbox to directly verify key consistency indicators such as field value propagation, null value propagation, and update time synchronization. If the indicator clearly meets the pass or fail conditions, a decision is directly given; if the indicator is in a fuzzy area (such as partial consistency), it is judged as "undeterminable," and the next verification method is triggered.

[0123] AI-powered semantic reasoning and completion based on graph context: When active probing fails to yield clear conclusions, the system resorts to semantic inference. It moves beyond the current kinship edge itself, analyzing it within the context of the entire kinship graph. Leveraging the powerful semantic understanding capabilities of Large Language Models (LLMs), the system comprehensively analyzes information such as the naming, annotations, data types, and script fragments of upstream and downstream tables / fields to calculate a comprehensive semantic matching score. Furthermore, the AI ​​can identify topological breakpoints or implicit transformation logic in the graph and proactively recommend candidate connections or transformation functions to intelligently complete broken links.

[0124] Human visual annotation feedback: This is the final fallback. For bloodline edges that cannot be determined by the first two automated methods, the system will push them along with all relevant contextual information (original script, detection results, AI inference suggestions, etc.) to the data governance workbench. Data governance experts can visually review all the evidence and make final human annotations. This process not only concludes the problem, but the annotation results also serve as valuable supervisory signals, feeding back into the AI ​​model and confidence assessment model, forming a closed loop of continuous learning.

[0125] Collaborative Triggering Mechanism: The entire verification process is not executed in parallel, but rather selectively triggered according to the "cost-benefit" optimization principle. The system always starts with the lowest-cost and most certain "cross-source proactive probing." Only when it returns an "uncertain" decision will the more expensive but more intelligent "AI semantic reasoning" be invoked as needed. Similarly, the problem is escalated to manual handling only when AI reasoning also fails to reach a high-confidence conclusion. This cascading triggering mechanism greatly optimizes resource utilization efficiency.

[0126] For example, to ensure the interpretability and output stability of the AI ​​semantic reasoning model in industrial data governance scenarios, its input construction, output constraints, and fine-tuning process are as follows:

[0127] Prompt Engineering: The model does not directly receive raw text, but instead receives structured graph context prompts. These prompts are required to include: metadata of upstream entities (including field names, data types, annotations, and statistical maximum / minimum values), corresponding metadata of downstream entities, original code snippets (if any), and a list of candidate matches recalled using cosine similarity calculated via vector indexing (such as FAISS).

[0128] Strongly Constrained Output Structure: To address the randomness issue in the output of large language models, system-level instructions mandate that large language models must output structured conclusions in JSON format. The output content is forcibly split into: name-level matching score, data type-level matching score, semantic role-level matching score, and a weighted comprehensive semantic matching score (range [0,1]) derived from these three, along with corresponding data flow relationship probability values.

[0129] Incremental Fine-tuning Mechanism: In the initial stage, the model uses a pre-trained general-purpose large language model. During closed-loop operation, the system collects the annotation results (such as manually confirmed correct mappings and supplementary broken connection rules) from the data governance workbench, which are then manually verified. This labeled data is encapsulated into a "structured prompt word - correct mapping output" instruction fine-tuning dataset. A low-rank adaptive (LoRA) incremental fine-tuning strategy is adopted to periodically iteratively update the parameters of the large language model, enabling the model's semantic reasoning ability to dynamically improve with business evolution.

[0130] For example, suppose there is a questionable lineage edge in the list to be verified: log_table.event_code -> fact_table.event_id.

[0131] Phase 1: Cross-source active detection verification:

[0132] The system extracted 10,000 related samples from the sandbox and found that the value range of `event_code` was [A,B,C,D], while the value range of `event_id` was [1,2,3,X]. The mappings A->1, B->2, and C->3 were clear, but there were a large number of records with D->X, where X was a special marker value. The detection results showed that the value propagation consistency index was 85%, which did not reach the preset 95% pass threshold, but was also not as low as the 50% discard threshold. Therefore, the system determined it to be "undeterminable" and passed this lineage edge to the next verification method.

[0133] The second stage involves AI semantic reasoning and completion.

[0134] The AI ​​model was activated and analyzed the context: the log_table was annotated as "application raw logs," and the event_code was annotated as "event type code"; the fact_table was the fact table, and the event_id was annotated as "normalized event ID." Simultaneously, the AI ​​found a dimension table dim_event in the graph, containing two columns, code and id, whose content exactly corresponds to the mapping A-1, B-2, C-3, but there was no record for DX. Based on this, the AI ​​inferred that D is a newly added event code not yet registered in the dimension table, while X is the default ID assigned by the system to an unknown code. The AI ​​gave a semantic matching score of 0.88 (high) and suggested that the lineage was basically valid, but the dimension table needed to be supplemented. Since the score did not reach the automatic pass threshold of 0.95, the system could not fully confirm it, so this case, along with the AI's complete analysis report, was pushed to the human workbench.

[0135] The third stage involves manual visual annotation feedback:

[0136] After reviewing all the information on the workbench, the data governance expert confirmed the AI's inference and manually added the mapping for D-4 to the dim_event table (correcting X to 4). He labeled this bloodline edge as "valid" and submitted it. Upon receiving the label, the system not only updated the state of the bloodline edge but also used this case as a positive sample for subsequent model training.

[0137] It should be noted that, in specific implementation scenarios, further extensions to the verification methods can be implemented based on the above solutions. These include: proactive probing, which can be extended to more types of verification metrics. For example, adding "data distribution similarity" (such as KL divergence) to verify the rationality of numerical field conversions, or adding "primary and foreign key constraint verification" to check the integrity of references between relational databases. AI semantic reasoning, in addition to using large language models, can also integrate specialized knowledge graph embedding models to vectorize entities and relationships in the kinship graph and assess kinship probability through distance calculations in the vector space. Alternatively, few-shot learning capabilities can be introduced, enabling AI to quickly adapt to new business scenarios based on a very small number of examples.

[0138] Extended solutions employing collaborative strategies include: dynamic strategy selection, where the collaborative triggering mechanism can be more intelligent. For example, the system can pre-determine which verification method is most likely to succeed based on the characteristics of the lineage to be verified (such as its business domain and data level), thus skipping inefficient intermediate steps. For the lineage of the financial risk control domain, AI + human dual verification may be initiated directly; while for the lineage of the log analysis domain, multiple active probes with different parameters are prioritized. Parallel collaboration, in specific scenarios, to accelerate the verification process, allows two methods (such as active probes and AI inference) to be executed in parallel in isolated environments, and finally, the judgment results of the two are combined (e.g., through voting or weighting) to make a final decision. As long as the final decision is based on the collaboration of multiple methods and subsequent actions can be adjusted according to the preliminary results, it falls within the scope of protection of this solution.

[0139] An extended approach employing a closed-loop human feedback mechanism is adopted, meaning that manually labeled results can be used not only for model training but also for generating automated verification rules. For example, if multiple experts use the same judgment logic when handling similar problems, the system can automatically extract a new, reusable verification rule and integrate it into the active detection or AI inference process, thereby gradually reducing reliance on human intervention. This transformation from "human intelligence" to "intelligence" is a key manifestation of the continuous evolution capability of this solution. All of the above optional solutions fall within the scope of protection of this application.

[0140] Step S104: Write the judgment result of the multi-means collaborative verification back to the initial blood relationship graph, update the topology, confidence status and verification audit information of the corresponding blood relationship edge, and re-execute the comprehensive confidence score calculation, blood relationship list screening and multi-means collaborative verification based on the updated blood relationship graph until the preset closed-loop termination condition is met.

[0141] This step solidifies the knowledge and decisions generated in the previous verification steps into the pedigree map, and drives the entire system to converge toward a higher precision and higher quality state through an iterative optimization mechanism.

[0142] Phase 1: Map Update and Information Persistence

[0143] Once a multi-method collaborative verification process reaches a final judgment on a questionable lineage edge (e.g., "valid," "invalid," "discarded," or "requires completion"), the system immediately writes this judgment back to the initial lineage graph. This update is comprehensive:

[0144] Topology update: If the decision is "valid" and involves link completion (e.g., AI inference discovers an intermediate transformation node), the corresponding node and edge will be added to the graph; if the decision is "invalid" or "abandoned", the corresponding lineage edge will be marked as invalid or directly deleted logically.

[0145] Confidence Status Update: Based on the decision, the overall confidence score of the bloodline edge will be reset. For example, a "valid" edge will be assigned a high confidence value close to 1.0; an "invalid" edge will be set to 0.

[0146] Verification and Audit Information Updates: To ensure process traceability, the system will attach detailed audit logs to the bloodline boundary, including the methods used in this verification (such as "active detection + AI inference"), trigger time, execution time, key evidence summaries (such as detection index values, AI matching scores), and the final judgment basis. This information is crucial for subsequent quality analysis and model optimization.

[0147] The second stage involves closed-loop iteration and termination determination.

[0148] After the graph is updated, the system does not stop working but immediately starts a new round of evaluation-verification loop. Based on the updated, more accurate graph, it re-executes steps S102 (calculation of comprehensive confidence score and screening of the verification list) and S103 (multi-method collaborative verification). This iterative process continues until a set of preset "closed-loop termination conditions" are met. These conditions aim to balance the thoroughness of verification with resource consumption, ensuring optimal lineage quality with limited input. Typical termination conditions include: the proportion of low-confidence lineage edges in the entire graph is sufficiently low; the improvement rate after several iterations has stagnated; or the preset maximum number of iterations has been reached to prevent infinite loops.

[0149] For example, continuing from the previous example of log_table.event_code -> fact_table.event_id:

[0150] Atlas Update: After an expert marks the bloodline edge as "valid," the system performs the following operations:

[0151] Topology: Since the AI ​​has inferred the existence of the dim_event dimension table in the middle, the system explicitly creates two new edges in the graph: log_table.event_code -> dim_event.code and dim_event.id -> fact_table.event_id, and removes the original direct mapping edges (or marks them as logical paths).

[0152] Confidence level: These two newly created bloodline edges are assigned a high confidence level of 0.98.

[0153] Audit information: The audit log for this bloodline has been updated, recording "confirmed by human experts at [time], based on a comprehensive judgment of active detection (85% consistency) and AI semantic reasoning (88% matching degree)".

[0154] Closed-loop iteration:

[0155] The system immediately initiated a new round of evaluation. In the new graph, not only were the confidence levels of the new edges high, but the overall link integrity feature score of the graph was also improved due to the introduction of the dim_event table.

[0156] After recalculating the confidence of all bloodline edges, the system found that the number of bloodline edges in the check list had decreased from 500 in the previous round to 420.

[0157] The system continues to verify these 420 items. After three rounds of such iterations, the number of items on the verification list stabilizes at around 50, and the decrease in the number of items in the two consecutive rounds is less than 1%. At this point, the system determines that the termination condition of "the improvement rate in the two consecutive rounds is less than the second preset ratio" has been met, thus ending this closed-loop verification cycle.

[0158] It should be noted that, in specific implementation scenarios, a version-based management scheme can be adopted on the basis of the above solutions, that is, version snapshot management can be implemented for the lineage graph. Each major update (such as completing a closed-loop verification) generates a new graph version. This not only allows for retrospective analysis of historical states but also supports A / B testing, i.e., running the verification process on the new version graph to verify its effectiveness before deciding whether to officially switch. An incremental update and propagation scheme can be adopted, meaning that when the state of a lineage edge is updated, its impact can be intelligently propagated along the graph topology. For example, repairing a broken upstream lineage may automatically improve the "link integrity" feature score of all its downstream lineage edges, potentially causing some downstream edges that were originally on the verification list to automatically "graduate" without needing to be verified again. All of the above optional solutions fall within the scope of protection of this application.

[0159] In some embodiments of this application, to specify and refine the "hierarchical parsing method," its internal execution logic, hierarchical progression, and conflict handling mechanism are clarified. Extracting field-level lineage mappings through the hierarchical parsing method includes:

[0160] Prioritize structured semantic parsing to extract field-level source-target mappings from SQL class processing tasks with available source code;

[0161] When structured semantic parsing coverage is insufficient, cross-language script data flow analysis is performed to recover hidden lineage links from script code;

[0162] When information gaps still exist, perform fallback inference based on task topology dependencies to infer field-level lineage mapping by combining task dependencies, field naming similarity, and data type compatibility.

[0163] When different parsing methods produce conflicting lineage mappings for the same field, conflict arbitration is performed based on the basic score of the parsing credibility of each parsing method. The lineage mapping with the highest basic score is retained as the main lineage edge in the active graph, and the remaining lineage mappings are written into the alternative attributes.

[0164] If the difference in the baseline scores of different analytical methods is less than the preset conflict threshold, then multiple conflicting bloodline edges will be written into the graph simultaneously, the conflict status field of the corresponding bloodline edge will be marked as a conflict pending state, and an initial low confidence value will be assigned for subsequent comprehensive confidence assessment and verification screening.

[0165] As mentioned above, the system first prioritizes structured semantic parsing. This method is primarily designed for SQL-based data processing tasks where the source code is directly available (e.g., Hive SQL, Spark SQL). By constructing and traversing the abstract syntax tree (AST) of these SQL statements, the system can accurately identify the mapping relationship between the target fields in the SELECT clause and the source fields in the FROM / JOIN clause, thereby directly extracting high-precision field-level source-target lineage mappings.

[0166] Secondly, when structured semantic parsing cannot cover all scenarios (for example, when data processing logic is implemented by non-SQL scripts such as Python, Java, or Shell), the system performs cross-language script data flow analysis. This method constructs a data flow graph by performing general program analysis on the script source code, tracing the entire process of variables from input to output, thereby recovering those hidden lineage links in complex code logic that cannot be discovered through simple rule matching.

[0167] Furthermore, when the two proactive parsing methods mentioned above still have information gaps, meaning that the lineage relationships of certain fields cannot be parsed at all, the system will initiate a fallback inference based on task topology dependencies. This method no longer relies on specific code content, but instead uses task dependencies in the ETL scheduling system as clues. It assumes that there is a potential data transfer relationship between the output table of the upstream task and the input table of the downstream task, and based on this, combines heuristic rules such as string similarity of field names (e.g., edit distance, stemming) and data type compatibility (e.g., INT can be converted to BIGINT, VARCHAR can be converted to STRING) to infer possible field-level mappings to ensure maximum coverage of lineage relationships.

[0168] Finally, due to the use of multiple parsing methods, conflicts may arise where different methods derive different target fields from the same source field. To address this, the system pre-determines a baseline reliability score for each parsing method (e.g., 0.95 for structured semantic parsing, 0.80 for script analysis, and 0.60 for fallback inference). In the event of a conflict, the system arbitrates the conflict: the lineage mapping with the highest baseline score is retained as the primary lineage edge in the active graph, while other conflicting mappings are stored as alternative solutions in the alternative attributes of that lineage edge for subsequent queries or verification.

[0169] Furthermore, to prevent valuable information from being overlooked due to overly absolute base scores, the system sets a preset conflict threshold (e.g., 0.1). If the difference in base scores for the resolution methods corresponding to multiple conflicting lineage mappings is less than this threshold (e.g., 0.82 vs 0.78), these mappings are considered to have similar confidence levels and are difficult to discard directly. In this case, the system simultaneously writes all conflicting lineage edges into the graph and marks their conflict status field as "conflict pending," while assigning them a low initial confidence value. These lineage edges in the "conflict pending" state will be given priority and processed in subsequent comprehensive confidence assessment and checklist screening steps.

[0170] In some embodiments of this application, in order to specifically define the calculation mechanism of the "comprehensive confidence score," the composition of its multi-dimensional features, the weighted fusion method, and the dynamic optimization strategy of the weight coefficients are clarified. The multi-dimensional feature indicators include parsing confidence, metadata integrity, data consistency verification, timeliness, and link integrity.

[0171] The comprehensive confidence score is obtained by weighted fusion of the scores of each dimension and their corresponding weight coefficients in the multidimensional feature indicators.

[0172] The weight coefficients are updated online through a logistic regression model. Specifically, after each closed-loop iteration, the bloodline edges that are confirmed to be real through the multi-method collaborative verification are used as positive samples, and the bloodline edges that are confirmed to be erroneous or invalid through the multi-method collaborative verification are used as negative samples. The scores of each dimension are used as the model input for training, and the feature coefficients of each dimension obtained from the training are normalized to obtain the updated weight coefficients.

[0173] As mentioned above, the multidimensional feature indicators specifically include the following five aspects: parsing reliability, which reflects the parsing method used to obtain the lineage edge, and different methods have different inherent reliability; metadata integrity, which measures the completeness of the source and target fields involved in the lineage edge in the metadata management system, such as whether business comments, data type definitions, primary and foreign key identifiers are included; data consistency verification, which evaluates whether the upstream and downstream fields meet logical expectations in terms of statistical characteristics such as value range distribution, null value rate, and uniqueness based on lightweight sample data comparison; timeliness, which examines the most recent update time of the data table or ETL task on which the lineage relationship depends to determine whether it is still in an active usage state; and link integrity, which analyzes the contextual connectivity of the lineage edge in its data processing link. If there are broken or abnormal nodes upstream or downstream, its own reliability will also be affected.

[0174] The calculation of the overall confidence score involves multiplying the scores of each of the five dimensions by their corresponding weight coefficients, summing the results, and normalizing the results to the [0,1] interval to form the final quantitative score.

[0175] The key is that these weight coefficients are not fixed, but are updated online through a logistic regression model. The specific update process is as follows: After each round of closed-loop iteration of "verification-update-reevaluation", the system collects all bloodline edges that have been clearly determined through multi-method collaborative verification (step S103) in this round as training samples. Among them, bloodline edges that are confirmed to be real are marked as positive samples, and bloodline edges that are confirmed to be erroneous or invalid are marked as negative samples.

[0176] To ensure the effectiveness and stability of model training, the system sets the following constraints for constructing the training sample set, for example:

[0177] Minimum sample size requirement: The total number of samples used for training in a single iteration must not be less than 200 (configurable) to ensure statistical significance;

[0178] Positive and negative sample ratio control: If the ratio of positive and negative samples is unbalanced (e.g., one class accounts for more than 80%), the system will use down-sampling or up-sampling (SMOTE-like synthesis) strategies to balance the ratio and avoid the model being biased towards the majority class.

[0179] Sample timeliness filtering: Only decision samples generated within the most recent N rounds (default N=5, configurable) are retained for training to prevent historical noise or outdated patterns from interfering with the current model;

[0180] Cold start processing: During the initial operation phase of the system (i.e., when there are not enough human or automatic judgment samples), the weight coefficients adopt preset expert experience initial values ​​(such as the highest weight for analytical credibility). After accumulating enough samples, the system switches to model-driven mode.

[0181] Using the five-dimensional feature scores of these selected and balanced samples as model input and their decision labels as output, the system employs an incremental logistic regression algorithm with L2 regularization to train the model online. The L2 regularization term effectively suppresses drastic weight fluctuations, improving model convergence stability; the incremental update mechanism avoids full retraining, ensuring the system's real-time responsiveness. After training, the regression coefficients corresponding to each dimension of the model's output reflect the discriminative power of that dimension in determining kinship authenticity. The system normalizes these regression coefficients (e.g., using the Softmax function) so that their sum equals 1, resulting in a new set of weight coefficients that better reflect the current data environment and governance experience. This updated set of weight coefficients will be used in the next round of comprehensive confidence score calculation, enabling the continuous evolution of the evaluation model.

[0182] To achieve the online update of the aforementioned weight coefficients, the specific construction and training process of an exemplary logistic regression model is as follows:

[0183] Model input (feature space): Define feature vector X=[S1,S2,S3,S4,S5], which is the score of a single bloodline edge on five dimensions: resolution credibility, metadata integrity, data consistency, timeliness, and link integrity.

[0184] Model output (prediction target): The model outputs the predicted probability P(Y=1|X). Here, the target variable Y is the authenticity label of the bloodline edge; Y=1 indicates that the bloodline edge has been verified as real and valid (positive sample), and Y=0 indicates that it has been confirmed as incorrect or invalid (negative sample).

[0185] Model training and parameter solving: After each closed-loop iteration, the system automatically collects the bloodline edges that have obtained deterministic labels Y through multi-method collaborative verification (especially manual annotation feedback and active detection) as the training sample set. The gradient descent optimization algorithm is used to train the logistic regression model and solve for the feature regression coefficient vector θ=[θ1,θ2,θ3,θ4,θ5] that minimizes the cross-entropy loss function.

[0186] Weight mapping calculation: After training convergence, extract the regression coefficients θ corresponding to each dimension of features. i To eliminate negative numbers and ensure the weight sum is 1, the Softmax function (or the absolute value normalization function) is used to map the regression coefficients to the weight coefficients for the next round. The calculation formula is as follows: Through this mechanism, the system can objectively quantify the feature dimensions that have the greatest impact on kinship quality under the current data environment.

[0187] In some embodiments of this application, to comprehensively consider business domain sensitivity, data hierarchy importance, and the actual feedback effect of the previous round of verification, the screening threshold is dynamically adjusted to avoid the "over-screening" or "under-screening" problems caused by using a fixed threshold. The application threshold is an adaptive application threshold, which is determined according to the following method:

[0188] Adaptive application threshold = clamp (base threshold + business domain adjustment amount + data level adjustment amount + verification effectiveness adjustment amount, lower threshold, upper threshold).

[0189] The adjustment amount of the business domain is determined based on the business domain labels involved in the bloodline edge;

[0190] The data level adjustment amount is determined based on the logical layering of the data warehouse spanned by the bloodline edge;

[0191] The adjustment amount of the verification effectiveness is determined based on the proportion of the bloodline edges that were confirmed to be incorrect or invalid in the bloodline list to be verified in the previous closed-loop iteration to the bloodline edges that have been verified.

[0192] The clamp is used to limit the adaptive application threshold to a preset threshold range defined by the lower threshold and the upper threshold.

[0193] As mentioned above, the adaptive application threshold is not a static constant, but rather dynamically calculated based on multi-dimensional contextual information. It is then constrained within a reasonable range using the clamp function to ensure system stability. The calculation formula is: Adaptive Application Threshold = clamp (Base Threshold + Business Domain Adjustment + Data Layer Adjustment + Verification Efficiency Adjustment, Lower Threshold Limit, Upper Threshold Limit). Where:

[0194] The base threshold is the default confidence level screening benchmark preset by the system (e.g., 0.7), which serves as the starting reference point for all scenarios.

[0195] The adjustment amount for a business domain is determined based on the business domain label of the data assets involved in the bloodline edge. Different business domains have different tolerances for data accuracy. For example, business domains involving "financial accounting" or "regulatory reporting" are considered high-sensitivity domains, and the system will assign them a positive adjustment amount (such as +0.1) to increase the application threshold, allowing bloodline edges with slightly lower overall confidence scores to be included in the bloodline edge check list, thereby improving the recall rate of potential problematic bloodlines in high-sensitivity business domains. On the other hand, low-sensitivity domains such as "user behavior log analysis" may be assigned a negative adjustment amount (such as -0.05) to lower the application threshold, reduce the number of low-risk bloodline edges included in the check, and reduce unnecessary check overhead. The system predefines the correspondence between business domain sensitivity levels and adjustment amounts, which is maintained through configuration. The business domain adjustment amount mapping is shown in Table 1 for example:

[0196] Table 1

[0197]

[0198] The data level adjustment amount is determined based on the logical layers of the data warehouse that the bloodline edge crosses (such as the ODS raw layer, DWD detailed layer, DWS summary layer, and ADS application layer). Generally, data closer to the downstream application layer has a wider impact and higher accuracy requirements. Therefore, if a bloodline edge connects the DWD to the ADS layer, it will be assigned a larger positive adjustment amount; while a bloodline edge from the ODS to the DWD layer, being upstream and with a larger fault tolerance margin, may receive a smaller or even negative adjustment amount. The adjustment amount is determined based on the logical layer (or cross-layer type) of the data warehouse where the target end of the bloodline edge resides. The data level adjustment amount mapping is shown in Table 2 for example:

[0199] Table 2

[0200]

[0201] The verification effectiveness adjustment is dynamically generated based on the actual verification results of the previous closed-loop iteration. It is calculated as follows: the number of bloodline edges ultimately confirmed as "erroneous" or "invalid" in the previous verification list is divided by the total number of bloodline edges that have completed verification in this round. If the effectiveness is high (e.g., >60%), it indicates that a high proportion of the bloodline edges entering the verification list in the previous round indeed have problems, and the current screening criteria have good problem recall capabilities. The screening criteria can be further tightened, therefore a positive adjustment is given to increase the application threshold. If the effectiveness is low (e.g., <20%), it indicates that a large amount of verification resources are used to verify inherently reliable bloodline edges, resulting in an overly broad screening range. In this case, the application threshold should be lowered to reduce the number of bloodline edges entering the verification list, therefore a negative adjustment is given. Here, let N be the total number of bloodline edges that have completed verification in the previous round. total The number of those identified as errors / failures is N. invalid Then the verification efficiency R=N invalid / N total The adjustment amount is calculated using the following piecewise function: .

[0202] The aforementioned mapping table and algorithm, as part of the system's built-in strategy module, are recalculated for the threshold before each closed-loop iteration. All adjustments are in absolute increments of the confidence score (typically within ±0.1). The `clamp` function ultimately limits the sum of these adjustments to a preset lower threshold (e.g., 0.5) and upper threshold (e.g., 0.9), preventing excessively large adjustments from causing threshold runaway and ensuring the system's robustness and controllability. The resulting adaptive application threshold is then compared with the comprehensive confidence score of the bloodline edge to determine whether it should be included in the next round of verification.

[0203] In some embodiments of this application, by constructing a unified in-memory computing sandbox and executing standardized probe query templates within it, objective and quantifiable empirical verification of field-level lineage relationships between heterogeneous data sources can be achieved. Multi-method collaborative verification includes cross-source proactive probe verification, which includes:

[0204] Establish a memory computing sandbox independent of the underlying heterogeneous data source;

[0205] The data source adapter extracts sample data of upstream and downstream fields of the bloodline edge within the same association key range, loads the sample data into the memory computing sandbox and establishes a hash index;

[0206] The probe query template is executed within the memory computing sandbox to calculate the consistency index of field values, the consistency index of null value propagation, and the synchronization index of update time.

[0207] If the field value propagation consistency index, null value propagation consistency index, and update time synchronization index meet the corresponding verification conditions, the comprehensive confidence score of the corresponding bloodline edge will be raised to the qualified range.

[0208] If the above indicators meet the corresponding discard criteria, the corresponding bloodline edge will be marked as discarded or invalid.

[0209] If the above indicators fail to reach a definite judgment, the corresponding bloodline edge remains in a pending verification state, and the latter verification method in the multi-means collaborative verification is triggered.

[0210] As mentioned above, firstly, the system establishes an in-memory computing sandbox independent of the underlying heterogeneous data sources. This sandbox is a temporary, isolated high-performance computing environment that does not depend on any specific database engine and can uniformly handle data formats and access protocols from different data sources (such as MySQL, Oracle, Hive, Kafka, etc.).

[0211] Secondly, the system uses a pre-configured data source adapter to extract small batches of sample data within the same range of association keys (e.g., primary key, business date, user ID, etc.) from the upstream and downstream tables of the lineage edge to be verified. This sample data is loaded into the aforementioned in-memory computing sandbox, and a hash index is created for the association key field to accelerate subsequent join and comparison operations.

[0212] Next, within the in-memory computing sandbox, the system executes a set of predefined probe query templates to calculate three key verification metrics: field value propagation consistency metrics, which measure whether upstream field values ​​appear in downstream fields according to expected rules (such as direct mapping or consistency after function transformation), usually expressed as the percentage of matching records; null value propagation consistency metrics, which verify whether downstream fields are also null when upstream fields are null (or conform to preset null value handling logic), reflecting the compliance of null value propagation behavior; and update time synchronization metrics, which compare the differences in the last update timestamps of upstream and downstream records under the same sample key to determine whether there are obvious time sequence misalignments or delay anomalies.

[0213] The "verification pass conditions" and "discard judgment conditions" for the above three indicators are not temporary settings or subjective judgments, but rather a set of numerical rules pre-configured and dynamically managed by the system. All judgment thresholds are stored in the configuration center as structured parameters, including a global default threshold set applicable to general scenarios, and a coverage threshold set that is configured differently according to the business domain label (such as "financial accounting" or "precision marketing"), data layer (such as ODS, DWD, or ADS), or sensitivity level of the lineage edge. Before performing a probe, the system automatically matches the most specific threshold configuration based on the metadata attributes of the current lineage edge (such as business_domain="risk control compliance", target_layer="ADS"); if no match is found, it reverts to the global default value. Exemplary default thresholds include: field value consistency pass threshold ≥ 95%, discard threshold < 30%; null value propagation consistency pass threshold ≥ 98%, discard threshold < 20%; update time difference pass threshold ≤ 1 hour, discard threshold > 24 hours. In addition, in advanced deployment mode, the system can also adaptively fine-tune the threshold based on historical manual review feedback to continuously optimize the judgment accuracy, while the judgment logic always remains deterministic and configurable.

[0214] Subsequently, the system evaluates the above indicators according to preset judgment rules:

[0215] If all three indicators meet their respective verification conditions (e.g., value consistency ≥ 95%, null value propagation consistency rate ≥ 98%, time difference ≤ 1 hour), then the blood relationship is considered to be fully verified at the data level. The system will raise the comprehensive confidence score of the corresponding blood relationship edge to the qualified range (e.g., above 0.9) and mark it as verified.

[0216] If any indicator meets the discard criteria (e.g., value consistency <30%, null value propagation is completely inconsistent, time difference exceeds the business tolerance window), then the lineage relationship is determined to be invalid or does not exist at all. The system will mark the corresponding lineage edge as "discarded" or "invalid" and hide or archive it in the graph.

[0217] If the indicator result is between pass and fail and fails to form a definitive judgment (e.g., moderate consistency but no obvious evidence of error), the system keeps the bloodline edge in a pending verification state and automatically triggers the next verification method in the multi-means collaborative verification process (such as AI semantic reasoning or human expert review) to obtain more dimensions of evidence for comprehensive judgment.

[0218] In some embodiments of this application, to address complex scenarios such as implicit mappings, naming ambiguities, or broken links that are difficult to cover using traditional parsing methods, intelligent completion and enhancement of kinship graphs are achieved by fusing metadata semantics, contextual features, and graph topology. Multi-method collaborative verification includes AI semantic reasoning completion combined with graph context, which includes:

[0219] Encode the metadata information of each field in the kinship graph into semantic vectors and construct a vector index;

[0220] Extract the table structure information, field semantic information, script fragments, and sampling features of the bloodline edges to be verified as the graph context;

[0221] Based on the vector index, a candidate matching list is recalled, and the candidate matching list is concatenated with the graph context to form a structured prompt word input into the large language model, outputting a comprehensive semantic matching score at the name level, data type level, and semantic role level;

[0222] When a graph topology breakpoint is detected, a large language model is used in conjunction with the graph context to recommend candidate connections, and the probability of the candidate connections is calculated using a graph probability model to complete the broken links.

[0223] When implicit conversion logic is detected, the conversion function is inferred using a large language model, and an independent conversion node is inserted into the kinship graph after verification.

[0224] As mentioned above, firstly, the system converts the metadata information of all fields in the lineage graph (including field name, annotation, table name, business tag, data type, etc.) into semantic vectors through a pre-trained semantic encoding model (such as Sentence-BERT), and builds an efficient vector index (such as FAISS) based on these vectors for subsequent fast semantic retrieval.

[0225] Secondly, for the bloodline edges currently awaiting verification, the system extracts their graph context information, including: the table structure (field list and type) of the upstream and downstream tables, the semantic description of the fields to be matched, the associated ETL script fragments (if any), and the statistical characteristics of the sampled data obtained in the previous steps (such as value distribution, null value rate, etc.). This information together constitutes the context of the bloodline edge.

[0226] Next, the system uses the semantic vector of the field to be verified to recall a list of semantically similar candidate fields from the aforementioned vector index. Then, this candidate list is concatenated with the extracted contextual information to form a structured prompt, which is input into the large language model. Based on this prompt, the large language model outputs three fine-grained matching evaluation results: name-level matching score (examining field naming similarity), data type-level matching score (examining type compatibility), and semantic role-level matching score (examining whether they play the same or related roles in the business process, such as "User ID" vs. "Customer Number"). The three scores are weighted and fused to form a comprehensive semantic matching score, which is used to help determine the rationality of the lineage.

[0227] Furthermore, when the system detects a topological breakpoint in the lineage graph (e.g., a downstream table cannot be traced back to any upstream source), it initiates a link completion mechanism. At this point, the large language model combines the graph context of the nodes on both sides of the breakpoint (such as table usage, field set, task scheduling relationships, etc.) to recommend one or more possible upstream candidate tables or fields as connection suggestions. Subsequently, the system calls a graph probability model (such as a link prediction model based on random walks or graph neural networks) to calculate the connection probability of each candidate connection under the current graph structure. If the probability of a candidate connection exceeds a preset threshold, it is written into the graph as a completed lineage edge. For example, the specific implementation logic is as follows:

[0228] Feature vector construction: For each pair of candidate connections recommended by the large language model (including breakpoint nodes and candidate nodes), its graph topological features and semantic features are extracted to construct a multi-dimensional feature vector. The feature vector specifically includes: the cosine similarity of the semantic vectors of the two nodes, the data type compatibility of the fields, the Jaccard structural similarity calculated based on the number of common neighbor nodes in the graph, and the logarithmic deviation of the number of entity records.

[0229] Probability Calculation and Output: The feature vector is input into a pre-trained binary classification graph probability model (e.g., XGBoost classifier) ​​based on a tree structure or multilayer perceptron. The model is trained under supervision using historical manually labeled breakpoint connection confirmation / denial records as positive and negative samples. After processing the input feature vector, the model outputs a predicted probability value in the interval [0,1], which is the candidate connection probability.

[0230] The completion execution logic is as follows: if the probability of the candidate connection is greater than the preset automatic completion threshold (e.g., 0.6), the edge is directly written into the graph to complete the link completion; if the probability is in the undetermined range (e.g., 0.3 to 0.6), it is pushed as a high-priority candidate to the manual visual annotation feedback module.

[0231] Furthermore, when the system identifies potential implicit transformation logic (e.g., the upstream field is "date of birth" and the downstream field is "age," but the transformation function is not explicitly recorded in the code), the large language model infers possible transformation rules based on the context (e.g., "YEAR(CURRENT_DATE) - YEAR(birth_date)"). This inferred transformation function is encapsulated as an independent transformation node and inserted into the middle of the original lineage edge, transforming the previously ambiguous "field → field" mapping into a clear "field → transformation node → field" structure. After this transformation logic is verified through subsequent cross-source probing (e.g., the calculation results are consistent with the sample data), the transformation node and its associated edges officially take effect, thus achieving explicit modeling of implicit logic.

[0232] In some embodiments of this application, to handle complex bloodline boundaries that cannot be clearly determined by automated means, expert judgment is introduced through human-machine collaboration, and human experience is fed back into the AI ​​model and confidence assessment system to achieve continuous system optimization. Multi-method collaborative verification includes visual manual annotation feedback, which includes:

[0233] Bloodline boundaries that cannot be determined by previous verification methods will be pushed to the data governance workbench;

[0234] The data governance workbench displays the upstream and downstream table structures, field semantics, original scripts, detection and verification results, and AI inference suggestions for questionable bloodline edges;

[0235] Receive manual annotation results and correct the confidence state and topological relationship of the corresponding bloodline edge based on the manual annotation results;

[0236] The manually labeled results are fed back to the AI ​​semantic reasoning model as supervision samples and to the multidimensional confidence evaluation model as calibration samples, so as to update the inference parameters of the AI ​​semantic reasoning model and the weight coefficients of the multidimensional confidence evaluation model.

[0237] As mentioned above, firstly, the system pushes data to the data governance workbench if a definitive judgment cannot be reached after preliminary verification methods (including cross-source proactive detection and verification, AI semantic reasoning completion, etc.). This workbench is an interactive interface for data governance experts or business leaders.

[0238] Secondly, the system centrally displays multi-dimensional contextual information about the questionable bloodline relationship within the workbench, including: upstream and downstream table structures, field semantics, original scripts, verification results of executed verification methods, and corresponding evidence. If the preceding verification methods include cross-source proactive detection and verification, the detection and verification results are displayed; if the preceding verification methods include AI semantic reasoning completion, AI reasoning suggestions and their matching scores are displayed. This information is organized visually, facilitating rapid understanding of the technical and business context of the bloodline relationship by the user.

[0239] Subsequently, the administrators make judgments based on the displayed information and submit manual annotation results. Annotation types include, but are not limited to: "Confirmed valid", "Confirmed invalid / discarded", "Upstream needs to be supplemented", "Implicit transformation exists", etc. Based on the annotation results, the system directly corrects the confidence status of the corresponding bloodline edge (such as forcibly setting its confidence to 1.0 or 0.0) and updates its topological relationship in the bloodline graph (such as deleting edges, adding edges, or inserting transformation nodes).

[0240] Finally, the manually labeled results are reused twice by the system: on the one hand, they are used as supervisory samples in the training process of the AI ​​semantic reasoning model to fine-tune its semantic matching and link recommendation parameters; on the other hand, they are used as calibration samples in the training set of the multidimensional confidence assessment model to participate in the online update of the logistic regression weight coefficients in the next round. In this way, the knowledge of human experts is structurally embedded into the model, making the subsequent automatic verification more closely aligned with actual business understanding, forming a virtuous cycle.

[0241] In some embodiments of this application, in order to reasonably control the input of verification resources and processing time while ensuring governance quality, and to achieve a balance between efficiency and accuracy, the preset closed-loop termination condition includes any one of the following conditions:

[0242] The proportion of bloodline edges in the entire graph whose overall confidence score is lower than the application threshold is less than or equal to a first preset ratio;

[0243] In two consecutive closed-loop iterations, the absolute value of the decrease in the percentage of bloodline edges with a comprehensive confidence score lower than the application threshold is less than the second preset ratio.

[0244] The closed-loop iteration round counter reaches its maximum limit.

[0245] When the closed-loop iteration round counter reaches the maximum upper limit round, the bloodline edge that has not yet been verified and confirmed is marked as a residual questionable state.

[0246] As mentioned above, the preset closed-loop termination condition is used to determine whether the current bloodline verification closed loop should end. Termination is triggered when any of the following conditions are met:

[0247] Quality compliance criteria: The system counts the number of kinship edges in the current full graph whose overall confidence score is lower than the current application threshold, and calculates the proportion of these edges to the total number of kinship edges in the graph. If this proportion is less than or equal to a first preset ratio (e.g., 1%), it indicates that the vast majority of kinship relationships have reached the credibility standard, the overall graph quality meets the business requirements, no further iteration is needed, and the loop can be closed.

[0248] Convergence Stagnation Condition: The system compares the changes in the "proportion of low-confidence bloodline edges" in two consecutive closed-loop iterations. If the absolute value of the decrease between these two iterations (i.e., the degree of improvement) is less than the second preset ratio (e.g., 0.1%), it indicates that the verification method is no longer effective in improving the map quality, the system enters the diminishing returns stage, and the cost-effectiveness of continuing iteration is too low. Therefore, it is determined to be converged and the process is terminated.

[0249] Resource exhaustion condition: The system maintains a closed-loop iteration round counter, which increments by 1 after each complete round of multi-method collaborative verification (including detection verification, AI inference, and human feedback). If the counter reaches the preset maximum number of rounds (e.g., 5 rounds), the closed loop will be forcibly terminated regardless of the current graph quality to prevent the entire process from being blocked for a long time due to individual difficult lineage edges.

[0250] When the third condition mentioned above is triggered (i.e., the maximum number of rounds is reached), the system will uniformly mark all lineage edges that have not yet been verified and confirmed (i.e., the confidence level is still below the threshold and has not been explicitly judged manually or automatically) as "residual doubtful state". This status identifier can be used for subsequent targeted analysis, priority reordering or special governance task allocation to ensure that the problem is traceable and manageable without affecting the normal completion of the main process.

[0251] In some embodiments of this application, to address the degradation of lineage reliability caused by dynamic factors such as metadata changes, task logic adjustments, or data source anomalies during data system operation, proactive perception and timely intervention of map quality are achieved. After the preset closed-loop termination condition is met, the following further steps are included:

[0252] Continuous monitoring of the kinship graph based on time-series trajectories is performed, and the comprehensive confidence score of each kinship edge is recorded periodically;

[0253] Linear fitting is performed on the confidence of the time series of a single bloodline edge within a set sliding window. If the fitting slope is less than zero and the cumulative decrease exceeds the set trend threshold, it is judged as an abnormal continuous downward trend.

[0254] If the difference in confidence level of a single bloodline edge exceeds the set threshold for sudden drop in adjacent evaluation periods, it is judged as a sudden drop anomaly.

[0255] When an abnormal continuous downward trend or a sudden drop is detected, the evaluation dimension with the largest drop in location reliability score is identified. Combined with metadata collection logs, structural change events, task modification events, or data source interruption events within the abnormal window period, a diagnostic report containing the root cause type and the affected scope is output, and the corresponding lineage edge is re-included in the lineage list to be verified.

[0256] As described above, after the preset closed-loop termination conditions are met, the system enters the continuous monitoring phase, which specifically executes the following steps:

[0257] First, the system continuously monitors the constructed kinship map based on time-series trajectories. Following a preset evaluation cycle (e.g., daily, hourly), it periodically records the comprehensive confidence score of each kinship edge, forming time-series data with time as the horizontal axis and confidence score as the vertical axis.

[0258] Secondly, for a single bloodline edge, within a set sliding window (e.g., the most recent 7 evaluation periods), its confidence time series is linearly fitted, and the slope of the fitted line is calculated. If the slope is less than zero (indicating an overall downward trend) and the cumulative decrease in the built-in confidence of the window exceeds the set trend threshold (e.g., from 0.95 to 0.80, a decrease of 0.15), then the bloodline edge is determined to have an abnormal continuous downward trend.

[0259] Simultaneously, the system also detects confidence level changes between adjacent evaluation periods. If the difference in confidence level decrease for a bloodline edge between two consecutive periods exceeds a set sudden drop threshold (e.g., a single drop exceeding 0.2), it is determined to be a sudden drop anomaly, reflecting a possible sudden logical change or data failure.

[0260] Once any of the above anomalies (continuous downward trend anomaly or sudden drop anomaly) is detected, the system will perform root cause analysis: First, locate the evaluation dimension with the largest drop in reliability score (such as field value consistency, null value propagation consistency, semantic matching score, etc.), and then associate it with various system event logs collected during the anomaly window period, including: metadata change logs (such as field type modification), table structure change events, ETL task script modification records, scheduling task failure information, data source connection interruption alarms, etc.

[0261] Based on the above multi-source information, the system automatically generates a diagnostic report, which includes: the type of anomaly (trend or sudden drop), the main degradation dimension, the most likely root cause type (such as "upstream table structure change", "task logic reconstruction", "source data interruption", etc.), and the affected bloodline edges and their downstream dependencies.

[0262] Finally, the system re-includes the abnormal bloodline edge in the bloodline list to be checked, so that it can be prioritized in the next round of incremental check or manual review process, thereby achieving closed-loop repair of map degradation.

[0263] In some embodiments of this application, in order to transform the credibility assessment results of lineage relationships into an operable runtime control strategy, the behavior of the data aggregation task is dynamically adjusted, thereby ensuring the reliability of the data link while avoiding data pollution, resource waste, or downstream failures caused by low-quality or invalid lineages. After the preset closed-loop termination condition is met, the following further steps are included:

[0264] A state change event is generated based on the updated bloodline edge confidence state, and the state change event is published to a distributed message queue;

[0265] The data aggregation engine asynchronously consumes the state change events and sends corresponding linkage control commands through API callbacks.

[0266] For verified lineage links, issue a command to restore full aggregation;

[0267] For related links where the confidence level is within the preset rate limiting range or the status is pending verification, a rate reduction and rate limiting command is issued.

[0268] For confirmed abandoned, ineffective, or broken bloodline links, issue a pause and block command to interrupt convergence.

[0269] Before executing the rate reduction and flow limiting command or the pause and blocking command for aggregation, generate a snapshot of the parameter configuration for the corresponding aggregation task;

[0270] Once the confidence status of the restricted lineage link is reassessed and deemed acceptable, the parameter configuration snapshot is loaded to restore the aggregation task status.

[0271] As described above, after the preset closed-loop termination condition is met, the system further executes the following linkage control process:

[0272] First, when the confidence status of any lineage edge is updated (e.g., from "pending verification" to "verified," from "valid" to "pending verification," or "discarded"), the system generates a corresponding state change event. This event includes information such as the lineage edge identifier, old state, new state, change time, and confidence score, and is published to a distributed message queue (such as Kafka or RocketMQ) to achieve decoupling and high-throughput notification.

[0273] Subsequently, the data aggregation engine, acting as a consumer, asynchronously subscribes to and consumes state change events in the message queue. Upon receiving an event, the aggregation engine, based on the new state of the lineage edge, issues corresponding linkage control commands to the corresponding aggregation task scheduling module through a predefined API callback mechanism.

[0274] The specific control strategies are as follows: For verified lineage links (i.e., those with confidence levels higher than the application threshold and confirmed to be effective), a full aggregation recovery command is issued, allowing the link to transmit all data at the normal rate; for lineage links with confidence levels in the preset rate limiting range (e.g., 0.5 ≤ confidence level < 0.8) or whose status is still "pending verification", a rate reduction and rate limiting command is issued to limit their data aggregation bandwidth or sampling rate, reducing the impact of potential risks; for lineage links that are confirmed to be abandoned, invalid, or broken (e.g., manually marked as invalid or determined by AI to have no reasonable connection), a pause aggregation blocking command is issued to immediately stop the relevant data transmission and prevent erroneous data from flowing into downstream systems.

[0275] Before executing the rate reduction and flow limiting command or the pause aggregation blocking command, the system will automatically generate a snapshot of the current parameter configuration of the corresponding aggregation task, including the source / target address, field mapping rules, scheduling frequency, flow limiting threshold, retry policy and other complete runtime configurations, and persist them.

[0276] Subsequently, if the restricted lineage link is verified or repaired and its confidence status is reassessed as qualified (e.g., rising to the verification pass range), the system will load the previously saved parameter configuration snapshot and restore the original running state of the aggregation task based on the snapshot, ensuring the consistency of business logic and avoiding deviations or omissions caused by manual configuration.

[0277] Compared with existing technologies, this application discloses a data lineage confidence verification method. This method asynchronously collects multi-source heterogeneous metadata and constructs an initial lineage graph. Then, it extracts multi-dimensional feature indicators (such as resolution credibility and data consistency) for each lineage edge in the graph and calculates a comprehensive confidence score. The system can accurately filter lineage edges with confidence scores below the application threshold to the verification list. This solves the problem in existing technologies where lineage credibility cannot be quantified, leading to an excessively large scope of manual verification. It allows verification resources to focus on a small number of truly questionable lineage edges, greatly improving identification efficiency.

[0278] To address questionable kinship edges in the verification list, this solution innovatively employs a "multi-method collaborative verification" strategy. This involves sequentially or selectively performing cross-source active detection verification, AI semantic reasoning completion, and manual visual annotation feedback. This hierarchical and progressive verification mechanism allows most questionable kinships to be determined through fully automated active detection, while complex or ambiguous cases can be assisted by AI models for reasoning. Only a very few extreme cases require manual intervention. More importantly, the verification results are written back to the initial kinship graph, updating its topology and confidence status, and triggering a new round of evaluation-verification loop until the preset closed-loop termination condition is met. This iterative closed loop of "evaluation-verification-update-reevaluation" fundamentally solves the problem of continuous decay in kinship quality under existing open-loop architectures, enabling the kinship graph to possess self-healing and accuracy-increasing capabilities.

[0279] The highly trusted bloodline graph is continuously maintained through the above closed-loop mechanism, providing a reliable basis for downstream data aggregation tasks. The system can dynamically issue linked control instructions such as recovery, current limiting or blocking according to the latest confidence state of bloodline edges, effectively preventing the spread of wrong data caused by invalid bloodline relationships, and improving the robustness and reliability of the entire data governance system.

[0280] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical memory, etc.) having computer-usable program code embodied therein.

[0281] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the present application. It is to be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions are provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus generate means for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams. Figure 1 one or more flows and / or blocks Figure 1 a device for implementing functions specified in one or more blocks.

[0282] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction means that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams. Figure 1 one or more flows and / or blocks Figure 1 a function specified in one or more blocks.

[0283] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams. Figure 1 one or more flows and / or blocks Figure 1 steps for implementing functions specified in one or more blocks.

[0284] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for verifying the confidence level of pedigree in data, characterized in that, include: Asynchronously collect metadata from multi-source heterogeneous data systems, and extract field-level lineage mappings through hierarchical parsing to construct an initial lineage relationship graph containing lineage edges and their attribute information; Multidimensional feature indicators are extracted from each bloodline edge in the initial bloodline relationship map, and a comprehensive confidence score for each bloodline edge is calculated based on a multidimensional confidence assessment model. Bloodline edges with a comprehensive confidence score lower than the application threshold are filtered into the bloodline list to be verified. Perform multi-method collaborative verification on the questionable bloodline edges in the bloodline list to be verified. The multi-method collaborative verification includes at least two of the following: cross-source active detection verification, AI semantic reasoning completion combined with graph context, and manual visual annotation feedback. Selectively trigger the next verification method based on the judgment result of the previous verification method. The judgment result of the multi-means collaborative verification is written back to the initial blood relationship graph, the topology, confidence status and verification audit information of the corresponding blood relationship edge are updated, and the comprehensive confidence score calculation, blood relationship list screening and multi-means collaborative verification are re-executed based on the updated blood relationship graph until the preset closed-loop termination condition is met. The extraction of field-level lineage mapping through hierarchical parsing includes: Prioritize structured semantic parsing to extract field-level source-target mappings from SQL class processing tasks with available source code; When structured semantic parsing coverage is insufficient, cross-language script data flow analysis is performed to recover hidden lineage links from script code; When information gaps still exist, perform fallback inference based on task topology dependencies to infer field-level lineage mapping by combining task dependencies, field naming similarity, and data type compatibility. When different parsing methods produce conflicting lineage mappings for the same field, conflict arbitration is performed based on the basic score of the parsing credibility of each parsing method. The lineage mapping with the highest basic score is retained as the main lineage edge in the active graph, and the remaining lineage mappings are written into the alternative attributes. If the difference in the basic scores of different analysis methods is less than the preset conflict threshold, then multiple conflicting bloodline edges will be written into the graph at the same time, the conflict status field of the corresponding bloodline edge will be marked as a conflict pending state, and an initial low confidence value will be assigned for subsequent comprehensive confidence assessment and verification screening. The cross-source active detection verification includes: Establish a memory computing sandbox independent of the underlying heterogeneous data source; The data source adapter extracts sample data from upstream and downstream fields of the bloodline edge within the same association key range, loads the sample data into the memory computing sandbox, and establishes a hash index. The probe query template is executed within the memory computing sandbox to calculate the consistency index of field values, the consistency index of null value propagation, and the synchronization index of update time. If the field value propagation consistency index, null value propagation consistency index, and update time synchronization index meet the corresponding verification conditions, the comprehensive confidence score of the corresponding bloodline edge will be raised to the qualified range. If the above indicators meet the corresponding discard criteria, the corresponding bloodline edge will be marked as discarded or invalid. If the above indicators fail to reach a definite judgment, the corresponding bloodline edge remains in a pending verification state, and the latter verification method in the multi-means collaborative verification is triggered.

2. The method as described in claim 1, characterized in that, Multidimensional feature indicators include parsing credibility, metadata integrity, data consistency verification, timeliness, and link integrity; The comprehensive confidence score is obtained by weighted fusion of the scores of each dimension and their corresponding weight coefficients in the multidimensional feature indicators. The weight coefficients are updated online through a logistic regression model. Specifically, after each closed-loop iteration, the bloodline edges that are confirmed to be real through the multi-method collaborative verification are used as positive samples, and the bloodline edges that are confirmed to be erroneous or invalid through the multi-method collaborative verification are used as negative samples. The scores of each dimension are used as the model input for training, and the feature coefficients of each dimension obtained from the training are normalized to obtain the updated weight coefficients.

3. The method as described in claim 1 or 2, characterized in that, The application threshold is an adaptive application threshold, which is determined according to the following method: Adaptive application threshold = clamp (base threshold + business domain adjustment amount + data level adjustment amount + verification effectiveness adjustment amount, lower threshold, upper threshold). The adjustment amount of the business domain is determined based on the business domain labels involved in the bloodline edge; The data level adjustment amount is determined based on the logical layering of the data warehouse spanned by the bloodline edge; The adjustment amount of the verification effectiveness is determined based on the proportion of the bloodline edges that were confirmed to be incorrect or invalid in the bloodline list to be verified in the previous closed-loop iteration to the bloodline edges that have been verified. The clamp is used to limit the adaptive application threshold to a preset threshold range defined by the lower threshold and the upper threshold.

4. The method as described in claim 1, characterized in that, The multi-method collaborative verification includes AI semantic reasoning completion combined with graph context, which includes: Encode the metadata information of each field in the kinship graph into semantic vectors and construct a vector index; Extract the table structure information, field semantic information, script fragments, and sampling features of the bloodline edges to be verified as the graph context; Based on the vector index, a candidate matching list is recalled, and the candidate matching list is concatenated with the graph context to form a structured prompt word, which is then input into a large language model to output a comprehensive semantic matching score at the name level, data type level, and semantic role level. When a graph topology breakpoint is detected, a large language model is used in conjunction with the graph context to recommend candidate connections, and the probability of the candidate connections is calculated using a graph probability model to complete the broken links. When implicit conversion logic is detected, the conversion function is inferred using a large language model, and an independent conversion node is inserted into the kinship graph after verification.

5. The method as described in claim 1, characterized in that, The multi-method collaborative verification includes manual visual annotation feedback, which includes: Bloodline boundaries that cannot be determined by previous verification methods will be pushed to the data governance workbench; The data governance workbench displays the upstream and downstream table structures, field semantics, original scripts, detection and verification results, and AI inference suggestions for questionable bloodline edges; Receive manual annotation results and correct the confidence state and topological relationship of the corresponding bloodline edge based on the manual annotation results; The manually labeled results are fed back to the AI ​​semantic reasoning model as supervision samples and to the multidimensional confidence evaluation model as calibration samples, so as to update the inference parameters of the AI ​​semantic reasoning model and the weight coefficients of the multidimensional confidence evaluation model.

6. The method as described in claim 1, characterized in that, The preset closed-loop termination condition includes any one of the following conditions: The proportion of bloodline edges in the entire graph whose overall confidence score is lower than the application threshold is less than or equal to a first preset ratio; In two consecutive closed-loop iterations, the absolute value of the decrease in the percentage of bloodline edges with a comprehensive confidence score lower than the application threshold is less than the second preset ratio. The closed-loop iteration round counter reaches its maximum limit. When the closed-loop iteration round counter reaches the maximum upper limit round, the bloodline edge that has not yet been verified and confirmed is marked as a residual questionable state.

7. The method as described in claim 1, characterized in that, After the preset closed-loop termination condition is met, the following is also included: Continuous monitoring of the kinship graph based on time-series trajectories is performed, and the comprehensive confidence score of each kinship edge is recorded periodically; Linear fitting is performed on the confidence of the time series of a single bloodline edge within a set sliding window. If the fitting slope is less than zero and the cumulative decrease exceeds the set trend threshold, it is judged as an abnormal continuous downward trend. If the difference in confidence level of a single bloodline edge exceeds the set threshold for sudden drop in adjacent evaluation periods, it is judged as a sudden drop anomaly. When an abnormal continuous downward trend or a sudden drop is detected, the evaluation dimension with the largest drop in location reliability score is identified. Combined with metadata collection logs, structural change events, task modification events, or data source interruption events within the abnormal window period, a diagnostic report containing the root cause type and the affected scope is output, and the corresponding lineage edge is re-included in the lineage list to be verified.

8. The method as described in claim 1, characterized in that, After the preset closed-loop termination condition is met, the following is also included: A state change event is generated based on the updated bloodline edge confidence state, and the state change event is published to a distributed message queue; The data aggregation engine asynchronously consumes the state change events and sends corresponding linkage control commands through API callbacks. For verified lineage links, issue a command to restore full aggregation; For related links where the confidence level is within the preset rate limiting range or the status is pending verification, a rate reduction and rate limiting command is issued. For confirmed abandoned, ineffective, or broken bloodline links, issue a pause and block command to interrupt convergence. Before executing the rate reduction and flow limiting command or the pause and blocking command for aggregation, generate a snapshot of the parameter configuration for the corresponding aggregation task; Once the confidence status of the restricted lineage link is reassessed and deemed acceptable, the parameter configuration snapshot is loaded to restore the aggregation task status.

Images