Metadata bloodline analysis method, system, device and medium

Abstract

The application provides a metadata blood relationship analysis method, system, device and medium. The method is applied to a metadata blood relationship analysis system. The metadata blood relationship analysis system comprises an acquisition module, an analysis module, an inference module and a verification module. Through the hierarchical cooperation and closed-loop optimization architecture design, the four core modules of acquisition, analysis, inference and verification are set from top to bottom. The data input, explicit analysis, intelligent completion, result quality control and standardized output are sequentially completed. The closed-loop link of acquisition-analysis-inference-verification-refining is formed. Each module is iterated to realize the metadata analysis capability of full-scene coverage and high-precision evolution. The analysis efficiency and quality are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a method, system, device and medium for metadata lineage analysis. Background Technology

[0002] Current metadata lineage parsing technologies mainly fall into two categories, but both have shortcomings: Rule-based matching, a mainstream approach, matches data flow information using a pre-defined rule base. However, when dealing with multi-format metadata, it requires manually customizing differentiated rules, resulting in low adaptation efficiency and a lack of universality. While traditional machine learning-based optimization schemes introduce model training modules such as data preprocessing, feature extraction, and decision trees, they largely rely on batch training. Adding new data or formats necessitates full retraining, leading to high resource consumption and a lack of real-time updates. Furthermore, traditional models struggle to effectively handle graph-structured metadata lineage data, failing to fully capture node interaction features and implicit relationships, resulting in limited parsing accuracy and a tendency for omissions or misjudgments. Therefore, a metadata lineage parsing method that adapts to multiple formats, supports incremental training, and accurately mines implicit relationships is urgently needed to address these issues. Summary of the Invention

[0003] In view of the above problems, the present invention is proposed to provide a metadata lineage resolution method, system, device and medium that overcomes or at least partially solves the above problems.

[0004] To achieve the above and other related objectives, this invention provides a metadata lineage resolution method, applied to a metadata lineage resolution system. The metadata lineage resolution system includes a data acquisition module, a parsing module, an inference module, and a verification module. The method includes: The acquisition module employs differentiated targeted acquisition technology to collect two types of core metadata—data warehouse modeling and business database—as well as end-to-end auxiliary data for data development tasks. After cleaning and standardization preprocessing, target data is obtained. The target data includes input and output parameters of data development tasks, SQL static data, and key logs of the computing engine. The parsing module performs parameter parsing, SQL static parsing, and calculation engine parsing on the target data in parallel. It also performs deduplication, completion, and conflict marking on the three types of parsing results to form a high-confidence fusion result. For implicit lineage data that still cannot be identified or has conflicts, it is pushed to the inference module. The XGBoost and GCN fusion model is continuously or periodically trained using the confirmed historical lineage analysis results as labeled samples through the inference module. The latest fusion model is used to infer the implicit lineage data input from the analysis module. After business rule correction and conflict verification, a new lineage edge with completeness and confidence level is output and merged with the high-confidence fusion result to obtain the optimized overall analysis result. The verification module uses a dual mechanism to verify the overall parsing results, and the parts that do not meet the standards are returned to the corresponding modules for re-parsing or reasoning, forming a closed-loop optimization. After the verification is qualified, the data is packaged and output in a unified format. The output data includes basic lineage information, parsing method traceability tags, and visualization adaptation data.

[0005] Optionally, the acquisition module includes a targeted acquisition unit, an auxiliary acquisition unit, and a preprocessing unit. The acquisition module employs differentiated targeted acquisition technology to collect core metadata from two categories—data warehouse modeling and business databases—as well as end-to-end auxiliary data for data development tasks. After cleaning and standardization preprocessing, target data is obtained, including: Based on the differences in data types and their hierarchical levels, data warehouse modeling metadata and business database metadata are collected separately through targeted acquisition units. The data warehouse modeling metadata includes logical layer information such as the data warehouse's layered architecture, model design documents, table relationship definitions, and field meaning descriptions. The business database metadata includes physical layer information such as table structure, field types, primary key and foreign key relationships, stored procedures, and triggers. The auxiliary acquisition unit collects end-to-end auxiliary data for the data development task; the end-to-end auxiliary data includes basic task information, task operation logs, and historical correlation data. The preprocessing unit performs deduplication, null value filling, and IQR outlier removal on the data warehouse modeling metadata, the business database metadata, and the end-to-end auxiliary data. Then, the cleaned heterogeneous data is uniformly converted into standard JSON format, and its core fields are extracted to generate standardized target data. The core fields include data type, system, associated task ID, and business tag.

[0006] Optionally, the parsing module includes a task parsing unit, a static parsing unit, an engine parsing unit, and a result fusion unit. The parsing module performs parallel parameter parsing, SQL static parsing, and computation engine parsing on the target data, and performs deduplication, completion, and conflict marking on the three types of parsing results to form a high-confidence fusion result. For implicit lineage data that still cannot be identified or contains conflicts, it is pushed to the inference module, including: The task parsing unit parses the input and output parameters of the data development task, identifies the source and target metadata at the task level, establishes the task-level lineage, associates task dependency parameters to complete the context task link, and outputs the task parameter parsing results. The static parsing unit calls the syntax parser to perform lexical analysis on SQL-type tasks to generate tokens and constructs an abstract syntax tree; it then traverses the abstract syntax tree to extract the relationships between tables and fields, generating precise lineage relationships at the field level. The engine parsing unit analyzes the input and output data paths, fragmentation information, and serialization formats in the execution logs of streaming and batch processing computing engine tasks. Combined with the task execution plan, it reconstructs the data flow, fills the parsing gaps in non-SQL and streaming scenarios, and forms log parsing results. The result fusion unit performs deduplication, completion, and conflict verification on the task parameter parsing results, the field-level precise lineage relationship, and the log parsing results to form a high-confidence fusion result; it simultaneously marks contradictory edges to be verified and filters out implicit lineage data that cannot be identified by parameters, SQL, and logs and pushes them to the inference module.

[0007] Optionally, the inference module includes a preparation unit, a training unit, and a completion unit. The inference module uses confirmed historical lineage analysis results as labeled samples to continuously or periodically train the XGBoost and GCN fusion model. It then uses the current optimal fusion model to infer the implicit lineage data input from the analysis module, followed by business rule correction and conflict verification, outputting a new lineage edge with completeness and sufficient confidence. This new edge is then merged with the high-confidence fusion result to obtain the optimized overall analysis result, including: The preparation unit uses the historical kinship analysis results confirmed by the verification module as annotation samples, and simultaneously retrieves other historical data provided by the acquisition module; only metadata features and task-related features are extracted from the other historical data, and standardized feature vectors are generated after unified encoding and normalization. Using the labeled samples as labels, the XGBoost model and GCN model are trained respectively by the training unit using the standardized feature vectors, and the current optimal fusion model is obtained by weighted fusion and K-fold cross-validation. The completion unit uses the current optimal fusion model to infer the implicit bloodline data pushed by the parsing module and outputs potential bloodline associations. After being corrected by business rules, the reasonableness probability of the bloodline relationship to be verified is calculated for the conflict verification mark. Only the new bloodline edges with the confidence level are merged with the high-confidence fusion results to form the optimized overall parsing result.

[0008] Optionally, the verification module utilizes a dual mechanism to verify the overall parsing result, returning any substandard parts to the corresponding module for re-parse or reasoning, forming a closed-loop optimization. After successful verification, the data is packaged and output in a unified format. The output data includes basic lineage information, parsing method tracing tags, and visualization adaptation data, including: The verification module performs manual sampling inspections on the key business domain lineage links of the overall analysis results according to a preset sampling ratio, and compares the qualified results with the preset historical backtracking threshold. The unqualified analysis results are directed back to the analysis module or inference module for re-optimization, forming a closed-loop quality control. After the double verification is passed, the output data is encapsulated in a unified JSON format; the output data includes basic lineage information, parsing method traceability tags, and lineage link visualization adaptation data.

[0009] Optionally, the verification module performs manual sampling inspection of key business domain lineage links in the overall analysis results according to a preset sampling ratio, compares the qualified results with a preset historical backtracking threshold, and redirects the substandard analysis results back to the analysis module or inference module for re-optimization, forming a closed-loop quality control, including: The verification module randomly checks the key business domain lineage links in the overall analysis results according to a preset sampling ratio, and screens qualified results. The qualified results are compared with known correct links in historical lineage data to calculate the accuracy and recall of the qualified results; If the accuracy or recall rate is less than the corresponding preset historical backtracking threshold, it is determined to be substandard, and the substandard parsing results are directed back to the parsing module or inference module for re-optimization, thereby realizing closed-loop quality control of verification, feedback and optimization.

[0010] Secondly, the present invention also provides a metadata lineage resolution system, the system comprising: The data acquisition module is used to collect core metadata from two categories—data warehouse modeling and business database—as well as end-to-end auxiliary data from data development tasks using differentiated targeted acquisition technology. After cleaning and standardization preprocessing, target data is obtained. The target data includes input and output parameters of data development tasks, SQL static data, and key logs of the computing engine. The parsing module is used to perform parameter parsing, SQL static parsing and calculation engine parsing in parallel on the target data, and to deduplicate, complete and mark conflicts in the three types of parsing results to form a high-confidence fusion result. For implicit lineage data that still cannot be identified or has conflicts, it is pushed to the inference module. The inference module is used to continuously or periodically train the XGBoost and GCN fusion model using the confirmed historical lineage analysis results as labeled samples; it uses the latest fusion model to infer the implicit lineage data input from the analysis module, and then, after business rule correction and conflict verification, outputs a new lineage edge that is complete and meets the confidence standard, which is then merged with the high-confidence fusion result to obtain the optimized overall analysis result. The verification module is used to verify the overall parsing result using a dual mechanism. The part that does not meet the standard is returned to the corresponding module for re-parsing or reasoning, forming a closed-loop optimization. After the verification is qualified, it is packaged and output in a unified format. The output data includes basic lineage information, parsing method traceability tags and visualization adaptation data.

[0011] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to cause the electronic device to perform the metadata lineage resolution method as described above.

[0012] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the metadata lineage resolution method as described above.

[0013] Fifthly, the present invention provides a computer program product including computer program code, which, when run on a computer, causes the computer to implement the method described above.

[0014] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects: This invention employs a layered, collaborative, and closed-loop optimized architecture, setting up four core modules from top to bottom: data acquisition, parsing, inference, and verification. These modules sequentially complete data input, explicit parsing, intelligent completion, result quality control, and standardized output, forming a closed-loop chain of acquisition-parsing-inference-verification-reprocessing. Each module iterates cyclically, collectively achieving metadata parsing capabilities that cover all scenarios and evolve with high precision, significantly improving parsing efficiency and quality. Attached Figure Description

[0015] Figure 1 The diagram shows a flowchart of a metadata lineage resolution method according to an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the functional modules of a metadata lineage analysis system in one embodiment of the present invention;

[0017] Figure 3 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0018] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0019] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0020] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0021] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0022] Unless otherwise stated, the term "multiple" means two or more.

[0023] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0024] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0025] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] Please see Figure 1 An embodiment of the present invention provides a metadata lineage resolution method, applied to a metadata lineage resolution system. The metadata lineage resolution system includes a data acquisition module, a parsing module, an inference module, and a verification module. The method may include the following steps S10~S40: Step S10: Using differentiated targeted acquisition technology, the acquisition module collects two types of core metadata (data warehouse modeling and business database) and full-link auxiliary data of data development tasks. After cleaning and standardization preprocessing, target data is obtained. The target data includes input and output parameters of data development tasks, SQL static data, and key logs of the computing engine.

[0027] Among them, differentiated targeted acquisition technology is used to efficiently and accurately acquire the required data by adopting targeted acquisition strategies and tools based on different data sources and data types.

[0028] To address the differences in characteristics between data warehouse modeling metadata and business database metadata, a "one-category-one-policy" approach is adopted for specific interface adaptation: For data warehouse modeling metadata, logical layer information such as data warehouse layer (ODS, DWD, DWS, ADS) metadata, model design documents, and table relationships is deeply collected through data warehouse management tool interfaces such as the Hive Metastore API. For business database metadata, physical layer information such as business database table structure, field types, primary keys, foreign keys, and stored procedures is collected through JDBC interfaces combined with database system views. Simultaneously, native adaptation is provided for mainstream databases and data warehouse tools such as MySQL, Oracle, Hive, and ClickHouse, achieving multi-environment compatibility and ensuring stable operation of the collection module across different technology stacks.

[0029] Data warehouse modeling metadata is used to characterize the data warehouse's layered architecture, model design, table relationships, field meanings, etc. Specifically, it includes logical layer information such as data warehouse layered metadata, model design documents, and table relationships. It is the core basis for building the data warehouse's logical system and sorting out data relationships.

[0030] Business database metadata characterizes the table structure, field types, primary and foreign key relationships, stored procedures, triggers, etc., of the business database. It is the fundamental data reflecting the relationship between the physical storage of the business database and the business logic. Collecting this metadata through a dedicated interface ensures the traceability of the business data chain.

[0031] End-to-end auxiliary data is used to characterize basic information, operation logs, and historical correlation data of data development tasks, which helps to understand and analyze the data flow process. Specifically, it includes basic task information (task name, execution cycle, parameter configuration, etc.), task operation logs (Spark / Flink Event logs, etc.), and historical correlation data (metadata change records, task tags, historical lineage results, etc.), providing complete contextual support for subsequent analysis and reasoning.

[0032] Target data is used to characterize the data warehouse modeling metadata, business database metadata, and end-to-end auxiliary data after preprocessing including cleaning (duplicate removal, null value filling, and outlier removal using the IQR quartile method) and standardization (unified JSON format, extraction of core fields). Target data is the core data carrier connecting the acquisition module with the downstream parsing and inference modules; it specifically includes input and output parameters of data development tasks, static SQL data, and key logs from the computing engine.

[0033] The input and output parameters of the data development task are used to characterize the relevant parameters that describe the input and output of the task, and are used to generate task-level lineage relationships.

[0034] SQL static data is used to characterize and describe static data and metadata information in SQL queries or data processing scripts, and is used to generate field-level lineage relationships.

[0035] The computation engine critical log is used to characterize and describe the key log information generated by the computation engine during task execution, and is used to supplement the lineage in streaming data scenarios.

[0036] In the specific implementation, firstly, the acquisition module employs differentiated targeted acquisition technology to simultaneously collect two types of core metadata: data warehouse modeling and business database, as well as end-to-end auxiliary data for data development tasks. Then, the collected data warehouse modeling metadata, business database metadata, and end-to-end auxiliary data for data development tasks are cleaned (including deduplication, null value imputation, and IQR quartile removal of outliers) and standardized preprocessed (including conversion to JSON format and extraction of core fields such as data type, system, associated task ID, and business tags) to obtain target data containing input and output parameters of data development tasks, SQL static data, and key logs from the computing engine. This ensures comprehensive and accurate parsing and management of metadata lineage.

[0037] Step S20: The parsing module performs parallel parameter parsing, SQL static parsing, and calculation engine parsing on the target data. The three types of parsing results are deduplicated, completed, and conflict marked to form a high-confidence fusion result. For implicit lineage data that still cannot be identified or has conflicts, it is pushed to the inference module.

[0038] The fusion result is used to characterize the set of high-confidence explicit lineage relationships formed after deduplication, completion, and conflict verification of the same batch of target data through multiple parsing methods such as task parameter parsing, SQL static parsing, and computing engine log parsing. This set only contains links that can be directly verified by rules, constitutes the benchmark view for subsequent machine learning modules to complete implicit lineages, and is used to merge with the inference results to form a complete and accurate data flow and dependency panorama.

[0039] Implicit lineage data is used to characterize lineage relationships that are not explicitly recorded in task parameters, SQL scripts, or computation engine logs and can only be inferred through semantic features, graph topology, or historical behavior patterns. Typical scenarios include dynamic table names, reflected SQL, temporary scripts, implicit cross-database conversions, and temporary tasks without logs. Such dependencies cannot be obtained directly through rule parsing and need to be inferred and completed by machine learning models based on historical samples and graph structures.

[0040] In its implementation, the parsing module receives the target data output by the acquisition module and then performs parallel execution parameter parsing on the input and output parameters of the data development task, SQL static data, and key logs of the computing engine. This includes SQL static parsing (parses table / field-level lineage in SQL scripts) and event log parsing (extracts runtime input and output paths generated by Spark / Flink, etc.). Subsequently, after deduplication, completion, and conflict verification, the three parsing results output a fusion result containing only high-confidence explicit lineage relationships. For lineage edges that are still unidentifiable or contradictory in this result (i.e., implicit lineage data), their feature vectors and context are marked as "data to be inferred" and pushed to the inference module. This fusion result contains only explicit links, serving as a baseline view for subsequent machine learning completion and verification, providing a foundation for a comprehensive and accurate understanding and management of data flow and dependencies.

[0041] Step S30: Using the confirmed historical lineage analysis results as labeled samples, the XGBoost and GCN fusion model is continuously or periodically trained through the inference module; the latest fusion model is used to infer the implicit lineage data input from the analysis module, and after business rule correction and conflict verification, a new lineage edge with completeness and confidence level is output and merged with the high-confidence fusion result to obtain the optimized overall analysis result.

[0042] Among them, the historical lineage analysis results are used to characterize the set of "correct lineage links" that are continuously aggregated and persisted by the collection module after being confirmed by the verification module. They serve as positive sample benchmarks for subsequent machine learning model training and are used to learn the correlation patterns between metadata.

[0043] Labeled samples are used to characterize, in the context of metadata lineage, lineage pairs (source-target metadata and their associated types) that have been verified as "correct" or "incorrect", and are labeled with explicit tags (1 indicates that there is a lineage, 0 indicates that there is no lineage) for supervised training of machine learning models.

[0044] The XGBoost model is used to represent a structured feature learner based on gradient boosting decision trees. It takes metadata and static attributes of tasks (such as task labels, field types, business domains, etc.) as input and outputs the probability of lineage in the "explicit feature" dimension to capture interpretable regular association patterns.

[0045] The GCN model (Graph Convolutional Network model) is used to represent the generation of node embeddings through multiple rounds of neighborhood aggregation iterations, taking the topology of the historical lineage graph as input, mining deep graph patterns that cannot be seen by dependency attributes alone, and outputting the probability of lineage existence in the "graph structure" dimension.

[0046] The fusion model is used to characterize the linear weighting of the XGBoost explicit probability and GCN graph structure probability of the same metadata pair according to the optimal weight of the validation set, and outputs a unique lineage confidence. It integrates explicit and implicit features to provide a unified decision basis for the final completion or rejection of lineage edges.

[0047] New lineage edges are used to represent lineage relationships (source / target metadata pairs) that are inferred or identified for the first time in this parsing cycle and have not yet been verified. These include both explicit edges newly discovered during parsing and implicit edges completed by model inference. They can only be incorporated into the historical truth database after being confirmed by the verification module.

[0048] The overall analysis results are used to characterize the high-confidence explicit lineage obtained through "task parameter analysis + SQL static analysis + computing engine log analysis", which is combined with the implicit lineage completed by the fusion model inference, and then deduplicated, completed, and conflict-checked to form a complete lineage map. The results include lineage information at the task level, field level, and full-link dimension of the streaming link.

[0049] In its implementation, the inference module only receives "unidentifiable / conflicting latent kinship data" and their feature vectors from the parsing module. It uses historical kinship parsing results (verified truth values) provided by the acquisition module as labeled samples, continuously or periodically training a fusion model of XGBoost (responsible for mining structured features such as task labels and field types) and GCN (responsible for mining graph structure features of historical kinship graphs). Subsequently, the latest fusion model is used to infer latent kinship metadata, generating potential kinship relationships. After correction by business rules and auxiliary verification of conflicting kinships, new kinship edges with sufficient confidence are output and merged with the high-confidence fusion results from the parsing module to form an optimized overall parsing result. This process accurately completes the rule-based parsing of unidentifiable kinship relationships through the machine learning fusion model, solving the problem of incomplete coverage, and verifies and corrects conflicting results, providing high-quality data support for subsequent double verification and standardized output.

[0050] Step S40: The overall parsing result is verified by the verification module using a dual mechanism. Any substandard parts are returned to the corresponding module for re-parsing or reasoning, forming a closed-loop optimization. After the verification is successful, the data is packaged and output in a unified format. The output data includes basic lineage information, parsing method traceability tags, and visualization adaptation data.

[0051] Among them, the dual mechanism is used to characterize the two-layer verification system of "manual sampling inspection + historical backtracking comparison". The manual review is carried out by sampling according to business domain, and the accuracy and recall rate are calculated with the verified truth database. The results that do not meet the standards are automatically returned for retraining or re-analysis, forming a closed loop optimization to ensure the accuracy and reliability of the overall lineage analysis results.

[0052] The output data, which represents the overall parsing results, is encapsulated in a unified JSON / CSV format after double verification. It includes basic lineage information, parsing method traceability tags, and a visual adaptation structure, which can be directly consumed by data governance, quality control and other systems, achieving plug-and-play functionality.

[0053] In its implementation, after receiving the overall parsing results pushed by the inference module, the verification module employs a dual mechanism of "manual sampling + historical backtesting comparison": manual review is conducted by stratified sampling according to business domain, and the results are compared with the verification truth database to calculate accuracy and recall. Substandard results are automatically returned to the parsing or inference module for reprocessing, forming a closed-loop optimization. Once verified, the data is encapsulated and output in a unified JSON format. The output data includes basic lineage links, parsing method traceability tags, and a visual adaptation structure, allowing direct integration with the data governance platform.

[0054] In this embodiment, a hierarchical collaborative and closed-loop optimized architecture is adopted, with four core modules set up from top to bottom: data collection, parsing, inference, and verification. These modules sequentially complete data input, explicit parsing, intelligent completion, result quality control, and standardized output, forming a closed-loop link of data collection, parsing, inference, verification, and reprocessing. Each module iterates cyclically to jointly achieve metadata parsing capabilities that cover all scenarios and evolve with high precision, significantly improving parsing efficiency and quality.

[0055] Based on the foregoing embodiments, a second embodiment of the metadata lineage analysis method of the present invention is proposed. In this embodiment, step S10 may include the following sub-steps S101~S103: Sub-step S101: Based on the differences in data types and their hierarchical levels, data warehouse modeling metadata and business database metadata are collected separately through the targeted collection unit. The data warehouse modeling metadata includes logical layer information such as the data warehouse's layered architecture, model design documents, table relationship definitions, and field meaning descriptions. The business database metadata includes physical layer information such as table structure, field types, primary key and foreign key relationships, stored procedures, and triggers.

[0056] The acquisition module includes a targeted acquisition unit, an auxiliary acquisition unit, and a preprocessing unit.

[0057] The targeted data collection unit uses differentiated interfaces (Hive Metastore API, JDBC, etc.) to accurately pull core metadata such as model design, table structure, and primary and foreign keys from the "logical layer" data warehouse modeling metadata and the "physical layer" business database metadata.

[0058] The auxiliary acquisition unit is responsible for collecting auxiliary data across the entire data development task chain, including basic task information, operation logs, historical change records, task tags and descriptions, etc., to provide context for subsequent analysis and reasoning.

[0059] The preprocessing unit performs cleaning operations such as deduplication, null value filling, and outlier removal (IQR) on the heterogeneous raw data returned by the first two units, and converts it into a standard JSON format. It then extracts core fields (data type, system, associated task ID, business tag, etc.) to generate standardized target data.

[0060] Data warehouse modeling metadata is used to characterize "design diagram" type information generated during the logical design phase of the data warehouse, which is independent of the platform implementation. This includes: subject area division, layer (ODS / DWD / DWS / ADS) definition, model ER diagram, table-level relationships, Chinese meaning of fields, indicator definitions, partitioning strategies, etc., and belongs to the logical layer assets.

[0061] Business database metadata is used to characterize the "runtime" information of the production business system database at the physical implementation level. It includes: table structure (column names, data types, precision), primary keys / foreign keys, indexes, views, stored procedures, triggers, sequences, constraints, table / field comments, and other system catalog data, which belong to physical layer assets.

[0062] In practical implementation, based on the differences in the data's hierarchical location, a "specialized interface adaptation" strategy is adopted through targeted collection units to collect two types of core metadata in a differentiated manner: data warehouse modeling metadata (focusing on lineage and skeleton) that is deeply collected through warehouse management tool interfaces such as the Hive Metastore API, including information such as data warehouse layering (ODS / DWD / DWS / ADS), model design documents, table relationship definitions, and field meaning descriptions; and business database metadata (focusing on data constraints) that is collected through JDBC interfaces and database system views, including business database table structure, field types, primary keys, foreign keys, stored procedures, triggers, etc., and is compatible with multiple environments such as MySQL, Oracle, PostgreSQL, Hive, and ClickHouse.

[0063] Sub-step S102 involves collecting full-link auxiliary data for the data development task through an auxiliary acquisition unit; the full-link auxiliary data includes basic task information, task operation logs, and historical related data.

[0064] Among them, the full-link auxiliary data is used to characterize the supporting information covering the entire lifecycle of data development tasks from "definition to operation to change", and is used to restore the task execution context, supplement lineage clues and provide training samples; specifically, it includes: basic task information, task operation logs and historical related data.

[0065] Basic task information includes: task name, project / business domain, execution cycle (scheduling frequency), task parameter configuration (input / output paths, database / table names, variables, etc.), task type (offline SQL, Spark, Flink, data synchronization, etc.), and dependencies (list of upstream task IDs).

[0066] Task execution logs include: computation engine event logs (JobStart / JobEnd, StageCompleted, TaskEnd, etc. in Spark / Flink) and scheduling platform execution records (start / end time, running status, number of retries, failure reason, and resource usage).

[0067] Historical data includes: metadata history change records (snapshots of table / field additions, modifications, and deletions), task tags and descriptions (such as "risk control statistics", "transaction analysis" and corresponding documents), and verified historical lineage resolution results (positive sample library).

[0068] In practice, the auxiliary acquisition unit pulls all the supporting data (including basic task information, task operation logs, and historical related data) generated throughout the entire lifecycle of the data development task ("definition-run-change") in one go, providing a complete context and training material for subsequent explicit parsing and implicit reasoning.

[0069] Sub-step S103 involves performing deduplication, null value filling, and IQR outlier removal on the data warehouse modeling metadata, the business database metadata, and the end-to-end auxiliary data through a preprocessing unit; then, the cleaned heterogeneous data is uniformly converted into standard JSON format, and its core fields are extracted to generate standardized target data; wherein, the core fields include data type, system to which it belongs, associated task ID, and business tag.

[0070] Among them, the core fields are used to characterize the minimum common attribute set extracted by the preprocessing unit for unified heterogeneous metadata, and are used for cross-module identification and association; specifically, they include: data type, system to which it belongs, associated task ID, and business tag.

[0071] IQR outlier removal is used to characterize the upper quartile Q3 and lower quartile Q1 of the core field, resulting in the interquartile range IQR = Q3 - Q1. Samples less than Q1 - 1.5 × IQR or greater than Q3 + 1.5 × IQR are identified as outliers and removed to ensure data quality.

[0072] In the specific implementation, the preprocessing unit performs deduplication, null value filling, and IQR outlier removal on the data warehouse modeling metadata, business database metadata, and end-to-end auxiliary data to remove duplicate, invalid, and incorrectly formatted data. Subsequently, the cleaned heterogeneous data is uniformly converted into standard JSON format, and core fields (data type, system, associated task ID, and business tag) are extracted to generate standardized target data for subsequent parsing and inference.

[0073] In this embodiment, the data warehouse modeling metadata and business database metadata are retrieved by the targeted retrieval unit. At the same time, the auxiliary collection unit gathers the historical related data of the data development task (including metadata change records, task tags, description information, etc.). After being cleaned and standardized by the preprocessing unit, standardized target data is generated for subsequent analysis and reasoning.

[0074] Based on the foregoing embodiments, a third embodiment of the metadata lineage analysis method of the present invention is proposed. In this embodiment, step S20 may include the following sub-steps S201~S204: Sub-step S201 involves parsing the input and output parameters of the data development task through the task parsing unit, identifying the source metadata and target metadata at the task level, establishing task-level lineage relationships, associating task dependency parameters to complete the context task link, and outputting the task parameter parsing results.

[0075] The parsing module includes a task parsing unit, a static parsing unit, an engine parsing unit, and a result fusion unit.

[0076] The task parsing unit is used to read the input and output parameters of data development tasks (such as the --input and --output parameters of Spark tasks, the reader / writer parameters of DataX running scripts, and the configuration of the source / target of the synchronization task) and the task dependency configuration, identify the task-level source and target metadata, establish the task-level lineage, and complete the upstream and downstream task links.

[0077] The static parsing unit is used to perform lexical / syntactic analysis on SQL scripts, generate an abstract syntax tree, extract the relationships between tables and fields, and output precise field-level lineage.

[0078] The engine parsing unit is used to parse key event logs of computing engines such as Spark / Flink, extract runtime input / output paths, sharding and serialization information, and fill the gaps in non-SQL and streaming scenarios.

[0079] The result fusion unit is used to deduplicate, complete, and verify conflicts of the three types of parsing results to form a high-confidence fusion result, and to filter out unidentifiable latent lineage data and push it to the inference module.

[0080] Task-level lineage relationships are used to represent only the "table / topic" granularity, describing which sources a task reads from and which targets it writes to, and associating it with its upstream and downstream tasks, without involving field mapping.

[0081] The task parameter parsing results are used to characterize the structured records output by the task parsing unit, which contain the above-mentioned task-level lineage relationships and upstream and downstream dependencies, and are used for subsequent fusion and completion.

[0082] In the specific implementation, the task parsing unit parses the input and output parameters of the data development task, identifies the task-level source (source table / field) and target metadata (destination table / field), and establishes the task-level lineage; at the same time, it associates task dependency parameters to complete the context task link and outputs the task parameter parsing results.

[0083] Sub-step S202 involves using a static parsing unit to call a syntax parser to perform lexical analysis on SQL-type tasks, generate tokens, and construct an abstract syntax tree. The abstract syntax tree is then traversed to extract the relationships between tables and between fields, generating precise lineage relationships at the field level.

[0084] In the specific implementation, the static parsing unit calls the syntax parser generated by ANTLR for SQL class tasks. First, lexical analysis is performed to generate tokens (such as SELECT, FROM, JOIN, field name, table name), and then an abstract syntax tree (AST) is constructed. Subsequently, the AST is traversed to extract the relationships between tables and fields (such as tables associated with JOIN, field mappings in SELECT clauses, and field associations in WHERE conditions), and the precise lineage relationship at the field level is output.

[0085] Sub-step S203 involves using the engine parsing unit to parse the input and output data paths, sharding information, and serialization formats in the execution logs of streaming and batch processing computing engine tasks. This process combines the task execution plan to reconstruct the data flow, filling the parsing gaps between non-SQL and streaming scenarios, and generating log parsing results.

[0086] In the specific implementation, the engine parsing unit collects key logs such as JobSubmitEvent, StageCompletedEvent, and TaskEndEvent from Spark / Flink to extract core information such as input and output data paths, sharding information, and serialization formats. Then, it combines the task execution plan to reconstruct the streaming / batch data flow path, supplement the metadata lineage in streaming data scenarios (such as the lineage relationship between Kafka Topics and computation result tables), fill the parsing gaps in non-SQL and streaming scenarios, and output the log parsing results.

[0087] In sub-step S204, the result fusion unit performs deduplication, completion, and conflict verification on the task parameter parsing results, the field-level precise lineage relationship, and the log parsing results to form a high-confidence fusion result; simultaneously, it marks the contradictory edges to be verified and filters out the implicit lineage data that cannot be identified by parameters, SQL, and logs and pushes it to the inference module.

[0088] In the specific implementation, the result fusion unit performs deduplication (removing duplicate lineages), completion (associating task-level and field-level lineages), and conflict verification (marking contradictory edges as "to be verified" and pushing them to the inference module) on the above three types of parsing results to form a high-confidence fusion result; at the same time, implicit lineage data that cannot be identified by parameters, SQL, and Event (such as implicit lineage data without explicit parameter / script / Event records) are filtered out and pushed to the inference module to clarify the target range for subsequent inference.

[0089] In this embodiment, three methods—task parameter parsing, SQL static parsing, and Spark / Flink Event parsing—are used in parallel to output high-confidence explicit lineage relationships after deduplication, completion, and conflict verification. Implicit lineage data that cannot be identified by parameters, SQL, and Event, as well as contradictory lineage edges, are then filtered out and pushed to the inference module for completion and verification.

[0090] Based on the foregoing embodiments, a fourth embodiment of the metadata lineage analysis method of the present invention is proposed. In this embodiment, step S30 may include the following sub-steps S301~S303: In sub-step S301, the preparation unit uses the historical bloodline analysis results confirmed by the verification module as annotation samples, and simultaneously retrieves other historical data provided by the acquisition module; only metadata features and task-related features are extracted from the other historical data, and standardized feature vectors are generated after unified encoding and normalization.

[0091] The reasoning module includes a preparation unit, a training unit, and a completion unit.

[0092] The preparation unit is used to import confirmed historical lineage truth values ​​and metadata change records, task tags and descriptions, extract metadata features and task-related features, and generate standardized feature vectors to provide a unified input for model training and inference.

[0093] The training unit trains XGBoost (explicit structured features) and GCN (graph topology features) separately using feature vectors, fuses them according to the optimal weights on the validation set, and tunes the parameters through K-fold cross-validation to output the current optimal fusion model.

[0094] The completion unit is used to load the latest fusion model, perform inference on unidentified or conflicting implicit lineage data, and output lineage probabilities. After being corrected by business rules, it assigns reasonable probabilities to conflicting edges, merges new lineage edges with the explicit fusion results with the confidence level, and forms an optimized overall parsing result.

[0095] Other historical data, used to represent all past information in the acquisition module except for "verified bloodline edges," is used solely for feature generation. Specifically, it includes: (1) Metadata side: Metadata history change records (table / field addition, modification, deletion snapshots), data types, business tags, subject domains, field naming rules, comments and other static attributes; (2) Task side: basic task information (name, type, project, execution cycle, input and output paths), task description text, keywords, tags, task execution sequence (start / end time, number of retries) and the lineage topology of other tasks in the same project (co-occurrence frequency, upstream and downstream density, etc.).

[0096] The data mentioned above are only used to generate standardized feature vectors for the model to learn potential lineage patterns and are not used for annotation.

[0097] Standardized feature vectors are used to represent fixed-dimensional numerical vectors generated by extracting original features from the metadata side (data type, business domain, field naming rules, etc.) and the task side (task type, descriptive keywords, temporal features, same project lineage patterns, etc.) using verified historical lineage truth values ​​as labels. These vectors are then uniformly encoded, filled with missing values, and normalized. They can be directly used for training and inference of XGBoost and GCN models.

[0098] In the specific implementation, the preparation unit uses the historical lineage analysis results confirmed by the verification module as annotation samples, and simultaneously retrieves other historical data (including metadata historical change records, task tags, and task description information) provided by the acquisition module. Subsequently, only the metadata features (data type, business tags, the business domain, and field naming rules) and task association features (type of associated task, keywords of task description, temporal association of task execution, and lineage pattern of other tasks under the same project) are extracted from the other historical data. After unified encoding and normalization, standardized feature vectors are generated for model training and inference.

[0099] In sub-step S302, the XGBoost model and GCN model are trained using the labeled samples as labels through the training unit and the standardized feature vectors, respectively. The optimal fusion model is obtained by tuning the parameters through weighted fusion and K-fold cross-validation.

[0100] Among them, the current optimal fusion model is used to characterize the unified inference model obtained by linearly weighting the dominant feature probability output by XGBoost and the graph structure probability output by GCN according to the optimal weight after K-fold cross-validation evaluation in the latest round of training, with the optimal validation set AUC or F1 as the criterion. It is used for the implicit lineage completion and conflict verification in the next cycle.

[0101] In the specific implementation, the training unit uses labeled samples (i.e., the matching relationship between "metadata and associated data") as labels, and divides the standardized feature vectors into training set, test set and validation set according to a preset ratio (e.g., 7:2:1), respectively, and then uses the XGBoost model (mining explicit structured features) and the GCN model (mining graph topological implicit patterns). After training, the model is weighted and fused according to the optimal weight of the validation set, and then the parameters are tuned through K-fold cross-validation to output the current optimal fused model.

[0102] In sub-step S303, the completion unit uses the current optimal fusion model to infer the implicit bloodline data pushed by the parsing module and outputs potential bloodline associations. After being corrected by business rules, the reasonableness probability is calculated for the bloodline relationship to be verified marked by conflict verification. Only the new bloodline edges with the confidence level are merged with the fusion results with the high confidence level to form the optimized overall parsing result.

[0103] Among them, potential lineage relationships are used to characterize the "candidate edges" output by the current optimal fusion model for inference of implicit lineage data. These are "source metadata → target metadata" relationships that have not yet been explicitly confirmed by parameters / SQL / Events but have a high probability calculated by the fusion model. They can only become formal lineage relationships after business rules are corrected and confidence levels are filtered.

[0104] In the specific implementation, the completion unit uses the current optimal fusion model to infer the implicit lineage data pushed by the parsing module and outputs potential lineage associations (source / destination, association type); then, the potential lineage associations are corrected according to business rules (priority for the same business domain, high probability for the same task label, etc.), and the rationality probability is calculated for the lineage relationship to be verified marked by conflict verification; only the new lineage edges with the confidence level are merged with the high-confidence fusion results in the parsing module to form the optimized overall parsing result.

[0105] In this embodiment, a machine learning fusion model is used to reason about historical data, supplementing implicit lineage relationships that cannot be identified by parameters, SQL, and Event, thus solving the problem of incomplete parsing coverage. At the same time, the model probability is used to provide a reasonableness score for conflict edges, assisting in verification and reducing misjudgments, thereby comprehensively improving the overall parsing accuracy.

[0106] Based on the foregoing embodiments, a fifth embodiment of the metadata lineage resolution method of the present invention is proposed. In this embodiment, step S40 may include the following sub-steps S401~S402: Sub-step S401 involves manually sampling the key business domain lineage links of the overall analysis results according to a preset sampling ratio through the verification module, comparing the qualified results with the preset historical backtracking threshold, and redirecting the unqualified analysis results back to the analysis module or inference module for re-optimization, thus forming a closed-loop quality control.

[0107] Among them, the preset sampling ratio is used to characterize the minimum sampling ratio (≥10%) set in advance in the manual verification process, which is used to ensure the coverage of the lineage verification of key business domains and to ensure the representativeness of manual sampling.

[0108] A preset historical backtracking threshold is used to characterize the lower limits of accuracy (≥95%) and recall (≥90%) in the historical backtracking verification process. This threshold is used to quantify the pass / fail standards, and results that do not meet the standards trigger backtracking optimization.

[0109] In the specific implementation, after receiving the overall parsing results output by the inference module, the verification module performs manual sampling inspection on the key business domain lineage links of the overall parsing results according to a preset sampling ratio (≥10%). Subsequently, the qualified results are compared with the historical truth database. If the accuracy is <95% or the recall is <90%, the unqualified parsing results are directed to the parsing module to re-optimize the parsing rules, or to the inference module to re-infer, forming a closed-loop quality control.

[0110] In sub-step S402, after the double verification passes, the output data is encapsulated in a unified JSON format; the output data includes basic lineage information, parsing method traceability tags, and lineage link visualization adaptation data.

[0111] In the specific implementation, after the double verification passes, the verification module encapsulates the output data in a unified JSON format. This output data includes basic lineage information (source, flow path, processing task, destination), parsing method traceability tags (task parameters / SQL / Event / machine learning), and lineage link visualization adaptation data (ECharts, Neo4j format). At the same time, a REST API is provided, which can be directly connected to data governance / quality systems to achieve highly reliable, traceable, and easy-to-implement lineage output.

[0112] Furthermore, in one embodiment, sub-step S401 may include the following sub-steps A10~A30: Sub-step A10 involves selecting key business domain lineage links through the verification module and randomly sampling the overall analysis results according to a preset sampling ratio to select qualified results. Sub-step A20: Compare the qualified results with known correct links in the historical lineage data, and calculate the accuracy and recall of the qualified results; In sub-step A30, if the accuracy or recall rate is less than the corresponding preset historical backtracking threshold, it is determined to be substandard, and the substandard parsing results are directed back to the parsing module or inference module for re-optimization, thereby realizing closed-loop quality control of verification, feedback and optimization.

[0113] In the specific implementation, the verification module randomly checks the key business domain lineage links in the overall parsing results according to a preset sampling ratio (≥10%), and selects qualified results. Then, the qualified results are compared with the known correct links in the historical lineage data to calculate the accuracy and recall of the qualified results. If the accuracy is <95% or the recall is less than 90%, it is judged as unqualified, and the unqualified parsing results are directed back to the parsing module or inference module for re-optimization, forming a closed-loop quality control of verification, feedback and optimization.

[0114] In this embodiment, the verification module uses a dual mechanism of "manual sampling + historical backtracking" to ensure the accuracy and reliability of the parsing results. It also outputs the results in a unified JSON format (including basic lineage, parsing method tags, and visualization data) and REST API to improve usability and compatibility, providing high-quality lineage support for data governance and quality control.

[0115] Based on the same inventive concept, the sixth embodiment of this invention also provides a metadata lineage resolution system corresponding to the metadata lineage resolution method of the foregoing embodiments. Since the principle of the system in the sixth embodiment of this invention for solving the problem is similar to the metadata lineage resolution method of the foregoing embodiments, the implementation of the system can refer to the implementation of the method; repeated details will not be elaborated further. Please refer to... Figure 2 The present invention provides a metadata lineage analysis system, the system of which may include: The acquisition module 10 is used to collect core metadata of two types, data warehouse modeling and business database, as well as the full-link auxiliary data of data development tasks, using differentiated targeted acquisition technology. After cleaning and standardization preprocessing, target data is obtained. The target data includes the input and output parameters of data development tasks, SQL static data, and key logs of the computing engine. The parsing module 20 is used to perform parameter parsing, SQL static parsing and calculation engine parsing in parallel on the target data, and to deduplicate, complete and mark the three types of parsing results to form a high-confidence fusion result. For the implicit lineage data that still cannot be identified or has conflicts, it is pushed to the inference module. The inference module 30 is used to continuously or periodically train the fusion model of XGBoost and GCN using the confirmed historical lineage analysis results as labeled samples; it uses the current optimal fusion model to infer the implicit lineage data input from the analysis module, and then, after business rule correction and conflict verification, outputs a new lineage edge that is completed and has a high confidence level, and merges it with the high confidence fusion result to obtain the optimized overall analysis result. The verification module 40 is used to verify the overall parsing result using a dual mechanism. The part that does not meet the standard is returned to the corresponding module for re-parsing or reasoning, forming a closed-loop optimization. After the verification is qualified, it is packaged and output in a unified format. The output data includes basic lineage information, parsing method traceability tags and visualization adaptation data.

[0116] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described metadata lineage resolution method.

[0117] Figure 3 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 3 The general will label all buses as bus systems.

[0118] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0119] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0120] In this embodiment of the invention, the memory 402 is used to store various categories of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 includes various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may include various applications, such as a media player, browser, etc., for implementing various application services. The metadata lineage resolution method provided in this embodiment of the invention can be included in the application program 4022.

[0121] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the metadata lineage resolution method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0122] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0123] In summary, this invention employs a layered, collaborative, and closed-loop optimized architecture design, setting up four core modules from top to bottom: data acquisition, parsing, inference, and verification. These modules sequentially complete data input, explicit parsing, intelligent completion, result quality control, and standardized output, forming a closed-loop chain of acquisition-parsing-inference-verification-reprocessing. Each module iterates cyclically, collectively achieving full-scenario coverage and high-precision evolution of metadata parsing capabilities, significantly improving parsing efficiency and quality.

[0124] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for resolving metadata lineage, characterized in that, The method is applied to a metadata lineage resolution system, which includes a data acquisition module, a parsing module, an inference module, and a verification module. The acquisition module employs differentiated targeted acquisition technology to collect two types of core metadata—data warehouse modeling and business database—as well as end-to-end auxiliary data for data development tasks. After cleaning and standardization preprocessing, target data is obtained. The target data includes input and output parameters of data development tasks, SQL static data, and key logs of the computing engine. The parsing module performs parameter parsing, SQL static parsing, and calculation engine parsing on the target data in parallel. It also performs deduplication, completion, and conflict marking on the three types of parsing results to form a high-confidence fusion result. For implicit lineage data that still cannot be identified or has conflicts, it is pushed to the inference module. The XGBoost and GCN fusion model is continuously or periodically trained using the confirmed historical lineage analysis results as labeled samples through the inference module. The current optimal fusion model is used to infer the implicit lineage data input from the analysis module. After business rule correction and conflict verification, a new lineage edge with completeness and confidence level is output and merged with the high-confidence fusion result to obtain the optimized overall analysis result. The verification module uses a dual mechanism to verify the overall parsing results, and the parts that do not meet the standards are returned to the corresponding modules for re-parsing or reasoning, forming a closed-loop optimization. After the verification is qualified, the data is packaged and output in a unified format. The output data includes basic lineage information, parsing method traceability tags, and visualization adaptation data.

2. The method according to claim 1, characterized in that, The acquisition module includes a targeted acquisition unit, an auxiliary acquisition unit, and a preprocessing unit. The acquisition module employs differentiated targeted acquisition technology to collect core metadata from two categories: data warehouse modeling and business databases, as well as end-to-end auxiliary data from data development tasks. After cleaning and standardization preprocessing, target data is obtained, including: Based on the differences in data types and their hierarchical levels, data warehouse modeling metadata and business database metadata are collected separately through targeted acquisition units. The data warehouse modeling metadata includes logical layer information such as the data warehouse's layered architecture, model design documents, table relationship definitions, and field meaning descriptions. The business database metadata includes physical layer information such as table structure, field types, primary key and foreign key relationships, stored procedures, and triggers. The auxiliary acquisition unit collects end-to-end auxiliary data for the data development task; the end-to-end auxiliary data includes basic task information, task operation logs, and historical correlation data. The preprocessing unit performs deduplication, null value filling, and IQR outlier removal on the data warehouse modeling metadata, the business database metadata, and the end-to-end auxiliary data. Then, the cleaned heterogeneous data is uniformly converted into standard JSON format, and its core fields are extracted to generate standardized target data. The core fields include data type, system, associated task ID, and business tag.

3. The method according to claim 1, characterized in that, The parsing module includes a task parsing unit, a static parsing unit, an engine parsing unit, and a result fusion unit. The parsing module performs parallel parameter parsing, SQL static parsing, and computation engine parsing on the target data, and performs deduplication, completion, and conflict marking on the three types of parsing results to form a high-confidence fusion result. For implicit lineage data that still cannot be identified or contains conflicts, it is pushed to the inference module, including: The task parsing unit parses the input and output parameters of the data development task, identifies the source and target metadata at the task level, establishes the task-level lineage, associates task dependency parameters to complete the context task link, and outputs the task parameter parsing results. The static parsing unit calls the syntax parser to perform lexical analysis on SQL-type tasks to generate tokens and constructs an abstract syntax tree; it then traverses the abstract syntax tree to extract the relationships between tables and fields, generating precise lineage relationships at the field level. The engine parsing unit analyzes the input and output data paths, fragmentation information, and serialization formats in the execution logs of streaming and batch processing computing engine tasks. Combined with the task execution plan, it reconstructs the data flow, fills the parsing gaps in non-SQL and streaming scenarios, and forms log parsing results. The result fusion unit performs deduplication, completion, and conflict verification on the task parameter parsing results, the field-level precise lineage relationship, and the log parsing results to form a high-confidence fusion result; it simultaneously marks contradictory edges to be verified and filters out implicit lineage data that cannot be identified by parameters, SQL, and logs and pushes them to the inference module.

4. The method according to claim 1, characterized in that, The inference module includes a preparation unit, a training unit, and a completion unit. The inference module uses confirmed historical lineage analysis results as labeled samples to continuously or periodically train the XGBoost and GCN fusion model. It then uses the current optimal fusion model to infer the implicit lineage data input from the analysis module, followed by business rule correction and conflict verification. Finally, it outputs a new lineage edge with completeness and sufficient confidence, which is merged with the high-confidence fusion result to obtain the optimized overall analysis result, including: The preparation unit uses the historical kinship analysis results confirmed by the verification module as annotation samples, and simultaneously retrieves other historical data provided by the acquisition module; only metadata features and task-related features are extracted from the other historical data, and standardized feature vectors are generated after unified encoding and normalization. Using the labeled samples as labels, the XGBoost model and GCN model are trained respectively by the training unit using the standardized feature vectors, and the current optimal fusion model is obtained by weighted fusion and K-fold cross-validation. The completion unit uses the current optimal fusion model to infer the implicit bloodline data pushed by the parsing module and outputs potential bloodline associations. After being corrected by business rules, the reasonableness probability of the bloodline relationship to be verified is calculated for the conflict verification mark. Only the new bloodline edges with the confidence level are merged with the high-confidence fusion results to form the optimized overall parsing result.

5. The method according to claim 1, characterized in that, The verification module uses a dual mechanism to verify the overall parsing result, and the unqualified part is returned to the corresponding module for re-parse or reasoning, forming a closed-loop optimization. After successful verification, the product is packaged and output according to a standardized format. The output data includes basic lineage information, parsing method tracing tags, and visualization adaptation data, including: The verification module performs manual sampling inspections on the key business domain lineage links of the overall analysis results according to a preset sampling ratio, and compares the qualified results with the preset historical backtracking threshold. The unqualified analysis results are directed back to the analysis module or inference module for re-optimization, forming a closed-loop quality control. After the double verification is passed, the output data is encapsulated in a unified JSON format; the output data includes basic lineage information, parsing method traceability tags, and lineage link visualization adaptation data.

6. The method according to claim 5, characterized in that, The verification module performs manual sampling inspections on key business domain lineage links in the overall analysis results according to a preset sampling ratio, compares the qualified results with preset historical backtracking thresholds, and redirects unqualified analysis results back to the analysis module or inference module for re-optimization, forming a closed-loop quality control, including: The verification module randomly checks the key business domain lineage links in the overall analysis results according to a preset sampling ratio, and screens qualified results. The qualified results are compared with known correct links in historical lineage data to calculate the accuracy and recall of the qualified results; If the accuracy or recall rate is less than the corresponding preset historical backtracking threshold, it is determined to be substandard, and the substandard parsing results are directed back to the parsing module or inference module for re-optimization, thereby realizing closed-loop quality control of verification, feedback and optimization.

7. A metadata lineage resolution system, characterized in that, Applied to, the system includes: The data acquisition module is used to collect core metadata from two categories—data warehouse modeling and business database—as well as end-to-end auxiliary data from data development tasks using differentiated targeted acquisition technology. After cleaning and standardization preprocessing, target data is obtained. The target data includes input and output parameters of data development tasks, SQL static data, and key logs of the computing engine. The parsing module is used to perform parameter parsing, SQL static parsing and calculation engine parsing in parallel on the target data, and to deduplicate, complete and mark conflicts in the three types of parsing results to form a high-confidence fusion result. For implicit lineage data that still cannot be identified or has conflicts, it is pushed to the inference module. The inference module is used to continuously or periodically train the XGBoost and GCN fusion model using the confirmed historical lineage analysis results as labeled samples; it uses the current optimal fusion model to infer the implicit lineage data input from the analysis module, and then, after business rule correction and conflict verification, outputs a new lineage edge that is completed and meets the confidence standard, which is then merged with the high-confidence fusion result to obtain the optimized overall analysis result. The verification module is used to verify the overall parsing result using a dual mechanism. The part that does not meet the standard is returned to the corresponding module for re-parsing or reasoning, forming a closed-loop optimization. After the verification is qualified, it is packaged and output in a unified format. The output data includes basic lineage information, parsing method traceability tags and visualization adaptation data.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method of any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method of any one of claims 1 to 6.

Images