Metadata automatic collection and bloodline analysis method for data middle platform

Abstract

The metadata automatic acquisition and blood relationship analysis method disclosed by the application is characterized in that: the method comprises the following steps: checking an access data source through a pluggable plug-in pool and classifying in layers; performing multi-modal cooperative acquisition of metadata on the data source in parallel; cleaning the metadata and constructing an entity correlation model; standardizing a full-type task script and binding entities; adopting an AST hard analysis and a large model soft analysis double-engine to process the script to generate a multi-granularity initial blood relationship graph; removing closed-loop edges, removing duplicates and normalizing the graph; storing the metadata and the graph in layers, asynchronously reanalyzing and covering old blood relationships according to a change event. The application solves the problems of insufficient automatic acquisition and analysis of metadata and blood relationship and insufficient dynamic linkage, reduces the operation and maintenance workload of manual sorting and rectification, and can steadily improve the stability of automatic management of metadata and blood relationship links of the middle platform.

Description

Technical Field

[0001] This invention belongs to the field of metadata governance and automated lineage resolution technology for data middle platforms, specifically involving a method for automatic metadata collection and lineage resolution for data middle platforms. Background Technology

[0002] As the types of data sources connected to enterprise data platforms continue to increase, and the volume of offline processing, real-time computing, and custom script processing continues to grow, the adaptability of existing metadata governance solutions is no longer sufficient to keep up with the pace of business iteration. Traditional metadata collection often relies on a single timed full-data collection mode, lacking a collaborative mechanism for incremental verification and real-time monitoring. This not only results in high computational consumption but also makes it prone to issues such as missed or delayed data collection due to changes in database and table fields.

[0003] The mainstream lineage parsing tools on the market have limited compatibility, and can only parse standardized SQL statements and output table-level dependencies. For commonly used business SQL statements such as multi-level nested SQL, custom UDF functions, Python and Shell compiled scripts, Flink streaming tasks, and DataX synchronous configuration files, there are currently no mature automated parsing methods. Operation and maintenance personnel can only manually sort out the links, resulting in low overall parsing efficiency and high link error rate.

[0004] Furthermore, in the existing governance system, metadata storage and lineage storage are independent, lacking a linkage mechanism. When data tables are added or deleted, fields are renamed, or field attributes are adjusted, existing historical lineage links cannot be automatically identified as invalid. The system retains a large number of erroneous and invalid dependencies, directly causing functions such as the middle platform data map, asset traceability, and impact prediction to fail. At the same time, most existing publicly available technical solutions focus on a single collection or parsing module, with an overall architecture biased towards theoretical design. They lack detailed implementation rules for link verification, linked updates, and anomaly fault tolerance, resulting in insufficient engineering adaptability and difficulty in supporting enterprises' routine and automated data governance work. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic metadata collection and lineage analysis method for data middleware, which solves the problems of insufficient automatic collection, analysis and dynamic linkage of metadata and lineage.

[0006] The technical solution adopted in this invention is: an automatic metadata collection and lineage resolution method for data middleware, comprising the following steps: Step 1: Verify and categorize the access data sources through the pluggable plugin pool; Step 2: Perform multimodal collaborative acquisition of metadata on the data source from Step 1 in parallel; Step 3: Clean the metadata from Step 2 and build an entity association model; Step 4: Standardize the preprocessing scripts for all types of tasks and bind them to the entities from Step 3; Step 5: Use a dual-engine approach of AST hard parsing and large-model soft parsing to process the script in Step 4 to generate multi-granularity initial lineage maps; Step 6: Remove closed loop edges, remove duplicates, and normalize the graph from Step 5. Step 7: Store the metadata from Step 3 and the graph from Step 6 in the database in layers, listen for change events, asynchronously re-parse and overwrite the old lineage.

[0007] The invention is further characterized in that, Step 1 specifically includes the following steps: Step 1.1: Define a unified abstract collection interface and build a pluggable data source plugin pool. The plugin has three basic interfaces: connection verification interface, metadata capture interface, and exception callback interface. Step 1.2: When adding a new data source, encapsulate the adaptation driver package according to the interface specification, and perform format verification, version compatibility verification and signature validity verification on the driver package. After the verification is passed, upload it to the plugin pool to complete the registration and adaptation. Step 1.3: After the plugin registration is completed, perform a smoke test by calling the metadata capture interface to perform a lightweight liveness detection query and verify whether the returned data structure conforms to the unified output specification. Step 1.4: Enter the data source connection address, access account, port and access permissions, and complete the port connectivity verification, account permission verification and database table read permission verification in sequence; Step 1.5: After the permission verification is passed, the data source is marked with five categories based on the data flow hierarchy: ODS source layer, DWD detail layer, DWS summary layer, DM application layer, and external business system layer, thus completing the standardized classification.

[0008] Step 2 specifically includes the following steps: Step 2.1: Connect the standardized and categorized data sources from Step 1 into the data collection process, and configure three types of collection modes to be executed in parallel: first-time full snapshot differential collection, periodic incremental comparison collection, and real-time monitoring and collection of CDC logs. Step 2.2: The data collection task adopts a distributed execution with cluster sharding. The full snapshot differential collection first traverses all database table fields, generates a unique MD5 snapshot fingerprint, and generates a baseline snapshot file and stores it in the version repository. The timed incremental comparison collection is executed according to a custom Cron cycle. It combines the data table update time and the snapshot MD5 fingerprint to determine the validity of data changes and only updates data with different fingerprints. The CDC log real-time monitoring collection listens to the database Binlog transaction log, WAL write-ahead log and Flink cluster operation log, captures DDL change events, field change events and partition change events, delivers them through the message queue, and completes the metadata update after parsing by the consumer.

[0009] Step 3 specifically includes the following steps: Step 3.1: Perform encoding standardization, null value filtering, and special character removal on the metadata collected in Step 2; Step 3.2: Merge entity data collected repeatedly from multiple channels using the database name, table name, and field name as unique primary keys; Step 3.3: Construct a five-element entity association model including data source, library, table, field, and task. The primary key of the entity is generated with a globally unique ID using the snowflake algorithm. The hierarchical relationship of each entity is bound by the database foreign key index: one data source corresponds to multiple libraries, one library corresponds to multiple tables, one table corresponds to multiple fields, and the relationship between tasks and tables is many-to-many, with the mapping relationship stored through an intermediate association table.

[0010] Step 4 specifically includes the following steps: Step 4.1: Connect to the API interface of the middle platform scheduling platform to batch collect offline SQL scripts, stored procedures, Python scripts, Shell scripts, synchronous configurations and real-time streaming tasks; Step 4.2: Remove comments, blank lines, and dynamic variables from the collected scripts line by line, and split multi-statement compound scripts into independent execution units according to execution keywords; Step 4.3: Divide the tasks into three categories based on script suffixes and syntax features: SQL tasks, code tasks, and configuration tasks. Step 4.4: Use the fuzzy matching of database table names plus manual calibration mode to bind the task ID with the standardized database table entity generated in step 3 to establish a one-to-one business relationship.

[0011] Step 5 specifically includes the following steps: Step 5.1: Use AST hard parsing to process the standardized SQL script output in step 4, call the dedicated parser of the corresponding computing engine to generate an abstract syntax tree, traverse the Query node, From node, Join node, Select node and Operator node of the syntax tree, extract table-level, field-level and operator-level dependencies layer by layer, and generate three-level structured lineage data of table-level, field-level and operator-level layer by layer. Step 5.2: Use a fine-tuned bloodline model with soft parsing to process non-standard tasks. Input script content and database table structure through fixed template prompts, and the model outputs structured JSON bloodline data. Step 5.3: Integrate the output results of the dual engines of AST hard analysis and fine-tuned bloodline large model soft analysis to generate a multi-granularity initial bloodline map.

[0012] Step 6 specifically includes the following steps: Step 6.1: Use the depth-first traversal algorithm to traverse the lineage links of the multi-granularity initial lineage graph output in Step 5, and identify and remove closed-loop cyclic dependency edges. Step 6.2: Use the task ID, upstream entity ID, and downstream entity ID as unique verification keys to complete the deduplication of duplicate lineages; Step 6.3: Convert the bloodline relationship into a node-edge structure with attributes. The nodes store metadata entity information, and the edges store operator, task, time and confidence attributes to complete the graph normalization.

[0013] Step 7 specifically includes the following steps: Step 7.1: Store the standardized metadata output in Step 3 into a relational database, store the normalized kinship graph output in Step 6 into a graph database, and store the change log into a time series database. Step 7.2: Retain metadata and corresponding lineage snapshots from the most recent versions; Step 7.3: Listen for metadata addition events, deletion events, renaming events, and attribute change events. After an event is triggered, the historical lineage is invalidated, and the corresponding task is scheduled to re-parse the data to complete the dynamic iteration update of the lineage.

[0014] The beneficial effects of this invention are as follows: This invention provides an automatic metadata collection and lineage resolution method for data middleware platforms. Through a unified interface of a pluggable plugin pool, it enables customized access to heterogeneous data sources. It integrates three collection modes: full, incremental, and CDC, ensuring the integrity and real-time performance of metadata collection. It adopts a dual-engine architecture of AST hard parsing and large-model soft parsing, expanding the scope of automated parsing for non-standard tasks and generating multi-granularity refined lineage graphs. Combined with depth-first traversal deduplication and change event-driven mechanisms, it achieves graph normalization and dynamic linkage iterative updates, eliminating invalid lineage accumulation, ensuring that metadata and lineage links are synchronized and consistent, significantly reducing manual operation and maintenance costs and improving the stability of middleware governance. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the automatic metadata collection and lineage analysis method for data middleware according to the present invention. Detailed Implementation

[0016] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0017] This invention provides an automatic metadata collection and lineage resolution method for data platforms. It achieves seamless integration of multi-source heterogeneous data sources without custom development through a pluggable data source plugin pool. It integrates three collection modalities: full snapshot differential, timed incremental comparison, and CDC real-time log monitoring, to automate the collection of metadata across structural, business, and operational dimensions. A dual-engine architecture combining hard parsing of the AST abstract syntax tree and soft parsing of a fine-tuned lineage model is employed to generate refined lineage graphs at the table, field, and operator levels. A depth-first traversal algorithm is used to identify and eliminate closed-loop dependencies, and a unique verification key is used to deduplicate duplicate lineages, achieving lineage graph normalization. Based on metadata version snapshots and change event-driven mechanisms, dynamic linkage updates between metadata and lineage links are achieved. This invention solves the technical problems of high data source adaptation costs, coarse-grained lineage resolution, inability to automatically parse non-standard tasks, and asynchronous metadata and lineage iteration in traditional metadata governance solutions. It can be widely applied to governance scenarios such as enterprise data platform asset traceability, change impact analysis, and data compliance auditing. Figure 1 As shown, it includes the following steps: S1. Data Source Plugin Registration and Standardized Verification: A custom unified abstract collection interface is defined to build a pluggable data source plugin pool. The plugin has three basic interfaces: connection verification, metadata capture, and exception callback. After the data source connection parameters are entered, port connectivity, account permissions, and database / table read permissions are verified in sequence. In accordance with the data platform layering specifications, the data source is labeled with five categories: ODS source layer, DWD detail layer, DWS summary layer, DM application layer, and external business system layer, thus completing the standardized classification.

[0018] S2. Multimodal Collaborative Automated Metadata Collection: Configures three collection modes to be executed in parallel, namely, full snapshot differential collection upon initial access, periodic incremental comparison collection, and real-time monitoring and collection of CDC logs; synchronously captures structural metadata, business metadata, and runtime metadata. Incremental collection adopts a dual verification mechanism of timestamp comparison and snapshot MD5 differential, and CDC collection caches change events through a message queue; collection tasks are executed in a distributed manner using cluster sharding, and single task exceptions are automatically retried 3 times. Retry failures trigger independent alarms without interrupting the overall process.

[0019] S3. Metadata Cleaning and Normalization and Platform Modeling: Perform encoding unification, null value filtering, and special character removal on the original metadata. Merge duplicate entities using the database name, table name, and field name as unique primary keys. Construct a five-element entity association model with data source, database, table, field, and task as the core. Bind the entity hierarchy relationship through foreign key indexes to adapt to the platform's layered retrieval logic.

[0020] S4. Standardized preprocessing of all types of task scripts: Connects to the middle platform scheduling platform API to batch collect offline SQL, stored procedures, Python / Shell scripts, synchronous configurations, and real-time streaming tasks; removes comments, blank lines, and dynamic variables line by line; splits multi-statement composite scripts into independent execution units according to execution keywords; classifies tasks into three categories—SQL, code, and configuration—based on script suffixes and syntax features; and establishes a unique binding relationship between task IDs and database table entities.

[0021] S5, Dual-engine Layered Refined Lineage Analysis: Standardized SQL scripts are processed using AST hard parsing, traversing the syntax tree's Query, From, Join, Select, and Operator nodes to extract table-level, field-level, and operator-level dependencies layer by layer; Non-standard tasks are processed using a fine-tuned lineage model soft parsing, inputting script content and database table structure through fixed template prompts, and the model outputs structured JSON lineage data, marking results with confidence levels below 75% as requiring further review; The outputs of the two engines are merged to generate a multi-granularity initial lineage map.

[0022] S6. Deduplication and Structured Normalization of Lineage Graph: A depth-first search (DFS) algorithm is used to traverse lineage links, identify and remove closed-loop cyclic dependencies; duplicate lineages are deduplicated using task ID, upstream entity ID, and downstream entity ID as unique verification keys; lineage relationships are converted into attributed node-edge structures, with nodes storing metadata entity information and edges storing operator, task, time, and confidence attributes, thus completing graph normalization.

[0023] S7. Data layering and change linkage iteration: Metadata is stored in a relational database, lineage graph is stored in a graph database, and change logs are stored in a time-series database; the system automatically retains the metadata of the most recent 30 versions and the corresponding lineage snapshots; it listens for metadata addition, deletion, renaming, and attribute change events, and automatically invalidates the historical lineage after the event is triggered, schedules the corresponding task to re-parse, and completes the dynamic iteration update of the lineage.

[0024] Through the above methods, this invention provides an automatic metadata collection and lineage analysis method for data middleware. It establishes a pluggable plugin architecture by standardizing the data source interaction interface, pre-setting standardized data source plugins for multiple categories. New heterogeneous data sources can be integrated into the middleware simply by adapting a dedicated driver file, without modifying the main middleware program code, thus improving the ease of data source expansion. It collaboratively operates three differentiated collection modes, coupled with dual verification logic of timestamps and file fingerprints, allowing for flexible matching of collection strategies based on business change frequency, covering metadata collection needs across all scenarios, from initial full archiving to periodic incremental updates and second-level change awareness. It adapts to two types of processing tasks in the middleware by building a dual-engine parsing architecture, using syntax tree node traversal rules to complete standardized script parsing and fixed prompt word rules and confidence verification mechanisms to complete semantic parsing of non-standard tasks, fully covering multi-level data lineage extraction. Combining graph traversal algorithms and dedicated verification keys, it completes lineage link regularization, using change hooks to perceive metadata changes in real time, automatically iterating and updating the lineage graph, and employing a version snapshot retention mechanism to form a closed-loop governance process from data source access to lineage iteration updates.

[0025] Example 1 This invention provides a method for automatic metadata collection and lineage resolution for data middleware platforms. In S1, a pluggable plugin pool has five types of plugins built-in: relational database, data warehouse computing engine, file data source, message queue, and non-relational storage. All plugins have a unified interface for input parameters, output parameters, and exception return formats. When adding a new data source, only the adaptation driver package needs to be uploaded, without modifying the main code of the middleware platform.

[0026] Example 2 This invention provides an automatic metadata collection and lineage resolution method for data middleware. In S2, the full snapshot differential collection first traverses all database table fields to generate a unique MD5 snapshot fingerprint; the timed incremental collection is executed according to a custom Cron cycle, updating only data with different fingerprints; CDC collection listens to Binlog / WAL / cluster logs, captures DDL, field, and partition change events, delivers them through a message queue, and completes the second-level metadata update after parsing by the consumer.

[0027] Example 3 This invention provides an automatic metadata collection and lineage parsing method for data middleware. In S5, the node traversal logic of AST hard parsing is as follows: identify From / Join nodes to determine the upstream data source, identify Insert / Overwrite nodes to determine the downstream landing entity, decompose Select fields, Case, aggregation operations, and grouping operators, match the mapping relationship between upstream and downstream fields, and generate a three-level structured lineage.

[0028] The large model's soft parsing prompts are fixed to include three parts: the original task script, the current data source table field structure, and the middle platform level labels. The model output fields are fixed to include upstream ID, downstream ID, dependent fields, operator type, and parsing confidence. Low-confidence results are manually reviewed before being stored in the database.

[0029] Example 4 This invention provides an automatic metadata collection and lineage resolution method for data middleware. In S6, circular dependency detection recursively traverses all lineage edges using DFS. If duplicate nodes appear in the traversal path, it is determined to be a closed-loop invalid link, which is directly removed and the operation log is recorded.

[0030] Example 5 This invention provides an automatic metadata collection and lineage parsing method for data middleware. In S7, the event-driven linkage mechanism is as follows: the system registers metadata change hooks, and after detecting entity changes, it queries all bound task IDs through reverse association index, invalidates old lineages in batches, asynchronously triggers re-parsing tasks, and the new lineage overwrites and replaces the existing data.

[0031] Example 6 This invention provides a method for automatic metadata collection and lineage resolution for data platforms. The specific execution logic of each step is as follows: S1. Data source plugin registration and standardized verification To enable convenient, no-development access to data sources, the plugin pool uniformly encapsulates three basic interaction interfaces: connection verification, metadata retrieval, and exception callback. It also standardizes the input parameters, output parameters, and exception return formats for all plugins. The connection verification interface's input parameters are fixed, including the data source address, port, username, and password. Its output parameters include connectivity status, error code, and error description. The metadata retrieval interface's input parameters include the collection scope and filtering conditions. Its output is a standardized list of metadata entities. All plugins are developed in accordance with this contract.

[0032] When adding a new data source, technical personnel package the adaptation driver according to the interface specifications. The system first performs format verification, version compatibility verification, and signature validity verification on the driver package. Packages that do not meet the specifications are directly rejected and specific error items are returned. After verification, the package is uploaded to the plugin pool to complete registration and adaptation. No changes to the main backend program code are required throughout the process. After plugin registration, a smoke test is automatically performed, calling the metadata capture interface to execute a lightweight activity probe query to verify whether the returned data structure conforms to the unified output specifications. The timeout threshold for a single interface call is set to 5 seconds; a timeout is considered a call failure. Three consecutive successful interface calls mark the plugin as available; three consecutive failures mark the plugin as unavailable, prohibiting subsequent data collection tasks from calling it. Specific error codes and error messages are returned uniformly in case of abnormal situations.

[0033] After plugin registration, basic parameters such as data source connection address, access account, port, and access permissions are entered. The system then performs link and permission verification step by step. First, the connectivity status of internal and external network ports is verified through the TCP handshake mechanism. After confirming that the link is unobstructed, the system verifies the account's permissions for table lookup, field reading, partition viewing, and runtime log reading operations item by item. Special attention is paid to verifying read permissions for the system dictionary table and the information_schema metadata table, as well as listening permissions for the Binlog log. If permissions are insufficient, the system returns the specific missing permission items, facilitating quick authorization by operations and maintenance personnel. After successful permission verification, the system automatically labels the ODS source layer, DWD detail layer, DWS summary layer, DM application layer, and external business system layer based on the data flow hierarchy. The label determination is automatically matched by default based on the data inflow direction, while also supporting manual correction. This completes the standardized classification of data source layers, providing a hierarchical basis for subsequent metadata modeling.

[0034] S2, Automated Collection of Multimodal Collaborative Metadata In line with the pace of business changes in the middle platform, the three types of data collection modes operate in parallel to adapt to the full lifecycle of metadata collection. Among them, full snapshot differential collection is mostly used for the first access of a brand-new data source. The collection adopts a scheduling strategy of parallel collection of multiple databases and serial collection of a single database. The maximum number of concurrent connections for a single data source is limited to 5 to avoid high-frequency queries overwhelming the source database. The system traverses the entire domain of resources, including databases, data tables, fields, indexes, and partitions, to fully collect metadata of three types: structure, business, and operation. After collection, the system concatenates the strings in a fixed order of "database name + table name + field name + field data type + field comment" and calculates the MD5 value as a unique fingerprint of the single table's metadata to ensure the consistency of incremental comparisons. At the same time, a baseline snapshot file is generated and stored in the version repository as a reference file for subsequent incremental data comparisons.

[0035] For routine business iterations, the system enables scheduled incremental comparison and data collection, with operations personnel able to preset Cron scheduling cycles. During collection, the system combines the last_modify_time update time of the data table with the snapshot MD5 fingerprint to dually determine the validity of data changes. The collection window is uniformly delayed by 2 minutes to avoid inconsistencies in incremental data caused by uncommitted source database transactions. The maximum time span for a single incremental collection is no more than 7 days. If the scheduled task is interrupted for more than 7 days, it automatically downgrades to full snapshot collection to avoid excessive pressure on the source database due to long-cycle incremental scans. Fingerprint differences are compared field by field to identify four types of changes: additions, deletions, type modifications, and comment modifications. Incremental collection and updates are only performed on changed tables and fields; unchanged metadata directly reuses existing data, effectively reducing cluster collection IO and computing power consumption.

[0036] For business scenarios involving high-frequency DDL structure changes, a real-time log collection link (CDC) can be enabled to simultaneously monitor database Binlog transaction logs, WAL write-ahead logs, and Flink cluster runtime logs. Captured events are first deduplicated, discarding duplicate DDL events within the same transaction, then sorted by transaction ID + timestamp and encapsulated into standardized change messages before being pushed to the Kafka message queue. Messages are partitioned and delivered by data source ID, with all change events from the same data source entering the same partition, ensuring events are consumed in the order they were generated in the logs and avoiding out-of-order updates that could lead to metadata version inconsistencies. The consumer persistently records the consumption offset to ZooKeeper, automatically resuming the offset upon service restart to prevent duplicate or missed consumption. The backend consumer continuously monitors the queue messages, parses the change content, and completes metadata updates within seconds.

[0037] To ensure data collection stability, the global data collection task relies on cluster sharding and parallel scheduling, splitting the task into multiple independent subtasks. If a single subtask encounters an error, it automatically retryes three times. If the retry exhaustion fails to resolve the issue, an error log is retained and an operational alert is pushed. If a single data source fails to collect data 10 times consecutively, a circuit breaker mechanism is automatically triggered, suspending the data collection task for that data source until manual investigation is performed, thus preventing unnecessary retries from consuming cluster resources. The abnormal subtasks run independently and in isolation, ensuring they do not disrupt the overall global data collection process.

[0038] S3, Metadata Cleaning and Normalization, and Mid-Platform Modeling The raw metadata collected from multiple channels is disorganized and contains a lot of redundant data, requiring unified standardization before modeling and use. The system first encodes all metadata in UTF-8 format, filtering invalid null values, illegal special characters, and garbled content. Newline characters and tabs in field comments are replaced with spaces to prevent formatting errors after data entry. Invalid entities are automatically identified; table names starting with tmp_, test_, or temp_, or table comments containing "test" or "temporary," are automatically removed as temporary invalid data. A unique primary key is used, consisting of the database name, table name, and field name. Entity data collected repeatedly from multiple channels is merged, and under the same primary key, the latest collected metadata overwrites the old value. Historical change records are synchronously written to the change log for retention, ensuring metadata purity.

[0039] After cleaning and streamlining, a relational model is built for five types of entities: data sources, databases, tables, fields, and tasks. The primary keys of these entities use the Snowflake algorithm to generate globally unique IDs. The Snowflake algorithm was chosen primarily because it avoids the need for a centralized ID generator, unlike database auto-incrementing primary keys, thus preventing primary key conflicts across multiple data sources. Table entities include attributes such as table name, Chinese comments, storage engine, creation time, estimated data volume, and hierarchical tags. Field entities include attributes such as field name, type, length, comments, whether it is a primary key, and whether it is a partitioned field. The hierarchical relationships between entities are bound using database foreign key indexes. The hierarchical relationships between entities are as follows: one data source corresponds to multiple databases, one database corresponds to multiple tables, and one table corresponds to multiple fields—all one-to-many relationships. Tasks and tables have a many-to-many relationship, with mapping relationships stored through an intermediate association table to avoid data redundancy. This model is compatible with the layered architecture of the middle platform and can complete unified structured modeling of heterogeneous metadata, facilitating subsequent middle platform asset retrieval and upstream / downstream hierarchical traceability business calls.

[0040] S4, Standardized Preprocessing of Scripts for All Task Types The system connects to the middleware scheduling platform via an open API interface to batch aggregate offline SQL, stored procedures, Python scripts, Shell scripts, DataX synchronization configurations, and Flink real-time streaming full-category processing task scripts within the platform. By default, only tasks with successful execution records within the last 7 days are retrieved; tasks that are offline or paused for more than 30 days are not included in the parsing scope, reducing unnecessary computing power consumption. Operations personnel can manually specify full aggregation as needed.

[0041] After aggregation, standardized preprocessing is carried out in steps: Script comments and redundant blank lines / spaces are removed in batches; dynamic business placeholder variables in the format ${variable name} are matched using regular expressions and uniformly replaced with static constant placeholders. Placeholders are not involved in field dependency determination when parsing lineage. Multi-level nested compound scripts are split based on semicolons, begin, and end execution keywords. Keywords within string constants are skipped during splitting to avoid mis-splitting valid statements, and the scripts are broken down into the smallest independent execution units. After splitting, the ANTLR parser is used to perform syntax pre-validation on each sub-unit. Syntax errors are recorded and marked for skipping parsing. Scripts that fail pre-validation three times consecutively are automatically added to the manual review list and will not be parsed again to avoid consuming scheduling resources. Tasks are divided into three categories—SQL, code, and configuration—based on script suffixes and syntax features. Finally, task IDs are bound to standardized database table entities in the middle platform. The binding uses a combination of fuzzy matching of database table names and manual calibration. Scripts with a matching degree below 80% are marked for manual confirmation, establishing a one-to-one business relationship.

[0042] S5, Dual-Engine Layered and Refined Bloodline Analysis By combining the differences in task writing methods in the middle platform, the two engines work together to complete the full lineage extraction, respectively adapting to standardized routine processing tasks and custom non-standard processing tasks, complementing each other to achieve complete parsing of the entire data lineage.

[0043] For grammatically compliant and formatted SQL scripts, a dedicated parser for the corresponding computing engine is invoked to generate an abstract syntax tree. The syntax tree's functional nodes are traversed level by level to determine data flow relationships. From and Join nodes are used to lock upstream dependent database table entities, while Insert, Overwrite, and Create nodes are used to determine downstream data storage entities. Table aliases and field aliases are recursively parsed and mapped to actual physical fields to avoid broken lineages caused by aliases. Subqueries and nested view nodes are recursively traversed to summarize field dependencies layer by layer from bottom to top. Select query fields, Case conditions, aggregation operations, and grouping operator nodes are further decomposed to accurately match upstream and downstream field mapping logic, generating a three-level structured lineage data at the table, field, and operator levels to ensure the integrity of field-level lineages in multi-level nested SQL statements. For UNION, UNION ALL, and multi-table JOIN nodes, upstream mappings are expanded one-to-one according to field positions to ensure no loss of field-level lineage. For SELECT* statements, all fields of the target table are automatically matched to generate the corresponding lineage.

[0044] For tasks with complex syntax structures, such as multi-level nested SQL, custom UDF functions, compiled code scripts, and non-standard synchronous configurations, where the AST cannot be directly decomposed and parsed, this invention sets up a large-scale model semantic soft parsing link as a supplement. This link supports access to a lineage parsing large model adapted to the business scenario. When called, the complete task script, the corresponding database table field structure list, and the middle platform level tags are integrated into a fixed-format prompt word input model. Negative constraint rules are added to the prompt words, forcing the output content to be in standard JSON format, prohibiting the addition of explanatory text, and ensuring that the fields strictly follow the agreed schema specifications.

[0045] The model output first undergoes a format validity check. If the format fails, the generation parameters are automatically adjusted and the process is retried once. If it still fails, it is marked as pending manual processing. Results that pass the validation include core data such as upstream and downstream entity IDs, related fields, operational operators, and parsing confidence scores. The system presets a 75% confidence score review threshold. Parsing results below the threshold are marked as pending manual review, while results that meet the threshold are directly stored and used in the database.

[0046] When integrating the lineage data output from both engines, if both AST and large model parsing results exist for the same task, the AST parsing result takes precedence, with the large model result serving as a supplement. Conflicts between the two are marked as items requiring review, and the final path is manually confirmed. Scripts that throw syntax errors or fail to generate a valid syntax tree during AST parsing automatically downgrade to the large model soft parsing path without manual switching. Finally, temporary test paths and duplicate paths from the same source are removed, and an initial lineage map with full coverage and multi-granular linkage is generated, facilitating subsequent unified cleaning and standardization.

[0047] S6. Bloodline map verification, deduplication, and structured normalization The initial lineage graph contains issues such as closed-loop dependencies, homologous duplicates, and invalid dangling links, requiring verification and normalization before storage. The system recursively traverses all lineage nodes using a Depth-First Search (DFS) algorithm, employing a non-recursive stack structure to avoid stack overflow due to excessive recursion depth. During traversal, a globally visited set of nodes is maintained to reduce duplicate path traversals and improve detection efficiency. The detection process sets a maximum traversal depth of 50 levels; exceeding this threshold triggers a long-link alarm and terminates the current branch traversal, preventing resource exhaustion in extreme scenarios. If the same entity node is repeatedly matched along the traversal path, it is identified as a closed-loop invalid dependency link, and the corresponding lineage edge is automatically removed while the operation log is retained for future reference.

[0048] To eliminate duplicate lineages, a unique verification key is generated by combining the task ID, upstream entity ID, and downstream entity ID. The verification key content of all links is compared. If the verification keys overlap but the operator attributes differ, the operator lists are merged, and only one edge is retained; historical parsing results are not directly discarded. If the verification keys completely overlap, it is determined to be a duplicate lineage generated by the same task. Only the link with the highest confidence and the most recent parsing time is retained, and the remaining redundant edges are logically deleted to reduce invalid data storage. An additional feature is added to determine dangling lineages: if the upstream and downstream entity IDs have no matching records in the metadata database, they are temporarily stored in the abnormal lineage database instead of being directly deleted. These are periodically reviewed and processed manually. The abnormal lineage database is automatically scanned weekly. Entities with complete matching records are automatically added back to the database. Entities without matching entities after 90 days are archived and deleted to control storage costs.

[0049] After the link cleaning is completed, it is uniformly converted into a standard attribute graph structure. The nodes carry the basic attributes of the metadata entity, and the links are bound with the task number, operator type, parsing time, and parsing confidence as auxiliary parameters. A global index is established for the node ID and hierarchical label, and a secondary index is established for the task ID and operator type of the edge to improve the efficiency of graph database traceability query and adapt to the business logic of graph database storage and fast traceability retrieval.

[0050] S7. Data tiered entry and change-linked iterative processing Based on the differentiated storage data requirements of the middle platform business, a hierarchical storage architecture was built to divide the storage carriers. Standardized structured metadata is stored in a MySQL relational database to support asset catalog queries and basic metadata retrieval; normalized lineage graphs are stored in a Neo4j graph database, with tag indexes built according to data layers to improve cross-layer tracing query efficiency and adapt to full-link tracing and change impact analysis; all metadata change events are stored in an InfluxDB time-series database to completely retain the data iteration trajectory, facilitating later problem tracing and review.

[0051] The system incorporates a metadata version control mechanism. Each update to the metadata structure and attributes synchronously generates a new version number, which is then linked to the corresponding version lineage graph. Each version generation automatically performs a consistency check, comparing the number of metadata entities with the number of lineage nodes; any discrepancies are flagged as unusable. The platform automatically retains the most recent 30 sets of historical version data. If multiple changes occur within the same day, only the last version is retained as a snapshot of that day, preventing rapid expansion of the number of versions due to frequent daily changes. Versions older than 30 days are automatically archived to cold storage. Operations personnel can retrieve historical data and perform version rollback operations as needed. Version rollback uses a linked rollback rule; when metadata versions are switched, the corresponding lineage graph is automatically rolled back synchronously, ensuring data consistency between the two.

[0052] To achieve synchronized iteration of metadata and lineage, the platform incorporates a dedicated change hook function that monitors data table additions / deletions, field renamings, field attribute modifications, and data hierarchy adjustments around the clock. Change events are categorized by their impact: table deletions and field deletions are considered high-risk events, immediately triggering re-parsing; field comment modifications and tag changes are considered low-risk events, aggregated and processed in batches hourly to prevent high-frequency, low-risk changes from consuming excessive resources. Upon detecting a valid change, a reverse lookup is performed using the edge attribute index of the graph database. Entering a single entity ID allows for the retrieval of all related upstream and downstream task links within seconds, eliminating the need for a full scan. Batch invalidation is then performed on the historical lineage links bound to that entity. Re-parsing tasks are added to an asynchronous task queue and scheduled for execution based on priority. Tasks that fail after three consecutive retries are moved to a dead-letter queue and an alert is pushed, preventing a surge of changes from overwhelming system resources. After generating a new compliant lineage graph, existing invalid links are automatically replaced, requiring no manual intervention throughout the entire process.

[0053] The system encapsulates standard RESTful interfaces, with core functionalities including metadata query, end-to-end lineage tracing, change impact analysis, and asset volume statistics. It supports paginated queries and conditional filtering for use by upper-layer applications such as data mapping and compliance auditing.

Claims

1. A method for automatic metadata collection and lineage resolution for data middleware platforms, characterized in that, Includes the following steps: Step 1: Verify and categorize the access data sources through the pluggable plugin pool; Step 2: Perform multimodal collaborative acquisition of metadata on the data source from Step 1 in parallel; Step 3: Clean the metadata from Step 2 and build an entity association model; Step 4: Standardize the preprocessing scripts for all types of tasks and bind them to the entities from Step 3; Step 5: Use a dual-engine approach of AST hard parsing and large-model soft parsing to process the script in Step 4 to generate multi-granularity initial lineage maps; Step 6: Remove closed loop edges, remove duplicates, and normalize the graph from Step 5. Step 7: Store the metadata from Step 3 and the graph from Step 6 in the database in layers, listen for change events, asynchronously re-parse and overwrite the old lineage.

2. The method for automatic metadata collection and lineage resolution for a data platform as described in claim 1, characterized in that, Step 1 specifically includes the following steps: Step 1.1: Define a unified abstract collection interface and build a pluggable data source plugin pool. The plugin has three basic interfaces: connection verification interface, metadata capture interface, and exception callback interface. Step 1.2: When adding a new data source, encapsulate the adaptation driver package according to the interface specification, and perform format verification, version compatibility verification and signature validity verification on the driver package. After the verification is passed, upload it to the plugin pool to complete the registration and adaptation. Step 1.3: After the plugin registration is completed, perform a smoke test by calling the metadata capture interface to perform a lightweight liveness detection query and verify whether the returned data structure conforms to the unified output specification. Step 1.4: Enter the data source connection address, access account, port and access permissions, and complete the port connectivity verification, account permission verification and database table read permission verification in sequence; Step 1.5: After the permission verification is passed, the data source is marked with five categories based on the data flow hierarchy: ODS source layer, DWD detail layer, DWS summary layer, DM application layer, and external business system layer, thus completing the standardized classification.

3. The method for automatic metadata collection and lineage resolution for a data platform as described in claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Connect the standardized and categorized data sources from Step 1 into the data collection process, and configure three types of collection modes to be executed in parallel: first-time full snapshot differential collection, periodic incremental comparison collection, and real-time monitoring and collection of CDC logs. Step 2.2: The data collection task adopts a distributed execution with cluster sharding. The full snapshot differential collection first traverses all database table fields, generates a unique MD5 snapshot fingerprint, and generates a baseline snapshot file and stores it in the version repository. The timed incremental comparison collection is executed according to a custom Cron cycle. It combines the data table update time and the snapshot MD5 fingerprint to determine the validity of data changes and only updates data with different fingerprints. The CDC log real-time monitoring collection listens to the database Binlog transaction log, WAL write-ahead log and Flink cluster operation log, captures DDL change events, field change events and partition change events, delivers them through the message queue, and completes the metadata update after parsing by the consumer.

4. The method for automatic metadata collection and lineage resolution for a data platform as described in claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: Perform encoding standardization, null value filtering, and special character removal on the metadata collected in Step 2; Step 3.2: Merge entity data collected repeatedly from multiple channels using the database name, table name, and field name as unique primary keys; Step 3.3: Construct a five-element entity association model including data source, library, table, field, and task. The primary key of the entity is generated with a globally unique ID using the snowflake algorithm. The hierarchical relationship of each entity is bound by the database foreign key index: one data source corresponds to multiple libraries, one library corresponds to multiple tables, one table corresponds to multiple fields, and the relationship between tasks and tables is many-to-many, with the mapping relationship stored through an intermediate association table.

5. The method for automatic metadata collection and lineage resolution for a data platform as described in claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4.1: Connect to the API interface of the middle platform scheduling platform to batch collect offline SQL scripts, stored procedures, Python scripts, Shell scripts, synchronous configurations and real-time streaming tasks; Step 4.2: Remove comments, blank lines, and dynamic variables from the collected scripts line by line, and split multi-statement compound scripts into independent execution units according to execution keywords; Step 4.3: Divide the tasks into three categories based on script suffixes and syntax features: SQL tasks, code tasks, and configuration tasks. Step 4.4: Use the fuzzy matching of database table names plus manual calibration mode to bind the task ID with the standardized database table entity generated in step 3 to establish a one-to-one business relationship.

6. The method for automatic metadata collection and lineage resolution for a data platform as described in claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5.1: Use AST hard parsing to process the standardized SQL script output in step 4, call the dedicated parser of the corresponding computing engine to generate an abstract syntax tree, traverse the Query node, From node, Join node, Select node and Operator node of the syntax tree, extract table-level, field-level and operator-level dependencies layer by layer, and generate three-level structured lineage data of table-level, field-level and operator-level layer by layer. Step 5.2: Use a fine-tuned bloodline model with soft parsing to process non-standard tasks. Input script content and database table structure through fixed template prompts, and the model outputs structured JSON bloodline data. Step 5.3: Integrate the output results of the dual engines of AST hard analysis and fine-tuned bloodline large model soft analysis to generate a multi-granularity initial bloodline map.

7. The method for automatic metadata collection and lineage resolution for a data platform as described in claim 1, characterized in that, Step 6 specifically includes the following steps: Step 6.1: Use the depth-first traversal algorithm to traverse the lineage links of the multi-granularity initial lineage graph output in Step 5, and identify and remove closed-loop cyclic dependency edges. Step 6.2: Use the task ID, upstream entity ID, and downstream entity ID as unique verification keys to complete the deduplication of duplicate lineages; Step 6.3: Convert the bloodline relationship into a node-edge structure with attributes. The nodes store metadata entity information, and the edges store operator, task, time and confidence attributes to complete the graph normalization.

8. The method for automatic metadata collection and lineage resolution for a data platform as described in claim 1, characterized in that, Step 7 specifically includes the following steps: Step 7.1: Store the standardized metadata output in Step 3 into a relational database, store the normalized kinship graph output in Step 6 into a graph database, and store the change log into a time series database. Step 7.2: Retain metadata and corresponding lineage snapshots from the most recent versions; Step 7.3: Listen for metadata addition events, deletion events, renaming events, and attribute change events. After an event is triggered, the historical lineage is invalidated, and the corresponding task is scheduled to re-parse the data to complete the dynamic iteration update of the lineage.

Images