A method for facilitating data lineage collection and analysis

Abstract

The present invention relates to a method for facilitating data lineage collection and analysis. The method obtains original lineage data including task execution logs, application metadata and cross-system dependencies, and performs standardization processing on the original lineage data to generate structured lineage information including asset unique identifiers, upstream and downstream association relationships and data operation types. The structured lineage information is synchronously written into a graph database and a distributed data warehouse, wherein the graph database stores real-time association topology and the distributed data warehouse stores full historical versions. A transaction consistency algorithm is used to ensure the atomicity of double-write operations. Single-asset level lineage tracking is performed based on the real-time topology of the graph database and global lineage analysis is performed based on the batch computing capability of the distributed data warehouse. Direct dependency paths, deep association networks and closed-loop link detection results are generated, forming a closed-loop management mechanism from data collection and analysis to optimization, overcoming the problem of disconnection between analysis results and collection ends in traditional solutions.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to a method for facilitating data lineage collection and analysis. Background Art

[0002] Data lineage management is a core component of data governance. It primarily establishes relationships between data assets by tracing the complete data chain from source to consumer. Traditional solutions typically use graph databases to store lineage topologies. While these solutions can intuitively display relationships, they have significant limitations in practical applications: a single storage architecture struggles to balance real-time query and historical analysis needs, and graph database performance bottlenecks can impact the efficiency of complex analytical scenarios. Cross-system data integration suffers from poor collection adaptability and incomplete metadata synchronization. Message processing anomalies can easily lead to data inconsistencies, and insufficient batch operation performance can hinder the efficiency of large-scale data processing. These solutions struggle to meet the comprehensive requirements of real-time, integrity, and traceability in enterprise-level data governance. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method for facilitating data lineage collection and analysis, thereby achieving comprehensive management of data lineage collection.

[0004] To achieve the above objectives, the present invention provides a method for facilitating data lineage collection and analysis, comprising:

[0005] Obtaining original lineage data, including task execution logs, application metadata, and cross-system dependencies;

[0006] Standardize the original lineage data to generate structured lineage information. Standardization includes data deduplication, entity identifier generation, and relationship topology verification. Structured lineage information includes asset unique identifiers, upstream and downstream relationships, and data operation types.

[0007] Synchronously write structured lineage information into a graph database and a distributed data warehouse. The graph database stores the real-time association topology, while the distributed data warehouse stores the full historical version. A transaction consistency algorithm is used to ensure the atomicity of dual-write operations.

[0008] Perform single-asset lineage tracking based on the real-time topology of the graph database, extract direct dependency paths and deep association networks, and perform global lineage analysis based on the batch computing capabilities of the distributed data warehouse to generate closed-loop link detection results.

[0009] Generate and output a lineage health assessment result based on the direct dependency path, deep association network, and closed-loop link detection results. The lineage health assessment result includes at least one of data fault warning, circular dependency identification, and hot asset tagging.

[0010] Furthermore, the results of the bloodline health assessment are associated with the feedback from the collection end of the original bloodline data through a dynamic message routing algorithm.

[0011] Furthermore, the original bloodline data is standardized to generate structured bloodline information, including:

[0012] Deduplication is performed on the original lineage data to eliminate duplicated task execution logs and redundant dependencies, obtaining deduplicated data entities and extracting data operation types from the task execution logs.

[0013] Generate a globally unique entity identifier for the deduplicated data entity, which is recorded as the asset unique identifier;

[0014] And, bind the asset unique identifier with the data asset type and the system level to which it belongs, to obtain a dependency relationship with the entity identifier;

[0015] Perform relationship topology verification on dependency relationships with entity identifiers, identify and eliminate upstream and downstream relationships with logical conflicts, and obtain verified upstream and downstream relationships;

[0016] The output includes structured lineage information including the asset's unique identifier, verified upstream and downstream relationships, and data operation types.

[0017] Furthermore, the original lineage data is deduplicated to eliminate duplicated task execution logs and redundant dependencies, obtaining deduplicated data entities. The data operation types are then extracted from the task execution logs, including:

[0018] Identify duplicated task execution logs by comparing hash values ​​and retain the task execution log with the latest timestamp.

[0019] Detect redundant dependencies based on a composite key of source system identifiers, table names, and field names;

[0020] Use regular expressions to match operation keywords in task execution logs to extract data operation types such as add, delete, modify, and query.

[0021] Generate a globally unique entity identifier for the deduplicated data entity, recorded as the asset unique identifier, including:

[0022] The entity identifier is generated using a three-part coding rule. The entity identifier includes a system code, an asset type code, and a serial number.

[0023] Ensure the uniqueness of entity identifiers through a distributed unique ID generator to obtain asset unique identifiers;

[0024] Bind the asset unique identifier with the data asset type and the system level to which it belongs to obtain the dependency relationship with the entity identifier, including:

[0025] Establish a data asset type classification tree, including three levels of database, table, and field;

[0026] Determine the system level based on the system topology diagram, bind the three-level relationship between data center, business domain, and subsystem, and obtain the dependency relationship with entity identifiers;

[0027] Perform relationship topology verification on dependency relationships with entity identifiers, identify and eliminate logically conflicting upstream and downstream relationships, and obtain verified upstream and downstream relationships, including:

[0028] Use a graph traversal algorithm to detect closed-loop dependency edges of upstream and downstream relationships, and disconnect the last formed dependency edge when a circular reference is found;

[0029] The metadata registry verifies the dangling nodes in the dependency relationships with entity identifiers, removes the associations pointing to non-existent assets, and obtains verified upstream and downstream association relationships.

[0030] Furthermore, the structured lineage information is synchronously written into the graph database and distributed data warehouse, including:

[0031] Write the asset's unique identifier as a vertex attribute in the graph database, with the vertex type corresponding to the data asset type;

[0032] Convert the verified upstream and downstream relationships into directed edges in the graph database. The attributes of the directed edges include the data operation type.

[0033] Furthermore, the structured lineage information is classified by data asset type, and fact tables and dimension tables are constructed in the distributed data warehouse. The primary key of the fact table is mapped to the unique identifier of the asset.

[0034] Create a bidirectional index table between the vertex ID of the graph database and the primary key of the fact table. The bidirectional index table includes system-level fields.

[0035] Furthermore, the atomicity of the double-write operation is ensured through a transaction consistency algorithm, including:

[0036] Record the vertex and edge information to be written in the graph database's write-ahead log. At the same time, cache the lineage information to be converted in a temporary table in the distributed data warehouse.

[0037] When the bidirectional index table is successfully updated, the vertex and edge write operation of the graph database and the insert operation into the data warehouse fact table are executed in parallel;

[0038] When a storage system write fails, the executed operations are rolled back according to the write-ahead log, and the residual data is cleared through the bidirectional index table;

[0039] Locate the failed record through the system-level fields and re-initiate synchronous writing of the system-level fields for the failed record.

[0040] Furthermore, we conduct single-asset lineage tracing based on the real-time topology of the graph database, extracting direct dependency paths and deep association networks, including:

[0041] Using the asset's unique identifier as the starting node, a single-hop traversal algorithm is used to extract the direct dependency path in the graph database. The direct dependency path consists of a single edge connecting the starting node and its directly adjacent nodes, and records the data asset type, data operation type, and system boundary jump markers between adjacent nodes.

[0042] Perform multi-order neighborhood expansion analysis based on the end nodes of the directly dependent paths, including:

[0043] By recursively traversing the in-degree edges in the graph database, an upstream dependency subnet is constructed. By recursively traversing the out-degree edges in the graph database, a downstream influence subnet is constructed. The upstream dependency subnet and the downstream influence subnet are merged to form a deep association network. Both the upstream dependency subnet and the downstream influence subnet contain node connections of at least two hops.

[0044] The method also includes:

[0045] Assign a basic weight value to each edge in the direct dependency path. The basic weight value is calculated based on the conversion cost of the corresponding data operation type, the system jump cost, and the source node data freshness.

[0046] Calculate the composite weight value for the multi-hop path in the deep association network. The composite weight value is the weighted accumulation of the basic weight values ​​of all individual edges on the multi-hop path;

[0047] The analysis results are rendered through a visualization engine. Direct dependency paths are displayed using solid edges, and multi-hop paths in deep association networks are displayed using gradient dashed edges.

[0048] Furthermore, a global lineage analysis is performed based on the batch computing capabilities of the distributed data warehouse to generate closed-loop link detection results, including:

[0049] Perform batch calculations on structured lineage information based on preset scanning strategies to obtain basic lineage elements, including table-level dependencies, job scheduling sequences, and cross-system transmission records.

[0050] Perform topological traversal on table-level dependencies to extract closed-loop dependency features, including circular reference paths and cross-system bidirectional data flows. Furthermore, perform time-efficiency analysis on job scheduling sequences to obtain link time-efficiency features.

[0051] Based on closed-loop dependency characteristics and link timeliness characteristics, an abnormal closed-loop judgment model is established. The abnormal closed-loop judgment model includes multiple judgment dimensions, which are configured as data update timeliness anomaly dimension, scheduling cycle conflict dimension, and resource contention deadlock dimension.

[0052] Build a full-link lineage map of core data assets and a sampling dependency chain of common data assets based on basic lineage elements;

[0053] The closed-loop dependency features, link time characteristics, abnormal closed-loop judgment model and full-link lineage map are associated and stored to generate closed-loop link detection results. The closed-loop link detection results include the topological structure characteristics of circular reference closed loops, the scheduling interval distribution of time-sensitive abnormal closed loops, the competitive resource characteristics of resource deadlock closed loops, and the severity level assessment parameters of various types of closed loops.

[0054] Furthermore, based on the direct dependency path, deep association network and closed-loop link detection results, the bloodline health assessment results are generated and output, including:

[0055] Extract path health parameters based on direct dependency paths. Path health parameters include dependency depth, cross-system hop count, and key node density.

[0056] Perform topological analysis on deep correlation networks to obtain network health characteristics, including node in-degree, edge weight dispersion, and subnetwork coupling;

[0057] Combined with the closed-loop link detection results, the closed-loop health indicators are analyzed to generate a lineage health assessment result including health score, repair priority, hotspot distribution and impact range.

[0058] Furthermore, the dynamic message routing algorithm is used to associate the bloodline health assessment results with the feedback from the original bloodline data collection end, including:

[0059] Build a feedback priority queue based on the results of the lineage health assessment. The feedback priority queue includes emergency repair items, optimization suggestions, and routine monitoring items.

[0060] Dynamically adjust message routing based on the load status of the data source system, prioritizing the routing of urgent repair items to the corresponding collection terminal;

[0061] Establish a mapping relationship between the bloodline health assessment results and the collection rules of the original bloodline data. When a bloodline abnormality is detected, the sampling frequency adjustment of the corresponding collection end is automatically triggered;

[0062] Compress bloodline health assessment result packets to ensure that high-priority feedback information is transmitted first in low-bandwidth environments;

[0063] During the feedback process, the receiving status of each collection end is monitored in real time, and an automatic retransmission link is started for the bloodline health assessment results that are not successfully received.

[0064] Furthermore, the original lineage data is obtained. The original lineage data includes task execution logs, application metadata, and cross-system dependencies, including:

[0065] Use distributed probes to collect task execution logs from each business system and extract task IDs, execution times, and upstream and downstream dependencies.

[0066] Extract application metadata from the database, including table structure change records, field-level data flow, and ETL processing logic;

[0067] Build dependency graphs based on cross-system call link analysis to identify data flow paths and interface call frequencies between systems;

[0068] Generate raw bloodline data.

[0069] Different from existing technologies, the above technical solution effectively solves the acquisition adaptability problem caused by data heterogeneity in traditional solutions by obtaining raw lineage data including task execution logs, application metadata and cross-system dependencies, and standardizing it to generate structured lineage information including asset unique identifiers, upstream and downstream associations and data operation types. By synchronously writing structured lineage information into a graph database and a distributed data warehouse, where the graph database stores the real-time association topology and the distributed data warehouse stores the full historical version, and adopting a transaction consistency algorithm to ensure the atomicity of double-write operations, it not only retains the real-time query advantage of the graph database, but also breaks through the performance bottleneck through the distributed data warehouse, avoiding the atomicity guarantee problem of metadata changes in traditional solutions. Based on the real-time topology of the graph database for single-asset level lineage tracking and the batch computing capability of the distributed data warehouse for global lineage analysis, direct dependency paths, deep association networks and closed-loop link detection results are generated simultaneously, significantly improving batch operation performance and high concurrency processing capabilities. Through the dynamic message routing algorithm, the bloodline health assessment results including data fault warning, circular dependency identification and hot asset tagging are fed back to the collection end, forming a closed-loop management mechanism from data collection, analysis to optimization, overcoming the problem of disconnection between analysis results and collection end in traditional solutions.

[0070] The above-mentioned records related to the content of the invention are only an overview of the technical solution of this application. In order to enable ordinary technicians in this field to understand the technical solution of this application more clearly, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purposes and other purposes, features and advantages of this application easier to understand, the following is an explanation in combination with the specific implementation methods and drawings of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The drawings are only used to illustrate the principles, implementation methods, applications, features and effects of the specific embodiments of the present invention and other related contents, and are not to be considered as limiting the present application.

[0072] In the drawings of the specification:

[0073] Figure 1 This is a schematic diagram of steps S101 to S105 described in the specific implementation method;

[0074] Figure 2 This is a schematic diagram of steps S201 to S204 described in the specific implementation method;

[0075] Figure 3 This is a schematic diagram of steps S301 to S303 described in the specific implementation method;

[0076] Figure 4 This is a schematic diagram of steps S401 to S404 described in the specific implementation method;

[0077] Figure 5 This is a schematic diagram of the architecture described in the specific implementation method. DETAILED DESCRIPTION

[0078] In order to explain in detail the possible application scenarios, technical principles, specific solutions that can be implemented, and the purpose and effects of this application, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this application and are therefore only examples and are not intended to limit the scope of protection of this application.

[0079] See also Figure 1 This embodiment provides a method for facilitating data lineage collection and analysis, including:

[0080] S101. Obtaining original lineage data, including task execution logs, application metadata, and cross-system dependencies;

[0081] S102. Standardize the original lineage data to generate structured lineage information. The standardization process includes data deduplication, entity identifier generation, and relationship topology verification. The structured lineage information includes asset unique identifiers, upstream and downstream association relationships, and data operation types.

[0082] S103. Synchronously write the structured lineage information into the graph database and the distributed data warehouse. The graph database stores the real-time association topology, and the distributed data warehouse stores the full historical version. The atomicity of the dual-write operation is ensured through a transaction consistency algorithm.

[0083] S104: Perform single-asset lineage tracking based on the real-time topology of the graph database, extract direct dependency paths and deep association networks, and perform global lineage analysis based on the batch computing capabilities of the distributed data warehouse to generate closed-loop link detection results.

[0084] S105. Generate and output the bloodline health assessment results based on the direct dependency path, deep association network and closed-loop link detection results. The bloodline health assessment results include at least one of data fault warning, circular dependency identification and hot asset tagging; and associate the bloodline health assessment results with the collection end feedback of the original bloodline data through a dynamic message routing algorithm.

[0085] In step S101, obtaining original lineage data refers to collecting original data including task execution logs, application metadata, and cross-system dependencies from the business system. Task execution logs record the running status and dependencies of data processing tasks, application metadata includes data table structures and ETL conversion rules, and cross-system dependencies reflect the data flow paths between different systems. By collecting log data in real time through distributed probes, combining metadata extraction tools to obtain structured definitions, and parsing system call links to build initial dependencies, unified access to multi-source heterogeneous data is achieved, providing a complete data foundation for subsequent analysis.

[0086] In step S102, standardizing the raw lineage data involves performing data deduplication, entity identifier generation, and relationship topology verification on the collected raw data, ultimately generating structured lineage information containing unique asset identifiers, upstream and downstream relationships, and data operation types. Data deduplication eliminates duplicated log records, entity identifier generation ensures that each data asset has a globally unique identifier, and relationship topology verification verifies the logical consistency of dependencies. By transforming heterogeneous data into a unified model through a standardized rule engine, this resolves analytical errors caused by data format differences and provides standardized input for dual storage writes.

[0087] In step S103, synchronously writing structured lineage information to the graph database and distributed data warehouse involves simultaneously writing the data to the graph database's real-time relational topology and the distributed data warehouse's full historical version through a transactional consistency algorithm. The graph database stores the most up-to-date lineage network, supporting low-latency queries; the distributed data warehouse stores historical versions of data by time partition. A two-phase commit protocol ensures dual-write atomicity, avoiding the risk of data inconsistency caused by asynchronous writes. The scalability of distributed storage also addresses graph database performance issues in high-concurrency scenarios.

[0088] In step S104, single-asset lineage tracing based on the real-time topology of the graph database involves extracting the direct dependency paths and deep association networks for a specific asset. Simultaneously, leveraging the batch computing capabilities of the distributed data warehouse, global lineage analysis is performed and closed-loop link detection results are generated. Direct dependency paths reflect the first-order upstream and downstream relationships of assets. Deep association networks use graph traversal algorithms to uncover implicit cross-level dependencies. Closed-loop link detection identifies anomalous patterns such as circular references. Combining the advantages of real-time querying and offline computing, this approach not only meets the needs of immediate analysis but also handles large-scale historical data association analysis.

[0089] In step S105, generating a lineage health assessment result involves outputting analysis conclusions based on direct dependency paths, deep association networks, and closed-loop link detection results, including data fault warnings, circular dependency identification, and hot asset markings. This analysis is then fed back to the original lineage data collection end via a dynamic message routing algorithm. Data fault warnings identify link interruption risks, while hot asset markings identify data entities that are frequently accessed or changed. Establishing a closed-loop control mechanism from analysis to collection enables the system to automatically optimize data collection strategies, resolving the disconnect between lineage analysis results and data sources.

[0090] This embodiment achieves unified modeling of cross-system data by acquiring multi-source heterogeneous original lineage data and standardizing it into structured information; synchronously maintains the graph database and distributed data warehouse through a transaction consistency double-write mechanism, taking into account both real-time query performance and historical data analysis needs; based on the dual analysis capabilities of real-time topology and batch computing, it significantly improves the depth and breadth of lineage tracking; and finally forms a closed-loop management through a dynamic feedback mechanism, effectively solving the defects of traditional solutions in data consistency, processing efficiency and system linkage, and providing full-link, traceable and self-optimizing management capabilities for data governance.

[0091] See also Figure 2 In a further embodiment, the original blood relationship data is standardized to generate structured blood relationship information, including:

[0092] S201. Deduplication of original lineage data is performed to eliminate duplicated task execution logs and redundant dependencies, obtaining deduplicated data entities, and extracting data operation types from the task execution logs;

[0093] S202: Generate a globally unique entity identifier for the deduplicated data entity, recording it as the asset unique identifier; and bind the asset unique identifier to the data asset type and the system level to which it belongs, to obtain a dependency relationship with the entity identifier;

[0094] S203: Perform relationship topology verification on dependency relationships with entity identifiers, identify and eliminate logically conflicting upstream and downstream relationships, and obtain verified upstream and downstream relationships;

[0095] S204. Output structured lineage information including the asset unique identifier, verified upstream and downstream association relationships, and data operation types.

[0096] In step S201, deduplication of the original lineage data refers to the detection and cleaning of duplicate data in the collected task execution logs and cross-system dependencies. Specifically, by comparing the log timestamps, task IDs, and data path characteristics, duplicate records caused by repeated collection or redundant reporting are eliminated, and data operation types including data reading, writing, and conversion are parsed from the task execution logs. This operation effectively solves the problem of noise interference in the original data, ensuring that each data entity processed subsequently has a clear business meaning and operational context, generating accurate lineage relationships.

[0097] In step S202, generating a unique asset identifier for the deduplicated data entity involves using a hash algorithm to combine the data asset type (e.g., database table, API interface, file, etc.) and the system level (e.g., business system, data warehouse, data lake, etc.) to generate a globally unique identifier. The unique asset identifier is strongly bound to the physical location and business attributes of the data asset, forming a dependency relationship with the entity identifier. This unified identification system addresses the discrepancies in naming conventions across systems, allowing cross-system data assets to be uniquely identified and accurately linked, resolving the issue of broken ancestry caused by confusing identifiers.

[0098] In step S203, performing a relational topology check on dependency relationships with entity identifiers refers to detecting logical conflicts in upstream and downstream relationships through a directed acyclic graph (DAG) verification algorithm, including circular dependencies, invalid references, and type mismatches. Relationships that violate data flow rules (such as reports that are reversely dependent on the original data table) are automatically eliminated, and only verified upstream and downstream relationships are retained. By replacing traditional manual review methods with an automated verification mechanism, the deviation in bloodline analysis caused by incorrect associations is greatly reduced, and the credibility of data relationships is improved.

[0099] In step S204, outputting structured lineage information encapsulates the fully processed asset unique identifier, verified upstream and downstream relationships, and data operation types into a unified output format. The asset unique identifier serves as the unique index for the data entity, the upstream and downstream relationships form the edges of the lineage network, and the data operation type identifies the specific method used to establish the relationship (e.g., SQL import, API call, etc.). The structured data generated in this step provides standardized input for subsequent dual storage writes. The comprehensive metadata description capability enables lineage analysis down to the granularity of specific operations, overcoming the problem of monotonous relationship types.

[0100] This implementation ensures the purity of raw lineage data through data deduplication, achieves precise matching of cross-system assets through globally unique identifiers, and eliminates logical anomalies through topological validation, ultimately outputting highly complete structured lineage information. This significantly improves the accuracy and consistency of lineage data, enabling subsequent storage and analysis to be conducted based on trustworthy data. It addresses flaws such as traceability distortion and missing associations caused by data quality issues, and provides standardized data preprocessing capabilities for end-to-end lineage management.

[0101] In a further embodiment, the original lineage data is deduplicated to eliminate duplicated task execution logs and redundant dependencies, obtaining deduplicated data entities, and extracting data operation types from the task execution logs, including:

[0102] Identify duplicated task execution logs by comparing hash values ​​and retain the task execution log with the latest timestamp.

[0103] Detect redundant dependencies based on a composite key of source system identifiers, table names, and field names;

[0104] Use regular expressions to match operation keywords in task execution logs to extract data operation types such as add, delete, modify, and query.

[0105] Generate a globally unique entity identifier for the deduplicated data entity, recorded as the asset unique identifier, including:

[0106] The entity identifier is generated using a three-part coding rule. The entity identifier includes a system code, an asset type code, and a serial number.

[0107] Ensure the uniqueness of entity identifiers through a distributed unique ID generator to obtain asset unique identifiers;

[0108] Bind the asset unique identifier with the data asset type and the system level to which it belongs to obtain the dependency relationship with the entity identifier, including:

[0109] Establish a data asset type classification tree, including three levels of database, table, and field;

[0110] Determine the system level based on the system topology diagram, bind the three-level relationship between data center, business domain, and subsystem, and obtain the dependency relationship with entity identifiers;

[0111] Perform relationship topology verification on dependency relationships with entity identifiers, identify and eliminate logically conflicting upstream and downstream relationships, and obtain verified upstream and downstream relationships, including:

[0112] Use a graph traversal algorithm to detect closed-loop dependency edges of upstream and downstream relationships, and disconnect the last formed dependency edge when a circular reference is found;

[0113] The metadata registry verifies the dangling nodes in the dependency relationships with entity identifiers, removes the associations pointing to non-existent assets, and obtains verified upstream and downstream association relationships.

[0114] In this embodiment, identifying repeatedly collected task execution logs through hash value comparison refers to performing MD5 hash calculation on the complete content of the task execution log, and determining it as a duplicate log when the same hash value is detected, and retaining the record with the latest timestamp as valid data. Detecting redundant dependencies based on the combination key of the source system identifier, table name, and field name refers to splicing these three elements into a composite key, and determining it as a redundant dependency when an identical combination key is detected. Using regular expressions to match operation keywords in task execution logs is to extract standardized operation types from the log text through predefined regular patterns (such as keywords such as SELECT, UPDATE, etc.), ensuring that the original lineage data has completed noise filtering before entering the processing flow.

[0115] In this embodiment, the three-segment encoding rule used to generate entity identifiers involves concatenating the system code (e.g., ERP system code 01), the asset type code (e.g., table type code 02), and a distributed serial number to form the identifier body. Ensuring the uniqueness of entity identifiers through a distributed unique ID generator involves invoking the Snowflake algorithm service to generate serial number segments, thus avoiding ID conflicts in a clustered environment. The established unique asset identifier system possesses clear semantic characteristics, capable of both identifying asset ownership and distinguishing asset categories, providing a stable anchor for cross-system lineage tracking.

[0116] In this embodiment, establishing a data asset type classification tree involves constructing a tree structure consisting of databases (root nodes), tables (intermediate nodes), and fields (leaf nodes), with each node associated with a type code. Determining the system hierarchy based on the system topology diagram involves associating assets along a hierarchical path from data center (e.g., DC01) to business domain (e.g., finance) to subsystem (e.g., reimbursement system) according to predefined hierarchical mapping rules. This hierarchical binding imbues kinship relationships with spatial dimensions, supporting multidimensional analysis based on business architecture.

[0117] In this embodiment, the use of a graph traversal algorithm to detect closed-loop dependency edges involves performing a depth-first search on the dependency graph. When a node's ancestor is found to point to itself, the resulting edge is automatically severed. Verifying dangling nodes through the metadata registry involves querying the central metadata repository to verify the existence of the target asset in the dependency relationship and eliminating invalid edges pointing to unregistered assets. This automated verification mechanism eliminates logical errors in the lineage graph and ensures the authenticity and validity of dependency relationships.

[0118] This implementation eliminates data redundancy through hash deduplication and composite key detection, constructs a resolvable unique asset identifier system using three-segment encoding, implements multi-dimensional asset location based on classification trees and system topology, and ensures the logical correctness of dependencies through graph algorithms and metadata validation. This achieves efficient purification and structured conversion of raw lineage data, enabling subsequent lineage analysis to be conducted based on standardized, reliable data, effectively resolving traceability distortions and missing associations caused by data quality issues.

[0119] See also Figure 3 In a further embodiment, the structured lineage information is synchronously written into the graph database and the distributed data warehouse, including:

[0120] S301. Write the asset unique identifier as a vertex attribute in the graph database, where the vertex type corresponds to the data asset type.

[0121] S302: Convert the verified upstream and downstream relationships into directed edges in a graph database, where the attributes of the directed edges include data operation types; and classify the structured lineage information by data asset type, constructing fact tables and dimension tables in a distributed data warehouse, with the primary key of the fact table maintaining a mapping relationship with the asset's unique identifier.

[0122] S303: Create a bidirectional index table of the vertex ID of the graph database and the primary key of the fact table, where the bidirectional index table includes system-level fields.

[0123] In step S301, writing the asset's unique identifier as a vertex attribute in the graph database involves storing the globally unique identifier generated during the normalization phase along with the corresponding data asset type (e.g., table, view, file, etc.) as a vertex in the graph database. Vertex types strictly map to the technical classification of the data asset. For example, database table vertices contain schema name and table name attributes, while API interface vertices contain service path and request method attributes. The graph database's schema constraints ensure the integrity and type safety of vertex attributes, enabling subsequent graph traversal queries to be optimized based on asset type, addressing the performance issues of relational databases in modeling complex network relationships.

[0124] In step S302, converting verified upstream and downstream relationships into directed edges in the graph database involves modeling the standardized, verified data flow relationships as directed edges connecting vertices, and embedding specific data operation types (such as SELECT, INSERT, PROCEDURE, etc.) in the edge attributes. Simultaneously, data is organized in a star schema within the distributed data warehouse, with fact tables (recording instances of lineage relationships) and dimension tables (describing asset attributes) constructed using unique asset identifiers as a link. This ensures a one-to-one correspondence between network relationships in the graph database and relational records in the data warehouse. This achieves a dual representation of relational data, retaining the intuitive relevance of the graph structure and the large-scale analytical capabilities of a data warehouse, overcoming the functional limitations of a single storage solution in specific scenarios.

[0125] In step S303, establishing a bidirectional index table involves creating a bridge table that maintains the mapping between graph database vertex IDs and data warehouse fact table primary keys, thereby recording additional system-level information (e.g., ODS, DWD, and APP layers). This index table is implemented using a distributed key-value store, supporting millisecond-level bidirectional query responses. The unified cross-storage addressing mechanism established in this step allows subsequent analysis tasks to freely switch query engines without repetitive data identification, eliminating the data association costs incurred by storage isolation.

[0126] This implementation precisely models data asset entities through graph database vertices, fully expresses operational semantics through directed edges, and leverages the star schema of a distributed data warehouse to achieve analysis-friendly data organization. The resulting bidirectional indexing system streamlines data associations between different storage engines. This simultaneously meets the low-latency graph query requirements for real-time lineage tracking and the batch computing power required for large-scale historical analysis, resolving the performance and scalability challenges inherent in traditional single-storage architectures.

[0127] See also Figure 4 In a further embodiment, the atomicity of the double write operation is ensured by a transaction consistency algorithm, including:

[0128] S401. Record the vertex and edge information to be written in the graph database write-ahead log. At the same time, cache the lineage information to be converted in a temporary table of the distributed data warehouse.

[0129] S402: After the bidirectional index table is successfully updated, the vertex and edge writing operation of the graph database and the insert operation into the data warehouse fact table are executed in parallel;

[0130] S403: When a storage system write fails, the executed operations are rolled back according to the write-ahead log, and residual data is cleared through the bidirectional index table;

[0131] S404: Locate the failed record through the system-level field, and re-initiate synchronous writing of the system-level field for the failed record.

[0132] In step S401, recording pending vertex and edge information in the graph database's write-ahead log means persisting all pending graph data changes (including vertex attribute updates and edge relationship adjustments) in the form of transaction logs to the write-ahead log (WAL) before actual data writing. Simultaneously, a temporary table is created in the distributed data warehouse to temporarily store the ETL-converted lineage information. This temporary table retains the complete fields of the original structured lineage information and is marked as pending. This ensures that even in the event of a system failure, unfinished write operations can be recovered through the log and temporary table, mitigating the risk of data loss in asynchronous dual-write solutions.

[0133] In step S402, after the bidirectional index table is successfully updated, a parallel dual write operation is performed. This means that only after the bidirectional index table (a bridge table that records the mapping between graph database vertex IDs and data warehouse primary keys) has completed the insertion of new records, will the graph database's graph structure update and the data warehouse's fact table insertion be triggered simultaneously. The graph database operation uses a batch API to submit the vertex and edge changes all at once, while the data warehouse writes to the fact and dimension tables through a distributed transaction interface. Using the bidirectional index table as a synchronization point ensures that the two storage systems maintain logically consistent write timing, avoiding temporary data inconsistencies caused by differences in write timing.

[0134] In step S403, when a storage system write fails, a rollback operation is performed. This involves restoring the graph database state by reversing the operations recorded in the write-ahead log (e.g., deleting written vertices, undoing edge relationships), while simultaneously cleaning up the corresponding uncommitted data in the data warehouse's temporary tables. Residual partially written records are located using a bidirectional index table, and a background cleanup process ensures data consistency across all storage systems. This approach is particularly effective in scenarios such as network partitions or node failures in distributed environments, eliminating dirty data more thoroughly than a simple retry strategy, ensuring the atomicity of cross-system transactions.

[0135] In step S404, locating and retrying failed records through system-level fields refers to using system-level fields (such as hierarchical identifiers such as ODS / DWD / APP) to accurately screen and classify the lineage information that failed to be written. Automatically identify failed records at a specific level, reconstruct the write request based on the original information in the pre-write log, and prioritize retrying data at the critical business level. Through a hierarchical and graded processing mechanism, recovery efficiency is optimized while ensuring eventual consistency, significantly reducing system load compared to a full retry, and is particularly suitable for large-scale lineage data synchronization scenarios.

[0136] This implementation establishes a reliable transaction preparation phase through write-ahead logs and temporary tables, uses a bidirectional index table as the synchronization coordination point for cross-system writes, and utilizes a comprehensive exception rollback and tiered retry mechanism to achieve highly consistent data synchronization between the graph database and the distributed data warehouse. While ensuring write performance, this solution thoroughly resolves the challenge of transaction atomicity across heterogeneous storage systems in a distributed environment, ensuring the accuracy and reliability of lineage data in end-to-end analysis and providing a solid underlying storage guarantee for data governance.

[0137] In a further embodiment, single-asset level lineage tracing is performed based on the real-time topology of the graph database to extract direct dependency paths and deep association networks, including:

[0138] Using the asset's unique identifier as the starting node, a single-hop traversal algorithm is used to extract the direct dependency path in the graph database. The direct dependency path consists of a single edge connecting the starting node and its directly adjacent nodes, and records the data asset type, data operation type, and system boundary jump markers between adjacent nodes.

[0139] Perform multi-order neighborhood expansion analysis based on the end nodes of the directly dependent paths, including:

[0140] By recursively traversing the in-degree edges in the graph database, an upstream dependency subnet is constructed. By recursively traversing the out-degree edges in the graph database, a downstream influence subnet is constructed. The upstream dependency subnet and the downstream influence subnet are merged to form a deep association network. Both the upstream dependency subnet and the downstream influence subnet contain node connections of at least two hops.

[0141] The method also includes:

[0142] Assign a basic weight value to each edge in the direct dependency path. The basic weight value is calculated based on the conversion cost of the corresponding data operation type, the system jump cost, and the source node data freshness.

[0143] Calculate the composite weight value for the multi-hop path in the deep association network. The composite weight value is the weighted accumulation of the basic weight values ​​of all individual edges on the multi-hop path;

[0144] The analysis results are rendered through a visualization engine. Direct dependency paths are displayed using solid edges, and multi-hop paths in deep association networks are displayed using gradient dashed edges.

[0145] In this embodiment, executing a single-hop traversal algorithm with the asset unique identifier as the starting node means that after locating the target asset vertex from the graph database, the adjacent nodes are scanned along the directly connected in- and out-edges to generate a straight-line path connected by a single edge. The recorded data asset type distinguishes different entities such as tables and fields, the data operation type is marked with specific operations such as SELECT and JOIN, and the system boundary jump marker identifies cross-system dependencies. Constructing upstream dependency subnets and downstream impact subnets through recursive traversal means starting from the end node of the direct dependency path, continuously backtracking along the in-degree edge to construct the upstream dependency chain, extending along the out-degree edge to construct the downstream impact chain, and finally merging to form a network structure containing multi-level nodes. The recursive process sets a depth threshold to prevent unlimited expansion, and retains at least two-hop relationships to ensure the value of network analysis.

[0146] Preferably, in this embodiment, assigning a base weight value to a single edge refers to comprehensively calculating the edge weight based on the data operation type conversion cost (e.g., the JOIN operation cost is higher than the SELECT operation cost), the system jump cost (cross-data center communication cost), and the source node data freshness (data update time). The composite weight value of the multi-hop path adopts exponential decay weighted accumulation to highlight the influence of proximal dependencies. Visual rendering uses solid edges to highlight the key connections of the direct dependency path and gradient dashed edges to display the secondary paths of the deeply associated network. The line width is positively correlated with the path weight. Color coding distinguishes different system boundaries, and node shape represents asset type. This presentation method enables intuitive understanding of complex bloodline networks and assists in quickly locating core data flows.

[0147] This implementation precisely captures direct dependencies through single-hop traversal, constructs a global association network using recursive expansion, quantifies path weights based on multi-dimensional indicators, and integrates visualization technology to present a three-dimensional representation of lineage topology. This enables penetrating analysis from the microscopic level of individual assets to the macroscopic level of data networks, ensuring data lineage tracking with both atomic-level accuracy and system-level integrity, providing a multi-granular analysis tool for assessing the impact of data changes and locating the root causes of faults.

[0148] In a further embodiment, a global lineage analysis is performed based on the batch computing capability of the distributed data warehouse to generate closed-loop link detection results, including:

[0149] Perform batch calculations on structured lineage information based on preset scanning strategies to obtain basic lineage elements, including table-level dependencies, job scheduling sequences, and cross-system transmission records.

[0150] Perform topological traversal on table-level dependencies to extract closed-loop dependency features, including circular reference paths and cross-system bidirectional data flows. Furthermore, perform time-efficiency analysis on job scheduling sequences to obtain link time-efficiency features.

[0151] Based on closed-loop dependency characteristics and link timeliness characteristics, an abnormal closed-loop judgment model is established. The abnormal closed-loop judgment model includes multiple judgment dimensions, which are configured as data update timeliness anomaly dimension, scheduling cycle conflict dimension, and resource contention deadlock dimension.

[0152] Build a full-link lineage map of core data assets and a sampling dependency chain of common data assets based on basic lineage elements;

[0153] The closed-loop dependency features, link time characteristics, abnormal closed-loop judgment model and full-link lineage map are associated and stored to generate closed-loop link detection results. The closed-loop link detection results include the topological structure characteristics of circular reference closed loops, the scheduling interval distribution of time-sensitive abnormal closed loops, the competitive resource characteristics of resource deadlock closed loops, and the severity level assessment parameters of various types of closed loops.

[0154] In this embodiment, performing batch computations based on a preset scanning strategy involves using a distributed parallel computing framework to fully scan lineage information, extracting table-level dependencies (e.g., the ETL relationship between Table A and Table B), job scheduling sequences (the order in which jobs are executed), and cross-system transmission records (e.g., Kafka message queue transmission). Topology traversal to extract closed-loop dependency features involves using graph algorithms to detect circular structures in table-level dependencies, identifying circular reference paths (e.g., Table A → Table B → Table C → Table A) and cross-system bidirectional flows (data transmission between System X and System Y). Job scheduling timeliness analysis calculates the deviation between the actual execution time of a job and the planned interval to obtain link timeliness characteristics (e.g., latency rate). Establishing an abnormal closed-loop determination model involves configuring three core dimensions: data update timeliness anomalies (updates to upstream tables are later than updates to downstream tables), scheduling cycle conflicts (mismatched job scheduling frequencies), and resource contention deadlocks (multiple jobs competing for the same resource). A quantitative threshold is set for each dimension, and an abnormality flag is triggered when the characteristic value exceeds the threshold.

[0155] In this embodiment, building a full-link lineage graph involves performing a complete link restoration of core assets (such as key report data). For common assets, random sampling is used to generate representative dependency chains. Graph storage includes node attributes (table structure information) and edge attributes (dependency type and timeliness). Associative storage of closed-loop detection results involves persisting the identified closed-loop topology (such as a ring path node sequence), timeliness anomaly distribution (delay duration statistics), resource contention characteristics (a list of locked contention resources), and severity levels (based on impact scoping) to the analysis results repository.

[0156] This embodiment achieves full lineage scanning through distributed batch computing, combines graph algorithms with time-sensitive analysis to identify closed-loop features, establishes a multi-dimensional anomaly determination model, and constructs a differentiated lineage graph storage system. It automatically detects key issues such as circular references, time-sensitive conflicts, and resource deadlocks from massive amounts of lineage data, providing a closed-loop governance basis for data architecture optimization and effectively preventing data quality incidents caused by dependency anomalies. Closed-loop severity assessment parameters support problem prioritization and improve data governance efficiency.

[0157] In a further embodiment, based on the direct dependency path, deep association network and closed-loop link detection results, a bloodline health assessment result is generated and output, including:

[0158] Extract path health parameters based on direct dependency paths. Path health parameters include dependency depth, cross-system hop count, and key node density.

[0159] Perform topological analysis on deep correlation networks to obtain network health characteristics, including node in-degree, edge weight dispersion, and subnetwork coupling;

[0160] Combined with the closed-loop link detection results, the closed-loop health indicators are analyzed to generate a lineage health assessment result including health score, repair priority, hotspot distribution and impact range.

[0161] In this embodiment, extracting path health parameters based on direct dependency paths refers to calculating the hierarchical depth of the dependency link (such as the maximum number of nested layers), the number of cross-system boundary times (such as the number of jumps between different data centers), and the proportion of key nodes (such as the frequency of appearance of core tables in the path). This quantitative indicator reflects the complexity and stability of a single data flow. The higher the depth value and the more cross-system hops, the greater the health risk. Topological analysis of a deep correlation network refers to counting the in-degree (number of dependencies) of each node to measure the importance of the asset, calculating the variance of the edge weight to evaluate the balance of dependency strength, and analyzing the connection density between subnets to determine the degree of modularity. Network health characteristics reveal the overall quality of the data architecture. High-in-degree nodes, weight-unbalanced edges, and strongly coupled subnets are all potential architectural problems.

[0162] In this embodiment, parsing the closed-loop health indicator refers to mapping the severity level in the closed-loop link detection results into a health score (0-100 points), setting priorities based on the scope of impact (number of systems involved) and repair costs (difficulty of transformation), and identifying hot spots (such as frequent time-sensitive anomalies in a certain business domain) through closed-loop type clustering. This embodiment generates a lineage health report that includes quantitative scores, processing priorities, problem distribution, and potential impacts through a three-dimensional evaluation of path parameters, network characteristics, and closed-loop indicators. This achieves standardized measurement and visual presentation of data lineage quality, helping to identify high-risk dependency paths, fragile network structures, and frequent closed-loop problems. The health assessment results can directly guide resource allocation, prioritize key issues with low scores, high priorities, and wide impacts, and improve the robustness of the overall data system.

[0163] In a further embodiment, the bloodline health assessment results are associated with the feedback from the collection end of the original bloodline data through a dynamic message routing algorithm, including:

[0164] Build a feedback priority queue based on the results of the lineage health assessment. The feedback priority queue includes emergency repair items, optimization suggestions, and routine monitoring items.

[0165] Dynamically adjust message routing based on the load status of the data source system, prioritizing the routing of urgent repair items to the corresponding collection terminal;

[0166] Establish a mapping relationship between the bloodline health assessment results and the collection rules of the original bloodline data. When a bloodline abnormality is detected, the sampling frequency adjustment of the corresponding collection end is automatically triggered;

[0167] Compress bloodline health assessment result packets to ensure that high-priority feedback information is transmitted first in low-bandwidth environments;

[0168] During the feedback process, the receiving status of each collection end is monitored in real time, and an automatic retransmission link is started for the bloodline health assessment results that are not successfully received.

[0169] In this embodiment, building a feedback priority queue refers to classifying the results of the lineage health assessment according to the degree of urgency: emergency repair items (such as serious closed-loop problems), optimization suggestion items (such as weight imbalanced edges) and routine monitoring items (such as ordinary dependency paths). The minimum heap data structure is used to implement dynamic sorting to ensure that high-priority items are always at the top of the queue to be processed. Dynamic adjustment of the message routing path refers to obtaining the CPU / memory load rate of each data source system through heartbeat detection. When the target system load exceeds the threshold, the emergency repair items are automatically routed to the backup access point or delayed transmission. The routing algorithm comprehensively evaluates network latency, system load and problem severity to achieve hierarchical scheduling of feedback information. Establishing a collection rule mapping relationship refers to configuring the linkage rules between health scores and sampling frequencies. For example, when the table-level dependency health score is lower than 60 points, the collection interval of the relevant lineage data of the table is shortened from 1 hour to 10 minutes.

[0170] In this embodiment, data packet compression adopts a columnar storage format, and numerical fields such as health score and priority are compressed using Delta encoding, and text fields are encoded using dictionary encoding. After compression, the data packet is tagged with a priority, and when the network bandwidth is insufficient, only the metadata of the emergency repair item is transmitted. Real-time monitoring of the receiving status means attaching a unique transaction ID to each feedback data packet, and the acquisition end replies with an ACK confirmation signal. If no confirmation is received within the timeout, the retransmission is backed off according to the priority index. The emergency repair item can be retried up to 5 times, and the regular monitoring item can be retried up to 2 times.

[0171] This embodiment achieves intelligent linkage between assessment results and data collection endpoints through dynamic message routing, establishing a closed-loop control system from problem detection to governance implementation. This enables precise allocation of lineage governance resources, real-time feedback on urgent issues triggers immediate action, high-frequency sampling ensures continuous tracking of abnormal data, and an adaptive transmission mechanism guarantees information reachability in diverse network environments. This significantly improves the timeliness and reliability of data lineage governance, forming a virtuous cycle of "assessment-feedback-optimization" and effectively reducing delays in repairing data link failures.

[0172] In a further embodiment, original lineage data is obtained, and the original lineage data includes task execution logs, application metadata, and cross-system dependencies, including:

[0173] Use distributed probes to collect task execution logs from each business system and extract task IDs, execution times, and upstream and downstream dependencies.

[0174] Extract application metadata from the database, including table structure change records, field-level data flow, and ETL processing logic;

[0175] Build dependency graphs based on cross-system call link analysis to identify data flow paths and interface call frequencies between systems;

[0176] Generate raw bloodline data.

[0177] In this embodiment, collecting task execution logs from various business systems through distributed probes involves deploying lightweight collection agents in production environments to capture the runtime logs generated by the scheduling engine in real time. The probes extract the task ID as a unique identifier, record execution timestamps accurate to the millisecond, and analyze upstream and downstream dependencies within the task configuration, such as input and output tables and message queues. An incremental synchronization mechanism is employed to transmit only newly added or modified log content, significantly reducing network transmission pressure while ensuring data integrity.

[0178] In this embodiment, extracting application metadata from the database involves connecting to the business system's metadata repository to automatically capture table structure change history and data processing logic. This focuses on capturing DDL operation records, such as field additions and type modifications, and parsing field-level mappings in stored procedures and ETL jobs, including column mapping rules and transformation conditions in SELECT-INSERT statements. For complex transformation logic, business processing features such as WHERE filter conditions and JOIN association rules are extracted.

[0179] In this embodiment, building a dependency graph based on cross-system call chain analysis involves integrating call logs from a distributed tracing system with traffic data from an API gateway. By analyzing the parameter paths of RPC calls, data input and output relationships between systems are identified. The frequency and throughput of interface calls are counted to quantify the strength of dependency relationships. Furthermore, the production and consumption records of the message middleware are analyzed to supplement the event-driven data flow path. Ultimately, a weighted directed graph structure is generated, where nodes represent system components and edge weights reflect the actual frequency of data interactions.

[0180] In this embodiment, generating raw lineage data refers to uniformly modeling the collected logs, metadata, and call link information. Entities such as task instances, data tables, and system interfaces, as well as their relationships, are converted into attribute graph models according to a preset standardized format, with each edge labeled with dependency type and time characteristics. Data consistency checks are performed before output to eliminate abnormal records with timestamp conflicts or missing references, and data version identifiers are automatically generated.

[0181] This embodiment achieves accurate capture and dynamic updating of full-link data lineage through the collaborative work of distributed log collection, deep metadata analysis, and cross-system link tracking. An incremental synchronization mechanism is used to ensure data timeliness, standardized modeling solves the integration problem of multi-source heterogeneous data, and versioning management supports traceability analysis of lineage changes. It significantly improves the granularity and accuracy of lineage data, lays a reliable data foundation for subsequent lineage health assessment, and minimizes the impact on production systems through lightweight collection design.

[0182] See also Figure 5For ease of understanding, the following examples are given to further understand the above content:

[0183] The existing technology has the following problems:

[0184] 1. Data is stored only in the graph database, and heterogeneous data storage is not supported. Daily usage scenarios are constrained by the performance of the graph database, which greatly limits the use cases.

[0185] 2. The message communication mechanism and processing logic are highly bound to the internal model, resulting in poor scalability and inability to quickly support the collection of third-party service lineage data.

[0186] 3. Atlas metadata changes (such as creating entities and updating relationships) are triggered asynchronously through the Hook mechanism. The atomicity of message consumption and metadata operations is not guaranteed (for example, if Kafka message processing fails, metadata written to Atlas must be manually rolled back). Compensation logic must be implemented using an external transaction coordinator (such as Apache Sky Walking or Seata).

[0187] 4. Insufficient batch operation performance. Frequent API calls may cause a surge in server-side pressure in high-concurrency scenarios.

[0188] 5. Atlas's metadata lifecycle management (such as expired data deletion and historical version tracing) relies on TTL (time-to-live) configuration or manual cleanup. It cannot automatically manage data versions and aging through a timeline mechanism like Hudi, which may lead to storage bloat or residual dirty data.

[0189] Furthermore, this example provides an architecture as follows:

[0190] The main framework is spingboot+kafka+neo4j+flinksql+hudi.

[0191] First, the collection end collects task and application lineage information through hooks, scheduled scripts, service push, and other methods, and pushes it to Kafka's common-topic message. The message contains message type, creator information, assets, and lineage relationship information.

[0192] The processing flow of the message type is:

[0193] Full coverage based on creator attributes;

[0194] Delete all content based on the creator's attributes;

[0195] New relationships are added based on the creator, and relationships that do not exist in the message are not deleted;

[0196] Secondly, the Springboot service consumes the common-topic, parses the lineage messages, and updates the neo4j data in batches. It converts the updated lineage information into asset and lineage relationship add / update / delete events, and sends them to the Kafka change data capture topic (cdc-topic).

[0197] Assets generate unique IDs based on the asset's qualified name;

[0198] The blood relationship generates a unique UUID (identification code) based on the upstream and downstream assets and the creator;

[0199] To ensure data consistency between Hudi and Neo4j, manual commit is used when consuming Kafka data. Consumption progress (commit offset) is recorded only when both Neo4j and CDC message sending are successful.

[0200] Through FlinkSQL, CDC topics are consumed in real time to flatten assets and relationships. Hudi records are updated using the unique UUIDs of assets and relationships, maintaining the accuracy of data in the data warehouse's Hudi tables and Neo4j.

[0201] Based on the data warehouse table, other computing engines such as Spark can be used to calculate the dependency information of all assets, such as all upstream and downstream assets. Figure 5 The archive table in can be stored as a partition per day.

[0202] This example has the following benefits:

[0203] 1. Data is written to both neo4j and hudi tables, improving system reliability through data redundancy and fault tolerance mechanisms.

[0204] 2. The newly added hudi table stores lineage data, which addresses the performance bottleneck problem of conventional graph databases that cannot query large amounts of data. It also solves the problem of being unable to query ultra-multi-layer lineage data and supports multi-node cross-service lineage queries, improving service availability and expanding usable scenarios.

[0205] 3. A unified message communication mechanism can quickly support the writing of new service lineage data, effectively improving the scalability of the system.

[0206] 4. By strongly binding message consumption with cross-service data operations, the atomicity of data writing is ensured, and automatic rollback is achieved in the event of failure, achieving consistency and reliable delivery of double data writing.

[0207] 5. Introducing the "one configuration, one lineage" mechanism, which simultaneously completes data insertion, update, and aging through a single Kafka message, greatly improving the efficiency of data changes and simplifying message content, while effectively reducing resource consumption.

[0208] 6. You can rely on neo4j to meet the upstream and downstream queries of a single asset, or perform more global lineage analysis based on the data warehouse hudi table.

[0209] Different from existing technologies, the above technical solution effectively solves the acquisition adaptability problem caused by data heterogeneity in traditional solutions by obtaining raw lineage data including task execution logs, application metadata and cross-system dependencies, and standardizing it to generate structured lineage information including asset unique identifiers, upstream and downstream associations and data operation types. By synchronously writing structured lineage information into a graph database and a distributed data warehouse, where the graph database stores the real-time association topology and the distributed data warehouse stores the full historical version, and adopting a transaction consistency algorithm to ensure the atomicity of double-write operations, it not only retains the real-time query advantage of the graph database, but also breaks through the performance bottleneck through the distributed data warehouse, avoiding the atomicity guarantee problem of metadata changes in traditional solutions. Based on the real-time topology of the graph database for single-asset level lineage tracking and the batch computing capability of the distributed data warehouse for global lineage analysis, direct dependency paths, deep association networks and closed-loop link detection results are generated simultaneously, significantly improving batch operation performance and high concurrency processing capabilities. Through the dynamic message routing algorithm, the bloodline health assessment results including data fault warning, circular dependency identification and hot asset tagging are fed back to the collection end, forming a closed-loop management mechanism from data collection, analysis to optimization, overcoming the problem of disconnection between analysis results and collection end in traditional solutions.

[0210] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concepts of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A method for facilitating data lineage collection and analysis, characterized in that: include: Obtaining original lineage data, including task execution logs, application metadata, and cross-system dependencies; Standardizing the original lineage data to generate structured lineage information, wherein the standardization includes data deduplication, entity identifier generation, and relationship topology verification. The structured lineage information includes a unique asset identifier, upstream and downstream association relationships, and data operation types. Synchronously write the structured lineage information into a graph database and a distributed data warehouse. The graph database stores the real-time association topology, and the distributed data warehouse stores the full historical version. The atomicity of the double-write operation is ensured through a transaction consistency algorithm. Perform single-asset lineage tracking based on the real-time topology of the graph database, extract direct dependency paths and deep association networks, and perform global lineage analysis based on the batch computing capabilities of the distributed data warehouse to generate closed-loop link detection results, including: Perform batch calculations on structured lineage information based on a preset scanning strategy to obtain basic lineage elements, including table-level dependencies, job scheduling sequences, and cross-system transmission records; Performing a topological traversal on the table-level dependency relationships to extract closed-loop dependency features, including circular reference paths and cross-system bidirectional data flows, and performing a time efficiency analysis on the job scheduling sequence to obtain link time efficiency features; Based on the closed-loop dependency characteristics and the link timeliness characteristics, an abnormal closed-loop determination model is established, wherein the abnormal closed-loop determination model includes multiple determination dimensions, which are configured as a data update timeliness abnormality dimension, a scheduling cycle conflict dimension, and a resource contention deadlock dimension; Construct a full-link lineage map of core data assets and a sampling dependency chain of common data assets based on the basic lineage elements; The closed-loop dependency features, link time characteristics, abnormal closed-loop judgment model, and full-link lineage map are associated and stored to generate the closed-loop link detection results, which include the topological structure characteristics of circular reference closed loops, the scheduling interval distribution of time-sensitive abnormal closed loops, the competitive resource characteristics of resource deadlock closed loops, and severity evaluation parameters for each type of closed loop; Generate and output a lineage health assessment result based on the direct dependency path, deep association network, and closed-loop link detection results, wherein the lineage health assessment result includes at least one of data fault warning, circular dependency identification, and hot asset marking; Furthermore, the bloodline health evaluation result is associated with the collection end feedback of the original bloodline data through a dynamic message routing algorithm.

2. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that: The original blood relationship data is standardized to generate structured blood relationship information, including: Deduplication is performed on the original lineage data to eliminate repeatedly collected task execution logs and redundant dependencies to obtain deduplicated data entities, and data operation types are extracted from the task execution logs; Generate a globally unique entity identifier for the deduplicated data entity, which is recorded as the asset unique identifier; and, binding the asset unique identifier with the data asset type and the system level to which it belongs, to obtain a dependency relationship with an entity identifier; Perform relationship topology verification on dependency relationships with entity identifiers, identify and eliminate upstream and downstream relationships with logical conflicts, and obtain verified upstream and downstream relationships; The output includes structured lineage information including the asset's unique identifier, verified upstream and downstream relationships, and data operation types.

3. The method for facilitating data lineage collection and analysis according to claim 2, characterized in that: Deduplication is performed on the original lineage data to eliminate duplicated task execution logs and redundant dependencies to obtain deduplicated data entities, and data operation types are extracted from the task execution logs, including: Identify duplicated task execution logs by comparing hash values ​​and retain the task execution log with the latest timestamp. Detect redundant dependencies based on a composite key of source system identifiers, table names, and field names; Use regular expressions to match operation keywords in task execution logs to extract data operation types such as add, delete, modify, and query. Generate a globally unique entity identifier for the deduplicated data entity, recorded as the asset unique identifier, including: The entity identifier is generated using a three-part coding rule. The entity identifier includes a system code, an asset type code, and a serial number. Ensure the uniqueness of the entity identifier through a distributed unique ID generator to obtain the asset unique identifier; Bind the asset unique identifier with the data asset type and the system level to which it belongs to obtain the dependency relationship with the entity identifier, including: Establish a data asset type classification tree, including three levels of database, table, and field; Determine the system level based on the system topology diagram, bind the three-level relationship between data center, business domain, and subsystem, and obtain the dependency relationship with entity identifiers; Perform relationship topology verification on dependency relationships with entity identifiers, identify and eliminate logically conflicting upstream and downstream relationships, and obtain verified upstream and downstream relationships, including: Use a graph traversal algorithm to detect closed-loop dependency edges of upstream and downstream relationships, and disconnect the last formed dependency edge when a circular reference is found; The metadata registry verifies the dangling nodes in the dependency relationships with entity identifiers, removes the associations pointing to non-existent assets, and obtains verified upstream and downstream association relationships.

4. The method for facilitating data lineage collection and analysis according to claim 2, characterized in that: Synchronously writing the structured lineage information into the graph database and the distributed data warehouse includes: Writing the asset unique identifier as a vertex attribute in the graph database, where the vertex type corresponds to the data asset type; Convert the verified upstream and downstream association relationships into directed edges in a graph database, where the attributes of the directed edges include the data operation type; Furthermore, the structured lineage information is classified according to data asset type, and a fact table and a dimension table are constructed in a distributed data warehouse, wherein the primary key of the fact table maintains a mapping relationship with the asset unique identifier; A bidirectional index table of vertex IDs of the graph database and primary keys of fact tables is established, wherein the bidirectional index table includes system-level fields.

5. The method for facilitating data lineage collection and analysis according to claim 4, characterized in that: The atomicity of double write operations is ensured through a transaction consistency algorithm, including: Record the vertex and edge information to be written in the graph database's write-ahead log. At the same time, cache the lineage information to be converted in a temporary table in the distributed data warehouse. When the bidirectional index table is successfully updated, the vertex edge writing operation of the graph database and the data warehouse fact table insertion operation are performed in parallel; When a storage system write fails, the executed operations are rolled back according to the write-ahead log, and residual data is cleared through the bidirectional index table; The failed record is located through the system-level field, and synchronous writing of the system-level field of the failed record is re-initiated.

6. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that: Perform single-asset lineage tracing based on the real-time topology of the graph database, extracting direct dependency paths and deep association networks, including: Using the asset unique identifier as the starting node, a single-hop traversal algorithm is used to extract a direct dependency path from the graph database. The direct dependency path is composed of a single edge connecting the starting node and its directly adjacent nodes, and records the data asset type, data operation type, and system boundary jump markers between adjacent nodes. Based on the end nodes of the directly dependent path, a multi-order neighborhood expansion analysis is performed, including: An upstream dependency subnet is constructed by recursively traversing the in-degree edges in the graph database, and a downstream influence subnet is constructed by recursively traversing the out-degree edges in the graph database. The upstream dependency subnet and the downstream influence subnet are merged to form a deep association network, wherein both the upstream dependency subnet and the downstream influence subnet contain a node connection relationship of at least two hops; The method further comprises: Assign a basic weight value to each edge in the direct dependency path. The basic weight value is calculated based on the conversion cost of the corresponding data operation type, the system jump cost, and the source node data freshness. Calculating a composite weight value for a multi-hop path in the deep association network, where the composite weight value is a weighted accumulation of the base weight values ​​of all individual edges on the multi-hop path; The analysis results are rendered through a visualization engine. Direct dependency paths are displayed using solid edges, and multi-hop paths in deep association networks are displayed using gradient dashed edges.

7. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that: Based on the direct dependency path, deep association network and closed-loop link detection results, a bloodline health assessment result is generated and output, including: Extracting path health parameters based on the direct dependency path, the path health parameters including dependency depth, cross-system hop count, and key node density; Performing topological analysis on the deep association network to obtain network health characteristics, where the network health characteristics include node in-degree, edge weight dispersion, and subnet coupling degree; The closed-loop health indicators are analyzed in combination with the closed-loop link detection results to generate a bloodline health assessment result including health score, repair priority, hotspot distribution and impact range.

8. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that: The bloodline health assessment result is associated with the feedback from the collection end of the original bloodline data through a dynamic message routing algorithm, including: Building a feedback priority queue based on the bloodline health assessment results, the feedback priority queue includes emergency repair items, optimization suggestion items, and routine monitoring items; Dynamically adjust the message routing path according to the load status of the data source system, and give priority to routing the emergency repair items to the corresponding collection terminal; Establish a mapping relationship between the bloodline health assessment results and the collection rules of the original bloodline data. When a bloodline abnormality is detected, the sampling frequency adjustment of the corresponding collection end is automatically triggered; Compress bloodline health assessment result packets to ensure that high-priority feedback information is transmitted first in low-bandwidth environments; During the feedback process, the receiving status of each collection end is monitored in real time, and an automatic retransmission link is started for the bloodline health assessment results that are not successfully received.

9. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that: Obtain original lineage data, which includes task execution logs, application metadata, and cross-system dependencies, including: Use distributed probes to collect task execution logs from each business system and extract task IDs, execution times, and upstream and downstream dependencies. Extract application metadata from the database, including table structure change records, field-level data flow, and ETL processing logic; Build dependency graphs based on cross-system call link analysis to identify data flow paths and interface call frequencies between systems; The original bloodline data is generated.

Images