Method convenient for data blood relationship collection and analysis

Abstract

The invention relates to a method convenient for data consanguinity collection and analysis, which comprises the following steps of: obtaining original consanguinity data comprising a task execution log, application metadata and a cross-system dependency relationship, and carrying out standardization processing on the original consanguinity data to generate structured consanguinity information comprising an asset unique identifier, an upstream and downstream association relationship and a data operation type; structured consanguinity information is synchronously written into a graph database and a distributed data warehouse, the graph database stores real-time association topology, the distributed data warehouse stores full-amount historical versions, and a transaction consistency algorithm is adopted to ensure the atomicity of double-write operation, so that the data storage efficiency is improved. Single-asset-level consanguinity tracking is performed based on real-time topology of a graph database, global consanguinity analysis is performed based on batch computing power of a distributed data warehouse, a direct dependence path, a deep association network and a closed-loop link detection result are generated, and a closed-loop management mechanism from data acquisition, analysis to optimization is formed. And the problem that an analysis result is disjointed from an acquisition end in a traditional scheme is solved.

Description

Technical Field

[0001] The present invention relates to the field of data management, and particularly to a method for facilitating data lineage collection and analysis. Background Art

[0002] Data lineage management is a core link in data governance, mainly by tracing the complete link of data from the source to the consumption end to establish the association relationship between data assets. Traditional solutions usually use graph databases to store lineage topologies. Although they can intuitively display the association relationship, there are obvious limitations in practical applications: a single storage architecture is difficult to balance the requirements of real-time query and historical analysis, and the performance bottleneck of graph databases will affect the response efficiency of complex analysis scenarios; there are poor collection adaptability and incomplete metadata synchronization in cross-system data integration; data inconsistency problems are likely to occur when message processing is abnormal, and the insufficient performance of batch operations will also affect the efficiency of large-scale data processing, making it difficult to meet the comprehensive requirements of real-time, integrity, and traceability in enterprise-level data governance. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method for facilitating data lineage collection and analysis to achieve comprehensive management of data lineage collection.

[0004] To achieve the above object, the present invention provides a method for facilitating data lineage collection and analysis, including: Obtaining original lineage data, where the original lineage data includes task execution logs, application metadata, and cross-system dependency relationships; Performing standardization processing on the original lineage data to generate structured lineage information. The standardization processing includes data deduplication, entity identifier generation, and relationship topology verification. The structured lineage information includes asset unique identifiers, upstream and downstream association relationships, and data operation types; Synchronously writing the structured lineage information into a graph database and a distributed data warehouse. The graph database stores real-time association topologies, and the distributed data warehouse stores full historical versions, and the atomicity of the dual-write operation is ensured through a transaction consistency algorithm; Performing single-asset-level lineage tracing according to the real-time topology of the graph database to extract direct dependency paths and deep association networks, and performing global lineage analysis according to the batch calculation ability of the distributed data warehouse to generate closed-loop link detection results; Generating and outputting a lineage health assessment result according to the direct dependency paths, deep association networks, and closed-loop link detection results. The lineage health assessment result includes at least one of data fault warning, circular dependency identification, and hot asset marking; And associating the lineage health assessment result with the feedback of the collection end of the original lineage data through a dynamic message routing algorithm.

[0005] Further, perform normalization processing on the original lineage data to generate structured lineage information, including: Deduplicate the original lineage data to eliminate duplicate task execution logs and redundant dependency relationships, obtain the deduplicated data entities, and extract the data operation types from the task execution logs; Generate a globally unique entity identifier for the deduplicated data entities, denoted as the asset unique identifier; Moreover, bind the asset unique identifier to the data asset type and the system hierarchy to which it belongs to obtain the dependency relationship with the entity identifier; Perform relationship topology verification on the dependency relationship with the entity identifier, identify and eliminate the upstream and downstream association relationships with logical conflicts, and obtain the verified upstream and downstream association relationships; Output the structured lineage information including the asset unique identifier, the verified upstream and downstream association relationships, and the data operation types.

[0006] Further, perform data deduplication on the original lineage data to eliminate duplicate task execution logs and redundant dependency relationships, obtain the deduplicated data entities, and extract the data operation types from the task execution logs, including: Identify duplicate task execution logs by comparing hash values and retain the task execution log with the latest timestamp; Detect redundant dependency relationships based on the combined key of the source system identifier, table name, and field name; Use regular expressions to match the operation keywords in the task execution logs and extract the data operation types of insert, delete, update, and query; Generate a globally unique entity identifier for the deduplicated data entities, denoted as the asset unique identifier, including: Generate the entity identifier using a three-segment coding rule, where the entity identifier includes the system code, asset type code, and serial number; Ensure the uniqueness of the entity identifier through a distributed unique ID generator to obtain the asset unique identifier; Bind the asset unique identifier to the data asset type and the system hierarchy to which it belongs to obtain the dependency relationship with the entity identifier, including: Establish a data asset type classification tree, including three levels: database, table, and field; Determine the system hierarchy to which it belongs according to the system topology diagram, bind the three-level relationship of data center - business domain - subsystem, and obtain the dependency relationship with the entity identifier; Perform relationship topology verification on the dependency relationship with the entity identifier, identify and eliminate the upstream and downstream association relationships with logical conflicts, and obtain the verified upstream and downstream association relationships, including: Use a graph traversal algorithm to detect closed-loop dependency edges of upstream and downstream relationships, and disconnect the last formed dependency edge when a circular reference is found; The metadata registry verifies the dangling nodes in the dependency relationships with entity identifiers, removes the associations pointing to non-existent assets, and obtains verified upstream and downstream association relationships.

[0007] Furthermore, the structured lineage information is synchronously written into the graph database and distributed data warehouse, including: Write the asset's unique identifier as a vertex attribute in the graph database, with the vertex type corresponding to the data asset type; Convert the verified upstream and downstream relationships into directed edges in the graph database. The attributes of the directed edges include the data operation type. Furthermore, the structured lineage information is classified by data asset type, and fact tables and dimension tables are constructed in the distributed data warehouse. The primary key of the fact table is mapped to the unique identifier of the asset. Create a bidirectional index table between the vertex ID of the graph database and the primary key of the fact table. The bidirectional index table includes system-level fields.

[0008] Furthermore, the atomicity of the double-write operation is ensured through a transaction consistency algorithm, including: Record the vertex and edge information to be written in the graph database's write-ahead log. At the same time, cache the lineage information to be converted in a temporary table in the distributed data warehouse. When the bidirectional index table is successfully updated, the vertex and edge write operation of the graph database and the insert operation into the data warehouse fact table are executed in parallel; When a storage system write fails, the executed operations are rolled back according to the write-ahead log, and the residual data is cleared through the bidirectional index table; Locate the failed record through the system-level fields and re-initiate synchronous writing of the system-level fields for the failed record.

[0009] Furthermore, we conduct single-asset lineage tracing based on the real-time topology of the graph database, extracting direct dependency paths and deep association networks, including: Using the asset's unique identifier as the starting node, a single-hop traversal algorithm is used to extract the direct dependency path in the graph database. The direct dependency path consists of a single edge connecting the starting node and its directly adjacent nodes, and records the data asset type, data operation type, and system boundary jump markers between adjacent nodes. Perform multi-order neighborhood expansion analysis based on the end nodes of the directly dependent paths, including: Construct an upstream dependency subnet by recursively traversing the in-degree edges in the graph database, construct a downstream influence subnet by recursively traversing the out-degree edges in the graph database, and merge the upstream dependency subnet and the downstream influence subnet to form a deep association network. Both the upstream dependency subnet and the downstream influence subnet contain node connection relationships with at least two hops. The method further includes: Assign a basic weight value to a single edge in the direct dependency path. The basic weight value is calculated based on the conversion cost of the corresponding data operation type, the system jump cost, and the freshness of the source node data. Calculate a composite weight value for the multi-hop paths in the deep association network. The composite weight value is the weighted accumulation of the basic weight values of all single edges on the multi-hop path. Render the analysis results through a visualization engine. The direct dependency path is displayed as a solid line edge, and the multi-hop paths in the deep association network are displayed as gradient dashed line edges.

[0010] Furthermore, perform global lineage analysis according to the batch computing ability of the distributed data warehouse to generate closed-loop link detection results, including: Perform batch computing on the structured lineage information based on a preset scanning strategy to obtain basic lineage elements, which include table-level dependency relationships, job scheduling sequences, and cross-system transmission records. Perform topological traversal on the table-level dependency relationships to extract closed-loop dependency features, which include circular reference paths and cross-system two-way data flows. Also, perform timeliness analysis on the job scheduling sequences to obtain link timeliness features. Based on the closed-loop dependency features and link timeliness features, establish an abnormal closed-loop determination model. The abnormal closed-loop determination model includes multiple determination dimensions, and the determination dimensions are configured as data update timeliness abnormal dimension, scheduling cycle conflict dimension, and resource competition deadlock dimension. Construct a full-linkage lineage map of core data assets and a sampling dependency chain of ordinary data assets according to the basic lineage elements. Associate and store the closed-loop dependency features, link timeliness features, abnormal closed-loop determination model, and full-linkage lineage map to generate closed-loop link detection results. The closed-loop link detection results include the topological structure features of circular reference closed-loops, the scheduling interval distribution of timeliness abnormal closed-loops, the competing resource features of resource deadlock closed-loops, and the severity level evaluation parameters of various types of closed-loops.

[0011] Furthermore, generate and output a lineage health assessment result according to the direct dependency path, deep association network, and closed-loop link detection results, including: Extract path health parameters based on the direct dependency path. The path health parameters include dependency depth, cross-system hop count, and critical node density. Perform topological analysis on the deep association network to obtain network health features, including node in-degree, edge weight dispersion, and subnet coupling degree; Analyze the closed-loop health indicators in combination with the closed-loop link detection results, and generate a blood relationship health assessment result including health score, repair priority, hot spot distribution, and influence range.

[0012] Furthermore, associate the blood relationship health assessment result with the feedback of the collection end of the original blood relationship data through the dynamic message routing algorithm, including: Construct a feedback priority queue based on the blood relationship health assessment result, which includes emergency repair items, optimization suggestion items, and regular monitoring items; Dynamically adjust the message routing path according to the load status of the data source system, and preferentially route the emergency repair items to the corresponding collection terminals; Establish a mapping relationship between the blood relationship health assessment result and the collection rules of the original blood relationship data, and automatically trigger the adjustment of the sampling frequency of the corresponding collection end when blood relationship anomalies are detected; Compress the blood relationship health assessment result data packet to ensure that high-priority feedback information is preferentially transmitted in a low-bandwidth environment; During the feedback process, monitor the reception status of each collection end in real time, and start an automatic retransmission link for the blood relationship health assessment result that fails to be received successfully.

[0013] Furthermore, obtain the original blood relationship data, which includes task execution logs, application metadata, and cross-system dependencies, including: Collect the task execution logs of each business system through distributed probes, and extract the task ID, execution time, and upstream and downstream dependencies; Extract application metadata from the database, including table structure change records, field-level data flow, and ETL processing logic; Construct a dependency relationship graph based on the cross-system call link analysis to identify the data transfer paths and interface call frequencies between each system; Generate the original blood relationship data.

[0014] Different from the prior art, the above technical solution obtains the original lineage data including task execution logs, application metadata, and cross-system dependencies, and performs standardization processing on it to generate structured lineage information including asset unique identifiers, upstream and downstream association relationships, and data operation types, effectively solving the acquisition adaptability problem caused by data heterogeneity in traditional solutions. By synchronously writing the structured lineage information into the graph database and the distributed data warehouse, where the graph database stores the real-time association topology and the distributed data warehouse stores the full historical versions, and using the transaction consistency algorithm to ensure the atomicity of the dual-write operation, it not only retains the real-time query advantage of the graph database but also breaks through the performance bottleneck through the distributed data warehouse, avoiding the atomicity guarantee problem of metadata changes in traditional solutions. Based on the real-time topology of the graph database for single-asset-level lineage tracking and the batch calculation ability of the distributed data warehouse for global lineage analysis, directly dependent paths, deep association networks, and closed-loop link detection results are synchronously generated, significantly improving the batch operation performance and high-concurrency processing ability. Through the dynamic message routing algorithm, the lineage health assessment results including data fault warning, cyclic dependency identification, and hot asset marking are fed back to the collection end, forming a closed-loop management mechanism from data collection, analysis to optimization, overcoming the problem of disconnection between the analysis results and the collection end in traditional solutions.

[0015] The above description of the invention content is only an overview of the technical solution of this application. In order to enable those of ordinary skill in the art to understand the technical solution of this application more clearly, and then to implement it according to the content recorded in the description and the drawings, and in order to make the above objects, other objects, features, and advantages of this application more easily understood, the following is described in conjunction with the specific embodiments and drawings of this application. Brief Description of the Drawings

[0016] The drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of the specific embodiments of the present invention and other related contents, and should not be considered as a limitation to this application.

[0017] In the accompanying drawings of the specification: Figure 1 It is a step schematic diagram of steps S101 to S105 described in the specific embodiment; Figure 2 It is a step schematic diagram of steps S201 to S204 described in the specific embodiment; Figure 3 It is a step schematic diagram of steps S301 to S303 described in the specific embodiment; Figure 4 It is a step schematic diagram of steps S401 to S404 described in the specific embodiment; Figure 5 It is the architecture schematic diagram described in the specific embodiment. Specific implementation manners

[0018] To describe in detail the possible application scenarios, technical principles, specific implementable solutions, achievable objectives and effects of the present application, etc., the following is a detailed description in conjunction with the listed specific embodiments and with reference to the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and therefore are only examples and cannot be used to limit the protection scope of the present application.

[0019] Please refer to Figure 1 , this embodiment provides a method for facilitating data lineage collection and analysis, including: S101. Obtain original lineage data, where the original lineage data includes task execution logs, application metadata, and cross-system dependency relationships; S102. Perform standardization processing on the original lineage data to generate structured lineage information. The standardization processing includes data deduplication, entity identifier generation, and relationship topology verification. The structured lineage information includes asset unique identifiers, upstream and downstream association relationships, and data operation types; S103. Synchronously write the structured lineage information into a graph database and a distributed data warehouse. The graph database stores real-time association topologies, and the distributed data warehouse stores full historical versions, and ensures the atomicity of the dual-write operation through a transaction consistency algorithm; S104. Perform single-asset-level lineage tracking based on the real-time topology of the graph database to extract direct dependency paths and deep association networks, and perform global lineage analysis based on the batch computing capabilities of the distributed data warehouse to generate closed-loop link detection results; S105. Generate and output a lineage health assessment result based on the direct dependency path, deep association network, and closed-loop link detection result. The lineage health assessment result includes at least one of data fault warning, cyclic dependency identification, and hot asset marking; and, associate the lineage health assessment result with the collection end feedback of the original lineage data through a dynamic message routing algorithm.

[0020] In step S101, obtaining the original lineage data means collecting original data including task execution logs, application metadata, and cross-system dependency relationships from the business system. The task execution logs record the running status and dependency relationships of data processing tasks, the application metadata includes data table structures and ETL conversion rules, and the cross-system dependency relationships reflect the data transfer paths between different systems. By using distributed probes to collect log data in real time, combined with metadata extraction tools to obtain structured definitions, and parsing system call links to construct initial dependency relationships, the unified access of multi-source heterogeneous data is realized, providing a complete data basis for subsequent analysis.

[0021] In step S102, the standardization process of the original lineage data refers to performing operations such as data deduplication, entity identifier generation, and relationship topology verification on the collected original data, and finally generating structured lineage information including asset unique identifiers, upstream and downstream association relationships, and data operation types. Data deduplication eliminates duplicate collected log records, entity identifier generation ensures that each data asset has a globally unique identifier, and relationship topology verification validates the logical consistency of dependency relationships. The heterogeneous data is transformed into a unified model through a standardization rule engine, solving the analysis error problem caused by data format differences and providing a standardized input for dual storage writing.

[0022] In step S103, synchronously writing the structured lineage information into the graph database and the distributed data warehouse means writing the data into the real-time association topology of the graph database and the full historical version of the distributed data warehouse simultaneously through a transaction consistency algorithm. The graph database stores the current latest lineage relationship network and supports low-latency queries; the distributed data warehouse stores historical version data by time partition. The two-phase commit protocol is adopted to ensure the atomicity of dual writing, avoiding the risk of data inconsistency caused by asynchronous writing, and at the same time using the scalability of distributed storage to solve the performance problem of the graph database in high-concurrency scenarios.

[0023] In step S104, performing single-asset-level lineage tracing based on the real-time topology of the graph database means extracting the direct dependency paths and deep association networks of specific assets, and at the same time performing global lineage analysis based on the batch computing ability of the distributed data warehouse and generating closed-loop link detection results. The direct dependency paths reflect the first-order upstream and downstream relationships of assets, the deep association networks mine cross-level implicit dependencies through graph traversal algorithms, and the closed-loop link detection identifies abnormal patterns such as circular references. Combining the advantages of real-time query and offline computing can not only meet the immediate analysis needs but also handle large-scale historical data association analysis.

[0024] In step S105, generating the lineage health assessment result means outputting analysis conclusions including data fault warning, circular dependency identification, and hot asset marking based on the direct dependency paths, deep association networks, and closed-loop link detection results, and feeding them back to the collection end of the original lineage data through a dynamic message routing algorithm. The data fault warning identifies the risk of link interruption, and the hot asset marking identifies data entities with high-frequency access or changes. Establishing a closed-loop control mechanism from analysis to collection enables the system to automatically optimize the data collection strategy and solves the problem of the disconnection between the lineage analysis results and the data source.

[0025] In this embodiment, by obtaining multi-source heterogeneous original lineage data and standardizing it into structured information, unified modeling of cross-system data is achieved; through the transaction consistency dual-write mechanism, the graph database and the distributed data warehouse are synchronously maintained, taking into account both real-time query performance and historical data analysis requirements; based on the dual analysis capabilities of real-time topology and batch computing, the depth and breadth of lineage tracing are significantly improved; finally, a closed-loop management is formed through the dynamic feedback mechanism, effectively solving the defects of traditional solutions in terms of data consistency, processing efficiency, and system linkage, providing a full-link, traceable, and self-optimizing management capability for data governance.

[0026] Please refer to Figure 2 , in a further embodiment, the original lineage data is standardized to generate structured lineage information, including: S201. Perform data deduplication on the original lineage data to eliminate duplicate task execution logs and redundant dependency relationships, obtain the deduplicated data entities, and extract the data operation types from the task execution logs; S202. Generate a globally unique entity identifier for the deduplicated data entities, denoted as the asset unique identifier; and bind the asset unique identifier to the data asset type and the system level to which it belongs to obtain the dependency relationship with the entity identifier; S203. Perform relationship topology verification on the dependency relationship with the entity identifier, identify and eliminate the upstream and downstream association relationships with logical conflicts, and obtain the verified upstream and downstream association relationships; S204. Output the structured lineage information including the asset unique identifier, the verified upstream and downstream association relationships, and the data operation types.

[0027] In step S201, performing data deduplication on the original lineage data means performing duplicate data detection and cleaning on the collected task execution logs and cross-system dependency relationships. Specifically, by comparing the log timestamps, task IDs, and data path characteristics, duplicate records caused by duplicate collection or redundant reporting are eliminated, and at the same time, data operation types including data reading, writing, and conversion are parsed from the task execution logs. This operation effectively solves the problem of noise interference in the original data, ensures that each data entity in the subsequent processing has a clear business meaning and operation context, and generates accurate lineage relationships.

[0028] In step S202, generating an asset unique identifier for the de-duplicated data entity means generating a globally unique identifier by using a hash algorithm in combination with the data asset type (such as database table, API interface, file, etc.) and the system hierarchy to which it belongs (such as business system, data warehouse, data lake, etc.). The asset unique identifier is strongly bound to the physical location and business attributes of the data asset, forming a dependency relationship with the entity identifier. By means of a unified identification system, the difference in naming specifications between different systems is solved, enabling cross-system data assets to be uniquely identified and accurately associated, and solving the problem of broken lineage caused by chaotic identification.

[0029] In step S203, performing a relationship topology check on the dependency relationship with the entity identifier means detecting logical conflicts existing in the upstream and downstream association relationships through a directed acyclic graph (DAG) verification algorithm, including situations such as circular dependencies, invalid references, and type mismatches. Automatically eliminate the association relationships that violate the data flow rules (such as a report having a reverse dependency on the original data table), and only retain the verified upstream and downstream association relationships. By replacing the traditional manual review method with an automated verification mechanism, the deviation of lineage analysis caused by incorrect associations is greatly reduced, and the credibility of the data relationship is improved.

[0030] In step S204, outputting structured lineage information means encapsulating the fully processed asset unique identifier, the verified upstream and downstream association relationships, and the data operation type into an output result in a unified format. Among them, the asset unique identifier serves as the unique index of the data entity, the upstream and downstream association relationships form the edges of the lineage network, and the data operation type annotates the specific way of establishing the relationship (such as SQL import, API call, etc.). The structured data produced in this step provides a standardized input for subsequent dual storage writing, and the complete metadata description ability enables lineage analysis to be refined to the specific operation granularity, overcoming the problem of single relationship type.

[0031] This embodiment ensures the purity of the original lineage data through data de-duplication, realizes the accurate matching of cross-system assets through the global unique identifier, excludes logically abnormal associations through topology check, and finally outputs highly complete structured lineage information. It significantly improves the accuracy and consistency of the lineage data, enables the subsequent storage and analysis links to work based on a reliable data foundation, solves defects such as traceability distortion and association loss caused by data quality problems, and provides a standardized data preprocessing ability for full-link lineage management.

[0032] In a further embodiment, data de-duplication is performed on the original lineage data to eliminate duplicate task execution logs and redundant dependency relationships, obtaining de-duplicated data entities, and extracting data operation types from the task execution logs, including: Identifying duplicate task execution logs by comparing hash values, and retaining the task execution log with the latest timestamp; Detect redundant dependency relationships based on the composite key of the source system identifier, table name, and field name; Use regular expressions to match operation keywords in the task execution log and extract data operation types of insert, delete, update, and query types; Generate a globally unique entity identifier for the de-duplicated data entity, denoted as the asset unique identifier, including: Adopt a three-segment encoding rule to generate the entity identifier, which includes a system code, an asset type code, and a serial number; Ensure the uniqueness of the entity identifier through a distributed unique ID generator to obtain the asset unique identifier; Bind the asset unique identifier to the data asset type and the system hierarchy it belongs to to obtain the dependency relationship with the entity identifier, including: Establish a data asset type classification tree, which includes three levels: database, table, and field; Determine the system hierarchy it belongs to according to the system topology diagram, bind the three-level relationship of data center - business domain - subsystem to obtain the dependency relationship with the entity identifier; Conduct relationship topology verification on the dependency relationship with the entity identifier, identify and eliminate upstream and downstream association relationships with logical conflicts to obtain the verified upstream and downstream association relationships, including: Adopt a graph traversal algorithm to detect the closed-loop dependency edges of the upstream and downstream association relationships, and disconnect the last formed dependency edge when a circular reference is found; Verify dangling nodes in the dependency relationship with the entity identifier through the metadata registry, eliminate associations pointing to non-existent assets, and obtain the verified upstream and downstream association relationships.

[0033] In this embodiment, identifying duplicate-collected task execution logs through hash value comparison means performing MD5 hash calculation on the complete content of the task execution log. When the same hash value is detected, it is determined as a duplicate log, and the record with the latest timestamp is retained as valid data. Detecting redundant dependency relationships based on the composite key of the source system identifier, table name, and field name means splicing these three elements into a composite key, and when exactly the same composite key is detected, it is determined as a redundant dependency. Using regular expressions to match operation keywords in the task execution log is to extract standardized operation types from the log text through predefined regular patterns (such as keywords like SELECT, UPDATE, etc.), ensuring that the original lineage data has completed noise filtering before entering the processing flow.

[0034] In this embodiment, generating entity identifiers using a three - segment encoding rule means concatenating the system encoding (such as the ERP system encoding 01), the asset type encoding (such as the table type encoding 02), and the distributed serial number to form the main body of the identifier. Ensuring the uniqueness of entity identifiers through a distributed unique ID generator means calling the snowflake algorithm service to generate the serial number segment to avoid ID conflicts in a cluster environment. The established asset unique identifier system has clear semantic features, which can not only identify the asset ownership but also distinguish asset categories, providing a stable anchor point for cross - system lineage tracing.

[0035] In this embodiment, establishing a data asset type classification tree means constructing a tree structure that includes databases (root nodes), tables (intermediate nodes), and fields (leaf nodes), with each node bound to a type encoding. Determining the hierarchical system to which an asset belongs based on the system topology map means, according to the pre - defined hierarchical mapping rules, associating the asset with the hierarchical path of data center (such as DC01) - business domain (such as the finance domain) - subsystem (such as the reimbursement system). The implemented hierarchical binding gives the lineage relationship spatial dimension attributes and supports multi - dimensional analysis according to the business architecture.

[0036] In this embodiment, using a graph traversal algorithm to detect closed - loop dependency edges means performing a depth - first search on the dependency graph. When it is found that the ancestor node of a certain node points to itself, the last generated edge is automatically cut off. Verifying dangling nodes through the metadata registry means querying the central metadata database to verify whether the target assets in the dependency relationship exist and removing the invalid edges pointing to unregistered assets. Eliminating logical errors in the lineage graph through an automated verification mechanism ensures the true validity of the dependency relationship.

[0037] In this embodiment, data redundancy is eliminated through hash deduplication and composite key detection. An analyzable asset unique identifier system is constructed using a three - segment encoding. Multi - dimensional positioning of assets is achieved based on the classification tree and system topology. The logical correctness of the dependency relationship is ensured by means of graph algorithms and metadata verification. The efficient purification and structured transformation of the original lineage data are realized, enabling subsequent lineage analysis to be carried out based on a standardized and trustworthy data foundation, effectively solving the problems of tracing distortion and association loss caused by data quality issues.

[0038] Please refer to Figure 3 , in a further embodiment, synchronously writing structured lineage information into a graph database and a distributed data warehouse includes: S301. Writing the asset unique identifier as a vertex attribute of the graph database, where the vertex type corresponds to the data asset type; S302: Convert the verified upstream and downstream relationships into directed edges in a graph database, where the attributes of the directed edges include data operation types; and classify the structured lineage information by data asset type, constructing fact tables and dimension tables in a distributed data warehouse, with the primary key of the fact table maintaining a mapping relationship with the asset's unique identifier. S303: Create a bidirectional index table of the vertex ID of the graph database and the primary key of the fact table, where the bidirectional index table includes system-level fields.

[0039] In step S301, writing the asset's unique identifier as a vertex attribute in the graph database involves storing the globally unique identifier generated during the normalization phase along with the corresponding data asset type (e.g., table, view, file, etc.) as a vertex in the graph database. Vertex types strictly map to the technical classification of the data asset. For example, database table vertices contain schema name and table name attributes, while API interface vertices contain service path and request method attributes. The graph database's schema constraints ensure the integrity and type safety of vertex attributes, enabling subsequent graph traversal queries to be optimized based on asset type, addressing the performance issues of relational databases in modeling complex network relationships.

[0040] In step S302, converting verified upstream and downstream relationships into directed edges in the graph database involves modeling the standardized, verified data flow relationships as directed edges connecting vertices, and embedding specific data operation types (such as SELECT, INSERT, PROCEDURE, etc.) in the edge attributes. Simultaneously, data is organized in a star schema within the distributed data warehouse, with fact tables (recording instances of lineage relationships) and dimension tables (describing asset attributes) constructed using unique asset identifiers as a link. This ensures a one-to-one correspondence between network relationships in the graph database and relational records in the data warehouse. This achieves a dual representation of relational data, retaining the intuitive relevance of the graph structure and the large-scale analytical capabilities of a data warehouse, overcoming the functional limitations of a single storage solution in specific scenarios.

[0041] In step S303, establishing a bidirectional index table involves creating a bridge table that maintains the mapping between graph database vertex IDs and data warehouse fact table primary keys, thereby recording additional system-level information (e.g., ODS, DWD, and APP layers). This index table is implemented using a distributed key-value store, supporting millisecond-level bidirectional query responses. The unified cross-storage addressing mechanism established in this step allows subsequent analysis tasks to freely switch query engines without repetitive data identification, eliminating the data association costs incurred by storage isolation.

[0042] This implementation precisely models data asset entities through graph database vertices, fully expresses operational semantics through directed edges, and leverages the star schema of a distributed data warehouse to achieve analysis-friendly data organization. The resulting bidirectional indexing system streamlines data associations between different storage engines. This simultaneously meets the low-latency graph query requirements for real-time lineage tracking and the batch computing power required for large-scale historical analysis, resolving the performance and scalability challenges inherent in traditional single-storage architectures.

[0043] See also Figure 4 In a further embodiment, the atomicity of the double write operation is ensured by a transaction consistency algorithm, including: S401. Record the vertex and edge information to be written in the graph database write-ahead log. At the same time, cache the lineage information to be converted in a temporary table of the distributed data warehouse. S402: After the bidirectional index table is successfully updated, the vertex and edge writing operation of the graph database and the insert operation into the data warehouse fact table are executed in parallel; S403: When a storage system write fails, the executed operations are rolled back according to the write-ahead log, and residual data is cleared through the bidirectional index table; S404: Locate the failed record through the system-level field, and re-initiate synchronous writing of the system-level field for the failed record.

[0044] In step S401, recording pending vertex and edge information in the graph database's write-ahead log means persisting all pending graph data changes (including vertex attribute updates and edge relationship adjustments) in the form of transaction logs to the write-ahead log (WAL) before actual data writing. Simultaneously, a temporary table is created in the distributed data warehouse to temporarily store the ETL-converted lineage information. This temporary table retains the complete fields of the original structured lineage information and is marked as pending. This ensures that even in the event of a system failure, unfinished write operations can be recovered through the log and temporary table, mitigating the risk of data loss in asynchronous dual-write solutions.

[0045] In step S402, after the bidirectional index table is successfully updated, a parallel dual write operation is performed. This means that only after the bidirectional index table (a bridge table that records the mapping between graph database vertex IDs and data warehouse primary keys) has completed the insertion of new records, will the graph database's graph structure update and the data warehouse's fact table insertion be triggered simultaneously. The graph database operation uses a batch API to submit the vertex and edge changes all at once, while the data warehouse writes to the fact and dimension tables through a distributed transaction interface. Using the bidirectional index table as a synchronization point ensures that the two storage systems maintain logically consistent write timing, avoiding temporary data inconsistencies caused by differences in write timing.

[0046] In step S403, when the storage system fails to write, performing a rollback operation means restoring the state of the graph database by reversing the operations recorded in the write-ahead log (such as deleting the written vertices, revoking edge relationships, etc.), and at the same time cleaning up the corresponding uncommitted data in the data warehouse temporary table. Locate the residual records that have been partially written according to the two-way index table, and ensure data consistency across storage systems through the background cleaning process. Especially for scenarios such as network partitioning or node downtime in a distributed environment, it can eliminate dirty data more thoroughly than a simple retry strategy, ensuring the atomicity requirements of cross-system transactions.

[0047] In step S404, locating and retrying failed records through system-level fields means using system-level fields (such as hierarchical identifiers like ODS / DWD / APP, etc.) to accurately screen and classify the lineage information with write failures. Automatically identify the failed records at a specific level, reconstruct the write request according to the original information in the write-ahead log, and preferentially retry the data at the critical business level. Through the hierarchical processing mechanism, while ensuring eventual consistency, it optimizes the recovery efficiency, significantly reducing the system load compared to full-scale retry, and is particularly suitable for large-scale lineage data synchronization scenarios.

[0048] This embodiment constructs a reliable transaction preparation stage through the write-ahead log and temporary tables, uses the two-way index table as the synchronization coordination point for cross-system writing, and cooperates with a perfect exception rollback and hierarchical retry mechanism to achieve strong consistency data synchronization between the graph database and the distributed data warehouse. While ensuring the writing performance, this solution completely solves the problem of transaction atomicity in a distributed environment across heterogeneous storage systems, ensuring the accuracy and reliability of lineage data in the full-link analysis, and providing a solid underlying storage guarantee for data governance.

[0049] In a further embodiment, single-asset-level lineage tracking is performed according to the real-time topology of the graph database, and the direct dependency path and the deep association network are extracted, including: Taking the asset unique identifier as the starting node, extracting the direct dependency path in the graph database through a single-hop traversal algorithm. The direct dependency path is composed of the starting node and its directly adjacent nodes connected by a single edge, and records the data asset type, data operation type, and system boundary jump mark between adjacent nodes; Based on the end node of the direct dependency path, perform multi-order neighborhood expansion analysis, including: Construct an upstream dependency subnet by recursively traversing the in-degree edges in the graph database, construct a downstream influence subnet by recursively traversing the out-degree edges in the graph database, and merge the upstream dependency subnet and the downstream influence subnet to form a deep association network. Both the upstream dependency subnet and the downstream influence subnet contain node connection relationships with at least two hops; The method further includes: Assign a basic weight value to a single edge in the direct dependency path. The basic weight value is calculated based on the conversion cost of the corresponding data operation type, the system jump cost, and the freshness of the source node data; Calculate the composite weight value for the multi-hop paths in the deep association network. The composite weight value is the weighted accumulation of the basic weight values of all single edges on the multi-hop path; Render the analysis results through a visualization engine. The direct dependency path is displayed with solid edges, and the multi-hop paths in the deep association network are displayed with gradient dashed edges.

[0050] In this embodiment, performing a single-hop traversal algorithm starting from the asset unique identifier means that after locating the target asset vertex in the graph database, scan the adjacent nodes along its directly connected incoming and outgoing edges to generate a linear path connected by a single edge. Record different entities such as the data asset type distinction table, fields, etc. The data operation type is marked with specific operations such as SELECT and JOIN, and the system boundary jump marker identifies cross-system dependencies. Constructing the upstream dependency subnet and the downstream impact subnet through recursive traversal means starting from the end node of the direct dependency path, continuously backtracking along the incoming edges to construct the upstream dependency chain, and extending along the outgoing edges to construct the downstream impact chain, and finally merging to form a network structure containing multi-level nodes. The depth threshold is set during the recursive process to prevent infinite expansion, and at least two-hop relationships are retained to ensure the value of network analysis.

[0051] Preferably, in this embodiment, assigning a basic weight value to a single edge means comprehensively calculating the edge weight according to the conversion cost of the data operation type (such as the JOIN operation cost is higher than SELECT), the system jump cost (cross-data center communication cost), and the freshness of the source node data (data update time). The composite weight value of the multi-hop path uses exponential decay weighted accumulation to highlight the influence intensity of proximal dependencies. The visualization rendering uses solid edges to highlight the key connections of the direct dependency path, and uses gradient dashed edges to display the secondary paths of the deep association network. The line width is positively correlated with the path weight. Color coding is used to distinguish different system boundaries, and the node shape represents the asset type. This presentation method realizes the intuitive understanding of complex lineage networks and assists in quickly locating the core data flow.

[0052] This embodiment accurately captures direct dependency relationships through single-hop traversal, constructs a global association network using recursive expansion, quantifies path weights based on multi-dimensional metrics, and combines visualization technology to achieve a three-dimensional presentation of lineage topology. It realizes the penetrative analysis from micro single assets to macro data networks, making data lineage tracking both accurate at the atomic level and complete at the system level, providing a multi-granularity analysis tool for data change impact assessment and root cause fault location.

[0053] In a further embodiment, global lineage analysis is performed according to the batch computing ability of the distributed data warehouse to generate closed-loop link detection results, including: Execute batch calculations on the structured blood relationship information based on a preset scanning strategy to obtain basic blood relationship elements, where the basic blood relationship elements include table-level dependency relationships, job scheduling sequences, and cross-system transmission records; Perform a topological traversal on the table-level dependency relationships to extract closed-loop dependency features. The closed-loop dependency features include circular reference paths and cross-system bidirectional data flows. Additionally, perform a timeliness analysis on the job scheduling sequence to obtain link timeliness features; Based on the closed-loop dependency features and link timeliness features, establish an abnormal closed-loop determination model. The abnormal closed-loop determination model includes multiple determination dimensions, and the determination dimensions are configured as data update timeliness abnormal dimensions, scheduling period conflict dimensions, and resource competition deadlock dimensions; Construct a full-link blood relationship graph of core data assets and a sampling dependency chain of ordinary data assets according to the basic blood relationship elements; Associate and store the closed-loop dependency features, link timeliness features, abnormal closed-loop determination model, and full-link blood relationship graph to generate a closed-loop link detection result. The closed-loop link detection result includes the topological structure features of the circular reference closed-loop, the scheduling interval distribution of the timeliness abnormal closed-loop, the competing resource features of the resource deadlock closed-loop, and the severity level evaluation parameters of each type of closed-loop.

[0054] In this embodiment, performing batch calculations based on a preset scanning strategy means using a distributed parallel computing framework to perform a full-scale scan of the blood relationship information, extracting table-level dependency relationships (such as the ETL relationship from table A to table B), job scheduling sequences (the order of job execution), and cross-system transmission records (such as Kafka message queue transmission). Performing a topological traversal to extract closed-loop dependency features means detecting circular structures in the table-level dependencies through graph algorithms and identifying circular reference paths (such as table A → table B → table C → table A) and cross-system bidirectional flows (systems X and Y transmit data to each other). The job scheduling timeliness analysis calculates the deviation between the actual execution time of the job and the planned interval to obtain link timeliness features (such as the delay rate). Establishing an abnormal closed-loop determination model means configuring three core dimensions: the data update timeliness abnormal dimension (the upstream table is updated later than the downstream table), the scheduling period conflict dimension (the job scheduling frequencies do not match), and the resource competition deadlock dimension (multiple jobs compete for the same resource). A quantitative threshold is set for each dimension, and an abnormal flag is triggered when the feature value exceeds the threshold.

[0055] In this embodiment, constructing a full-link blood relationship graph means restoring the complete link of core assets (such as key report data), and randomly sampling ordinary assets to generate representative dependency chains. The graph storage includes node attributes (table structure information) and edge attributes (dependency type and timeliness). Associating and storing the closed-loop detection result means persisting the identified closed-loop topological structure (such as the sequence of nodes in the circular path), the timeliness abnormal distribution (delay duration statistics), the resource competition features (list of resources contended for locks), and the severity level (scored based on the impact scope) to the analysis result library.

[0056] In this embodiment, full-scale lineage scanning is achieved through distributed batch computing. By combining graph algorithms with timeliness analysis, closed-loop features are identified, a multi-dimensional anomaly determination model is established, and a differentiated lineage graph storage system is constructed. It realizes the automatic detection of key issues such as circular references, timeliness conflicts, and resource deadlocks from a large amount of lineage data, provides a basis for closed-loop governance for data architecture optimization, and effectively prevents data quality accidents caused by dependency anomalies. The closed-loop severity assessment parameters support the prioritization of issue handling, improving the efficiency of data governance.

[0057] In a further embodiment, based on the direct dependency path, deep association network, and closed-loop link detection results, a lineage health assessment result is generated and output, including: Extract path health parameters based on the direct dependency path. The path health parameters include dependency depth, cross-system hop count, and critical node density; Conduct topological analysis on the deep association network to obtain network health features. The network health features include node in-degree, edge weight dispersion, and subnet coupling degree; Combine the closed-loop link detection results to analyze the closed-loop health indicators, and generate a lineage health assessment result including health score, repair priority, hotspot distribution, and impact scope.

[0058] In this embodiment, extracting path health parameters based on the direct dependency path means calculating the hierarchical depth of the dependency link (such as the maximum nesting level), the number of times crossing the system boundary (such as the number of jumps between different data centers), and the proportion of critical nodes (such as the frequency of occurrence of core tables in the path). This quantitative indicator reflects the complexity and stability of a single data stream. The higher the depth value and the more cross-system hops, the greater the health risk. Conducting topological analysis on the deep association network means counting the in-degree of each node (the number of times it is depended on) to measure the importance of assets, calculating the variance of edge weights to evaluate the balance of dependency intensity, and analyzing the connection density between subnets to judge the modularity degree. The network health features reveal the overall quality of the data architecture. High-in-degree nodes, unbalanced weight edges, and strongly coupled subnets are all potential architecture problems.

[0059] In this embodiment, parsing the closed-loop health metrics means mapping the severity level in the closed-loop link detection results to a health score (0-100 points), setting priorities according to the scope of influence (number of systems involved) and repair cost (difficulty of transformation), and identifying hotspots through closed-loop type clustering (such as frequent timeliness anomalies in a certain business domain). In this embodiment, through the three-dimensional evaluation of path parameters, network characteristics, and closed-loop metrics, a lineage health report containing quantitative scores, processing priorities, problem distributions, and potential impacts is generated. It realizes the standardized measurement and visual presentation of data lineage quality, helps identify high-risk dependency paths, fragile network structures, and frequent closed-loop problems. The health assessment results can directly guide resource allocation, prioritize the handling of key problems with low scores, high priorities, and wide impacts, and improve the robustness of the overall data system.

[0060] In a further embodiment, the lineage health assessment results are associated with the feedback of the original lineage data collection end through a dynamic message routing algorithm, including: Construct a feedback priority queue based on the lineage health assessment results, where the feedback priority queue includes emergency repair items, optimization suggestion items, and regular monitoring items; Dynamically adjust the message routing path according to the load status of the data source system, and preferentially route the emergency repair items to the corresponding collection terminals; Establish a mapping relationship between the lineage health assessment results and the collection rules of the original lineage data, and automatically trigger the adjustment of the sampling frequency of the corresponding collection end when lineage anomalies are detected; Compress the lineage health assessment result data packet to ensure the preferential transmission of high-priority feedback information in a low-bandwidth environment; During the feedback process, monitor the reception status of each collection end in real time, and start an automatic retransmission link for the lineage health assessment results that are not successfully received.

[0061] In this embodiment, constructing a feedback priority queue means classifying the lineage health assessment results according to the urgency level: emergency repair items (such as serious closed-loop problems), optimization suggestion items (such as unbalanced weight edges), and regular monitoring items (such as ordinary dependency paths). The minimum heap data structure is used to achieve dynamic sorting to ensure that high-priority items are always at the top of the pending queue. Dynamically adjusting the message routing path means obtaining the CPU / memory load rate of each data source system through heartbeat detection. When the load of the target system exceeds the threshold, automatically route the emergency repair items to the standby access point or delay the transmission. The routing algorithm comprehensively evaluates network latency, system load, and problem severity to achieve hierarchical scheduling of feedback information. Establishing the mapping relationship of collection rules means configuring the linkage rules between the health score and the sampling frequency. For example, when the health score of the table-level dependency is lower than 60 points, shorten the collection interval of the lineage data related to this table from 1 hour to 10 minutes.

[0062] In this embodiment, the data packet compression adopts a columnar storage format. Numerical fields such as health scores and priorities are compressed using Delta encoding, and text fields are encoded using a dictionary. After compression, priority tags are added to the data packets, and only the metadata of emergency repair items is transmitted when the network bandwidth is insufficient. Real-time monitoring of the reception status means attaching a unique transaction ID to each feedback data packet, and the acquisition end replies with an ACK confirmation signal. If the confirmation is not received within the timeout period, retransmission is retried according to the priority index. The emergency repair items are retried at most 5 times, and the regular monitoring items are retried at most 2 times.

[0063] In this embodiment, the intelligent linkage between the evaluation result and the acquisition end is realized through dynamic message routing, and a closed-loop control system from problem detection to governance implementation is constructed. It realizes the precise allocation of data lineage governance resources, the real-time feedback of emergency problems triggers immediate processing, high-frequency sampling ensures continuous tracking of abnormal data, and the adaptive transmission mechanism ensures information accessibility in different network environments. It significantly improves the timeliness and reliability of data lineage governance, forms a virtuous cycle of "evaluation - feedback - optimization", and effectively reduces the repair delay of data link failures.

[0064] In a further embodiment, the original data lineage data is obtained. The original data lineage data includes task execution logs, application metadata, and cross-system dependencies, including: Collect the task execution logs of each business system through distributed probes, and extract the task ID, execution time, and upstream and downstream dependencies; Extract application metadata from the database, including table structure change records, field-level data flow, and ETL processing logic; Based on the analysis of the cross-system call link, construct a dependency relationship graph to identify the data transfer paths and interface call frequencies between systems; Generate the original data lineage data.

[0065] In this embodiment, collecting the task execution logs of each business system through distributed probes means deploying lightweight collection agents to the production environment to capture the running logs generated by the scheduling engine in real time. The probe extracts the task ID as the unique identifier, records the execution timestamp accurate to milliseconds, and parses the upstream and downstream dependencies such as input and output tables and message queues in the task configuration. An incremental synchronization mechanism is adopted to only transmit the newly added or changed log contents, significantly reducing the network transmission pressure while ensuring data integrity.

[0066] In this embodiment, extracting application metadata from the database involves connecting to the business system's metadata repository to automatically capture table structure change history and data processing logic. This focuses on capturing DDL operation records, such as field additions and type modifications, and parsing field-level mappings in stored procedures and ETL jobs, including column mapping rules and transformation conditions in SELECT-INSERT statements. For complex transformation logic, business processing features such as WHERE filter conditions and JOIN association rules are extracted.

[0067] In this embodiment, building a dependency graph based on cross-system call chain analysis involves integrating call logs from a distributed tracing system with traffic data from an API gateway. By analyzing the parameter paths of RPC calls, data input and output relationships between systems are identified. The frequency and throughput of interface calls are counted to quantify the strength of dependency relationships. Furthermore, the production and consumption records of the message middleware are analyzed to supplement the event-driven data flow path. Ultimately, a weighted directed graph structure is generated, where nodes represent system components and edge weights reflect the actual frequency of data interactions.

[0068] In this embodiment, generating raw lineage data refers to uniformly modeling the collected logs, metadata, and call link information. Entities such as task instances, data tables, and system interfaces, as well as their relationships, are converted into attribute graph models according to a preset standardized format, with each edge labeled with dependency type and time characteristics. Data consistency checks are performed before output to eliminate abnormal records with timestamp conflicts or missing references, and data version identifiers are automatically generated.

[0069] This embodiment achieves accurate capture and dynamic updating of full-link data lineage through the collaborative work of distributed log collection, deep metadata analysis, and cross-system link tracking. An incremental synchronization mechanism is used to ensure data timeliness, standardized modeling solves the integration problem of multi-source heterogeneous data, and versioning management supports traceability analysis of lineage changes. It significantly improves the granularity and accuracy of lineage data, lays a reliable data foundation for subsequent lineage health assessment, and minimizes the impact on production systems through lightweight collection design.

[0070] See also Figure 5 For ease of understanding, the following examples are given to further understand the above content: The existing technology has the following problems: 1. Data is stored only in the graph database, and heterogeneous data storage is not supported. Daily usage scenarios are constrained by the performance of the graph database, which greatly limits the use cases.

[0071] 2. The message communication mechanism and processing logic are highly bound to the internal model, resulting in poor scalability and inability to quickly support the collection of third-party service lineage data.

[0072] 3. Metadata changes in Atlas (such as creating entities and updating relationships) are triggered asynchronously through the Hook mechanism, and the atomicity of message consumption and metadata operations is not guaranteed (e.g., when Kafka message processing fails, the metadata already written to Atlas needs to be manually rolled back). It relies on an external transaction coordinator (such as Apache Sky Walking or Seata) to implement the compensation logic.

[0073] 4. The performance of batch operations is insufficient. In high-concurrency scenarios, the server pressure may surge due to frequent API calls.

[0074] 5. Atlas's lifecycle management of metadata (such as deleting expired data and tracing historical versions) depends on TTL (time-to-live) configuration or manual cleaning, and it cannot automatically manage data versions and aging through the timeline mechanism like Hudi, which may lead to storage expansion or residual dirty data.

[0075] Furthermore, this example provides an architecture as follows: The main framework is spingboot + kafka + neo4j + flinksql + hudi.

[0076] First, the collection end collects the lineage information of tasks and applications through methods such as hooks, scheduled scripts, and service pushes, and pushes it to the common topic message (common-topic) of Kafka. The message contains the message type, creator information, asset, and lineage relationship information. The processing flow of the message type is as follows: Full overwrite according to the creator attribute; Full deletion according to the creator attribute; Add relationships according to the creator, without deleting relationships that do not exist in the message; Secondly, the Spingboot service consumes the common-topic, parses the lineage message, batch-updates the neo4j data, converts the updated lineage information into add / update / delete events of assets and lineage relationships, and sends them to the change data capture topic (cdc-topic) of Kafka. Assets generate a unique ID based on the qualified name of the asset; Lineage relationships generate a unique uuid (identification code) based on the upstream asset, downstream asset, and creator; To ensure the consistency of hudi and neo4j data, when consuming Kafka data, the manual commit method is adopted. Only when both the neo4j and cdc message sends are successful, the consumption progress (commit offset) is recorded. Through Flink SQL, consume the CDC topic in real time, flatten the assets and lineage relationships, and update the records in Hudi based on the unique UUIDs of the assets and lineage relationships to ensure the accuracy of the data in the data warehouse Hudi table and Neo4j. Based on the data warehouse tables, other computing engines such as Spark can be used to calculate the dependency information of all assets, such as all upstream and downstream assets. Figure 5 The archived tables in [[]] can be stored with one partition per day.

[0077] This example has the following beneficial effects: 1. Data is written to Neo4j and the Hudi table in a dual-write manner, improving the reliability of the system through data redundancy and fault tolerance mechanisms.

[0078] 2. The new Hudi table is introduced to store lineage data, making up for the performance bottleneck problem that conventional graph databases cannot query large amounts of data, solving the problem of unable to query ultra-multi-layer lineage data, and supporting multi-node cross-service lineage query, improving the availability of the service and expanding the applicable scenarios.

[0079] 3. The unified message communication mechanism can quickly support the writing of new service lineage data, effectively improving the scalability of the system.

[0080] 4. By strongly binding message consumption and cross-service data operations, the atomicity of data writing is ensured, and automatic rollback occurs in case of failure, achieving the consistency and reliable delivery of data dual-write.

[0081] 5. The "one configuration, one lineage" mechanism is introduced to simultaneously complete data insertion, update, and aging through a single Kafka message, greatly improving the efficiency of data change and simplifying the message content, while effectively reducing resource consumption.

[0082] 6. It can rely on Neo4j to satisfy the upstream and downstream queries of a single asset, and more global lineage analysis can also be performed based on the data warehouse Hudi table.

[0083] Different from the prior art, the above technical solution obtains the original lineage data including task execution logs, application metadata, and cross-system dependencies, and performs standardization processing on it to generate structured lineage information including asset unique identifiers, upstream and downstream association relationships, and data operation types, effectively solving the acquisition adaptability problem caused by data heterogeneity in traditional solutions. By synchronously writing the structured lineage information into the graph database and the distributed data warehouse, where the graph database stores the real-time association topology and the distributed data warehouse stores the full historical versions, and using the transaction consistency algorithm to ensure the atomicity of the dual-write operation, it not only retains the real-time query advantage of the graph database but also breaks through the performance bottleneck through the distributed data warehouse, avoiding the atomicity guarantee problem of metadata changes in traditional solutions. Based on the real-time topology of the graph database for single-asset-level lineage tracking and the batch calculation ability of the distributed data warehouse for global lineage analysis, directly dependent paths, deeply associated networks, and closed-loop link detection results are synchronously generated, significantly improving the batch operation performance and high-concurrency processing ability. Through the dynamic message routing algorithm, the lineage health assessment results including data fault warning, cyclic dependency identification, and hot asset marking are fed back to the collection end, forming a closed-loop management mechanism from data collection, analysis to optimization, overcoming the problem of the disconnection between the analysis result and the collection end in traditional solutions.

[0084] Finally, it should be noted that although the above embodiments have been described in the text and drawings of the specification of the present application, the patent protection scope of the present application cannot be limited thereby. Any technical solution obtained by equivalent structure or equivalent process substitution or modification using the content recorded in the text and drawings of the specification of the present application based on the substantial concept of the present application, as well as any technical solution directly or indirectly implemented in other related technical fields by the above embodiments, is included in the patent protection scope of the present application.

Claims

1. A method for facilitating data lineage collection and analysis, characterized in that Including: Obtain original lineage data, where the original lineage data includes task execution logs, application metadata, and cross-system dependencies; Perform standardization processing on the original lineage data to generate structured lineage information. The standardization processing includes data deduplication, entity identifier generation, and relationship topology verification. The structured lineage information includes asset unique identifiers, upstream and downstream association relationships, and data operation types; Synchronously write the structured lineage information into a graph database and a distributed data warehouse. The graph database stores real-time association topologies, and the distributed data warehouse stores full historical versions, and ensures the atomicity of the dual-write operation through a transaction consistency algorithm; Perform single-asset-level lineage tracking based on the real-time topology of the graph database to extract direct dependency paths and deep association networks, and perform global lineage analysis based on the batch calculation ability of the distributed data warehouse to generate closed-loop link detection results; Generate and output a lineage health assessment result based on the direct dependency path, deep association network, and closed-loop link detection result. The lineage health assessment result includes at least one of data fault warning, cyclic dependency identification, and hot asset marking; And, associate the lineage health assessment result with the feedback of the collection end of the original lineage data through a dynamic message routing algorithm.

2. The method for facilitating data lineage collection and analysis according to claim 1, wherein Perform standardization processing on the original lineage data to generate structured lineage information, including: Perform data deduplication on the original lineage data, eliminate duplicate task execution logs and redundant dependencies, obtain deduplicated data entities, and extract data operation types from the task execution logs; Generate a globally unique entity identifier for the deduplicated data entity, denoted as an asset unique identifier; And, bind the asset unique identifier to the data asset type and the system level to which it belongs to obtain a dependency relationship with an entity identifier; Perform relationship topology verification on the dependency relationship with an entity identifier, identify and eliminate logically conflicting upstream and downstream association relationships, and obtain verified upstream and downstream association relationships; Output structured lineage information including asset unique identifiers, verified upstream and downstream association relationships, and data operation types.

3. The method for facilitating data lineage collection and analysis according to claim 2, characterized in that: Perform data deduplication on the original lineage data, eliminate duplicate task execution logs and redundant dependencies, obtain deduplicated data entities, and extract data operation types from the task execution logs, including: Identify duplicate task execution logs through hash value comparison and retain the task execution log with the latest timestamp; Detect redundant dependencies based on a composite key of the source system identifier, table name, and field name; Use regular expressions to match operation keywords in the task execution logs and extract data operation types of add, delete, modify, and query types; Generate a globally unique entity identifier for the deduplicated data entity, denoted as an asset unique identifier, including: Generate an entity identifier using a three-segment encoding rule. The entity identifier includes a system code, an asset type code, and a serial number; Ensure the uniqueness of the entity identifier through a distributed unique ID generator to obtain the asset unique identifier; Bind the asset unique identifier with the data asset type and the system level to which it belongs to obtain the dependency relationship with the entity identifier, including: Establish a data asset type classification tree, including three levels of database, table, and field; Determine the system level based on the system topology diagram, bind the three-level relationship between data center, business domain, and subsystem, and obtain the dependency relationship with entity identifiers; Perform relationship topology verification on dependency relationships with entity identifiers, identify and eliminate logically conflicting upstream and downstream relationships, and obtain verified upstream and downstream relationships, including: Use a graph traversal algorithm to detect closed-loop dependency edges of upstream and downstream relationships, and disconnect the last formed dependency edge when a circular reference is found; The metadata registry verifies the dangling nodes in the dependency relationships with entity identifiers, removes the associations pointing to non-existent assets, and obtains verified upstream and downstream association relationships.

4. The method for facilitating data lineage collection and analysis according to claim 2, wherein Synchronously writing the structured lineage information into the graph database and the distributed data warehouse includes: Writing the asset unique identifier as a vertex attribute in the graph database, where the vertex type corresponds to the data asset type; Convert the verified upstream and downstream association relationships into directed edges in a graph database, where the attributes of the directed edges include the data operation type; Furthermore, the structured lineage information is classified according to data asset type, and a fact table and a dimension table are constructed in a distributed data warehouse, wherein the primary key of the fact table maintains a mapping relationship with the asset unique identifier; A bidirectional index table of vertex IDs of the graph database and primary keys of fact tables is established, wherein the bidirectional index table includes system-level fields.

5. The method for facilitating data lineage collection and analysis according to claim 4, wherein The atomicity of double write operations is ensured through a transaction consistency algorithm, including: Record the vertex and edge information to be written in the graph database's write-ahead log. At the same time, cache the lineage information to be converted in a temporary table in the distributed data warehouse. When the bidirectional index table is successfully updated, the vertex edge writing operation of the graph database and the data warehouse fact table insertion operation are performed in parallel; When a storage system write fails, the executed operations are rolled back according to the write-ahead log, and residual data is cleared through the bidirectional index table; The failed record is located through the system-level field, and synchronous writing of the system-level field of the failed record is re-initiated.

6. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that: Perform single-asset lineage tracing based on the real-time topology of the graph database, extracting direct dependency paths and deep association networks, including: Using the asset unique identifier as the starting node, a single-hop traversal algorithm is used to extract a direct dependency path from the graph database. The direct dependency path is composed of a single edge connecting the starting node and its directly adjacent nodes, and records the data asset type, data operation type, and system boundary jump markers between adjacent nodes. Based on the end nodes of the directly dependent path, a multi-order neighborhood expansion analysis is performed, including: An upstream dependency subnet is constructed by recursively traversing the in-degree edges in the graph database, and a downstream influence subnet is constructed by recursively traversing the out-degree edges in the graph database. The upstream dependency subnet and the downstream influence subnet are merged to form a deep association network, wherein both the upstream dependency subnet and the downstream influence subnet contain a node connection relationship of at least two hops; The method further comprises: Assign a basic weight value to a single edge in the direct dependency path. The basic weight value is calculated based on the conversion cost of the corresponding data operation type, the system jump cost, and the freshness of the source node data; Calculate a composite weight value for the multi-hop paths in the depth association network. The composite weight value is the weighted sum of the basic weight values of all single edges on the multi-hop path; Render the analysis results through a visualization engine. The direct dependency path is displayed as a solid line edge, and the multi-hop paths in the depth association network are displayed as gradient dashed line edges.

7. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that, Perform global lineage analysis based on the batch computing ability of the distributed data warehouse to generate closed-loop link detection results, including: Execute batch computing on the structured lineage information based on a preset scanning strategy to obtain basic lineage elements, where the basic lineage elements include table-level dependency relationships, job scheduling sequences, and cross-system transmission records; Perform topological traversal on the table-level dependency relationships to extract closed-loop dependency features, where the closed-loop dependency features include circular reference paths and cross-system bidirectional data flows, and perform aging analysis on the job scheduling sequence to obtain link aging features; Based on the closed-loop dependency features and link aging features, establish an abnormal closed-loop determination model. The abnormal closed-loop determination model includes multiple determination dimensions, and the determination dimensions are configured as data update aging exception dimensions, scheduling period conflict dimensions, and resource competition deadlock dimensions; Construct a full-link lineage graph of core data assets and a sampling dependency chain of ordinary data assets based on the basic lineage elements; Associate and store the closed-loop dependency features, link aging features, abnormal closed-loop determination model, and full-link lineage graph to generate the closed-loop link detection results. The closed-loop link detection results include the topological structure features of circular reference closed loops, the scheduling interval distribution of aging abnormal closed loops, the competing resource features of resource deadlock closed loops, and the severity level evaluation parameters of various types of closed loops.

8. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that: Generate and output a lineage health assessment result based on the direct dependency path, depth association network, and closed-loop link detection results, including: Extract path health parameters based on the direct dependency path. The path health parameters include dependency depth, cross-system hop count, and critical node density; Perform topological analysis on the depth association network to obtain network health features. The network health features include node in-degree, edge weight dispersion, and subnet coupling degree; Analyze the closed-loop health indicators in combination with the closed-loop link detection results to generate a lineage health assessment result including health score, repair priority, hotspot distribution, and impact scope.

9. The method for facilitating data lineage collection and analysis according to claim 1, wherein Associate the lineage health assessment result with the feedback of the collection end of the original lineage data through a dynamic message routing algorithm, including: Construct a feedback priority queue based on the lineage health assessment result. The feedback priority queue includes urgent repair items, optimization suggestion items, and regular monitoring items; Dynamically adjust the message routing path according to the load status of the data source system, and preferentially route the urgent repair items to the corresponding collection terminals; Establish a mapping relationship between the lineage health assessment result and the collection rules of the original lineage data, and automatically trigger an adjustment of the sampling frequency of the corresponding collection end when lineage anomalies are detected; Compress the blood relationship health assessment result data packet to ensure the priority transmission of high-priority feedback information in a low-bandwidth environment; During the feedback process, monitor the reception status of each collection end in real time, and initiate an automatic retransmission link for the blood relationship health assessment results that are not successfully received.

10. The method for facilitating data lineage collection and analysis according to claim 1, characterized in that Obtain the original blood relationship data, where the original blood relationship data includes task execution logs, application metadata, and cross-system dependencies, including: Collect the task execution logs of each business system through distributed probes, and extract the task ID, execution time, and upstream and downstream dependencies; Extract application metadata from the database, including table structure change records, field-level data flow, and ETL processing logic; Build a dependency relationship graph based on cross-system call link analysis to identify the data transfer paths and interface call frequencies between systems; Generate the original blood relationship data.

Images