Iterative updating method for data pedigree map

Abstract

The application relates to the technical field of data management, and discloses an iterative updating method for a data bloodline graph, which comprises the following steps: capturing data increment and metadata changes; constructing a directed acyclic graph of data bloodline relationships; performing hierarchical modeling on the data bloodline relationship graph; performing iterative updating on the bloodline graph based on the hierarchical modeling result; optimizing the processing and storage of the incremental data through distributed computing; using a stream processing technology to realize real-time bloodline graph updating; using a machine learning model to predict bloodline path changes and detect and repair abnormal nodes; and capturing data increment and metadata changes, which comprises obtaining data snapshots, including new snapshots and old snapshots of the data. Through the stream processing technology, the machine learning model and the automatic repair mechanism, the application realizes efficient real-time updating of the data bloodline graph, intelligent path prediction and automatic repair of abnormal nodes, and improves the system stability and updating efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to an iterative updating method for data lineage graphs. Background Art

[0002] Data lineage graphs are a crucial tool for modern data management. By describing the flow and dependencies of data within a system, they support scenarios such as data governance, data traceability, and compliance analysis. The iterative update approach for data lineage graphs involves continuously maintaining the integrity and real-time nature of data lineage graphs through incremental capture and dynamic update techniques as the data environment changes.

[0003] Currently, existing methods for iteratively updating data lineage graphs often employ batch processing, capturing incremental data through snapshot comparisons and leveraging dynamic graph structure update techniques to maintain the graph. Some technologies also incorporate topological sorting to implement hierarchical updates of dependencies, thereby improving update efficiency. Furthermore, existing technologies support the processing and storage of large-scale data in a distributed environment, offering good scalability and meeting the needs of iterative data updates at a certain scale.

[0004] However, existing technologies still have shortcomings in terms of real-time performance, intelligence, and resource optimization. For example, existing technologies are insufficiently responsive to high-frequency data changes and are unable to meet real-time update requirements. They lack the ability to predict changes in data dependency paths, and the degree of optimization of update operations is limited. In addition, the detection and repair of abnormal nodes mostly rely on manual operations, which reduces the system's autonomous maintenance efficiency. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention provides an iterative update method for data lineage maps, which solves the problems of the existing technology in iterative update of data lineage maps, lack of real-time performance, lack of predictive ability for data change paths, and low efficiency in abnormal node detection and repair.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for iteratively updating a data lineage map, comprising the following steps:

[0007] Capture data increments and metadata changes;

[0008] Construct a directed acyclic graph of data lineage relationships;

[0009] Perform hierarchical modeling on the data lineage relationship graph;

[0010] Iteratively update the bloodline map based on the hierarchical modeling results;

[0011] Optimize the processing and storage of incremental data through distributed computing;

[0012] Use stream processing technology to achieve real-time updates of bloodline graphs;

[0013] Use machine learning models to predict changes in bloodline paths and detect and repair abnormal nodes.

[0014] Preferably, capturing data increments and metadata changes includes:

[0015] Get data snapshots, including new and old snapshots of data;

[0016] Identify incremental data by comparing snapshots;

[0017] Perform timestamp filtering, hash value verification, and version comparison on incremental data to extract new, changed, and deleted data;

[0018] Capture metadata changes, including additions, modifications, and deletions at the table and field levels;

[0019] Normalize the recognition results into incremental data event format for subsequent processing.

[0020] Preferably, the construction of a directed acyclic graph of data lineage relationships includes:

[0021] Define data nodes, including table-level and field-level data entities;

[0022] Define data flow relationships, including input and output relationships;

[0023] Build a graph of node sets and edge sets based on data dependencies;

[0024] Ensure that the constructed graph structure satisfies the acyclic property.

[0025] Preferably, the hierarchical modeling of the data blood relationship graph includes:

[0026] Divide the nodes in the graph into levels through topological sorting;

[0027] Calculate node priority based on node access frequency and change frequency;

[0028] Nodes with higher priority are divided into the core layer, and nodes with lower priority are divided into the edge layer;

[0029] Mark the dependencies of nodes at each layer to ensure that they are executed first by layer during iterative updates.

[0030] Preferably, the hierarchical modeling results are used to iteratively update the bloodline map, including:

[0031] For core layer nodes, incremental update operations are performed in real time;

[0032] For edge layer nodes, incremental update operations are performed in batches according to time windows;

[0033] Detect data dependency conflicts during incremental updates and mark conflicting nodes and paths;

[0034] Call the rule engine to automatically repair conflicting nodes and paths.

[0035] Preferably, the optimizing the processing and storage of incremental data through distributed computing includes:

[0036] Use a distributed graph database to store bloodline graphs;

[0037] Partition and store the data lineage graph according to the access frequency and priority of the nodes;

[0038] Use a distributed computing framework to perform parallel processing on incremental data;

[0039] The processing results are synchronized to the graph database through the batch write interface.

[0040] Preferably, the use of stream processing technology to achieve real-time kinship map updates includes:

[0041] Monitor data change events through the stream processing engine;

[0042] Parse the captured event stream into incremental update requests;

[0043] Parse data dependencies in real-time event streams and generate incremental relationships;

[0044] The bloodline graph in the graph database is updated in real time based on the analysis results.

[0045] Preferably, the use of a machine learning model to predict changes in bloodline pathways includes:

[0046] Extract the characteristics of historical lineage change data, including access frequency, dependency, and change frequency;

[0047] Use time series models to predict data lineage change paths;

[0048] Generates priority update incremental data requests based on prediction results.

[0049] Preferably, the detecting and repairing of abnormal nodes includes:

[0050] Use anomaly detection algorithms to identify abnormal nodes in the lineage graph;

[0051] Generate repair strategies for detected abnormal nodes;

[0052] Execute the repair strategy and rebuild the dependency relationships of abnormal nodes;

[0053] Record the repair operation log for subsequent audit.

[0054] Preferably, the incremental data capture method includes:

[0055] For large-scale distributed data sources, incremental data is captured in parallel by data sharding;

[0056] Check the consistency of data sources across systems during the capture process and identify possible synchronization errors;

[0057] Implement fault-tolerance processing for identified erroneous data, including retry mechanisms or data compensation strategies;

[0058] The captured incremental data is marked as traceable batch tasks for subsequent iterative updates.

[0059] The present invention provides an iterative update method for data lineage graphs. It has the following beneficial effects:

[0060] 1. The present invention realizes efficient iterative updating of data lineage graphs through technologies such as data sharding, distributed computing, and stream processing engines. It has the ability to capture, process quickly, and update accurately in real time in a dynamic data environment, and can adapt to the multi-level requirements of large-scale distributed data sources and complex data dependencies.

[0061] 2. This invention introduces a machine learning model to predict changes in lineage paths, combines anomaly detection with automatic repair mechanisms, and builds an intelligent graph update framework, which significantly improves the update efficiency and the system's autonomous optimization capabilities, while ensuring the logical integrity and stability of data lineage.

[0062] 3. The present invention optimizes the allocation of update resources through node hierarchical modeling and priority division, and achieves data consistency and resource utilization in high-concurrency scenarios through strategies such as batch management and transaction control. It has good scalability and fault tolerance, and is adaptable to multi-scenario business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the specification of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0065] Please see the attached Figure 1, an embodiment of the present invention provides an iterative update method for a data lineage graph, comprising the following steps:

[0066] S1. Capturing data increments and metadata changes. This enables efficient and accurate capture of data increments and metadata changes, providing reliable input for subsequent iterative updates of the lineage map while reducing the processing cost of the full data set.

[0067] S2. Construct a directed acyclic graph of data lineage relationships. This graph clearly describes data flow and dependencies, provides an accurate basic data structure for hierarchical modeling and iterative updates, and improves the accuracy and stability of graph updates.

[0068] S3. Hierarchical modeling of the data lineage relationship graph significantly improves the efficiency of updating the lineage graph. Prioritizing updates at the core layer ensures the timeliness of key nodes, while batch updates at the edge layer reduce system load.

[0069] S4. Iteratively update the bloodline map based on the hierarchical modeling results. This iterative update method optimizes update efficiency through a hierarchical mechanism while ensuring the consistency of the data bloodline map, thereby reducing the computational overhead of full reconstruction.

[0070] S5. Optimize the processing and storage of incremental data through distributed computing. The distributed computing framework effectively improves the efficiency of incremental data processing. Combined with distributed storage, it optimizes graph access performance and scalability, ensuring system stability in large-scale data scenarios.

[0071] S6. Use stream processing technology to achieve real-time updates of the bloodline map. Stream processing technology enables real-time updates of the bloodline map, ensuring that data changes are reflected in the map in a timely manner, meeting the needs of high-real-time scenarios.

[0072] S7. Use machine learning models to predict changes in bloodline paths and detect and repair abnormal nodes. The machine learning model improves the accuracy of bloodline map change predictions and enables automatic detection and repair of abnormal nodes, thereby improving the intelligence level and maintenance efficiency of the system.

[0073] Please see the attached Figure 1 In a preferred embodiment of the present invention, S1, capturing data increments and metadata changes includes:

[0074] Data snapshots are obtained, including new and old snapshots of data. The essence of data snapshots is a complete record of the state of data. By comparing new and old snapshots, data changes can be identified, thereby avoiding the computational cost of full data reconstruction. Data snapshots use a version management system for unified identification, such as assigning a unique version number and timestamp to each snapshot for subsequent incremental calculation. Version management of snapshots improves the traceability of data and provides efficient input for iterative updates.

[0075] By comparing snapshots, the incremental part of the data is identified, and the difference between snapshots is calculated through set operations:

[0076]

[0077] wherein, is the new snapshot dataset, is the old snapshot dataset, is the incremental dataset.

[0078] Field-level comparison: If the data values in two snapshots are consistent but the field attributes (such as type) have changed, record the field change event separately.

[0079] Result marking: Mark the identified incremental data as three types of events: addition, change, or deletion, for subsequent hierarchical modeling and iterative updating.

[0080] Difference calculation quickly identifies the changed data part by comparing the content and structure of new and old datasets, reducing resource expenditure for processing full data.

[0081] Difference records are stored in a key-value structure to ensure that data changes can be efficiently mapped to specific graph nodes.

[0082] Timestamp filtering, hash value checking, and version comparison are performed on incremental data to extract new data, changed data, and deleted data. The combination of timestamp filtering and hash checking ensures accurate capture of data changes. The version comparison mechanism further classifies incremental data types, providing structured input data for hierarchical modeling and iterative updating. Through the joint use of timestamp, hash value, and version comparison, new, changed, and deleted data are accurately extracted, providing an efficient input mechanism for subsequent processing of incremental data and reducing redundant calculations.

[0083] Capture metadata changes, including table-level and field-level additions, modifications, and deletions. Table-level and field-level metadata capture is based on structured comparison of snapshots, which can accurately locate metadata changes and ensure synchronization of metadata changes with bloodline graph updates. Capturing metadata changes provides complete context information, laying a structured foundation for the construction and update of bloodline graphs, while supporting complex metadata management in dynamic data environments.

[0084] Standardize the recognition results into incremental data event format for subsequent processing. Standardized event format design: Encapsulate all recognized incremental data and metadata changes into a unified event format:

[0085] Event type: add, change, delete;

[0086] Data type: table-level data, field-level data;

[0087] Event content: changed data value, field attribute, or table attribute;

[0088] Event time: the timestamp of the occurrence time;

[0089] Event storage and transmission: Standardized events are stored in queues or databases and transmitted to subsequent hierarchical modeling and iterative update modules through stream processing engines (such as Kafka);

[0090] The design of a standardized event format ensures that different types of incremental data can be processed uniformly, reducing the complexity caused by heterogeneous data structures and improving the scalability of subsequent modules;

[0091] Standardized incremental events provide a unified input interface for subsequent iterative updates, ensuring the simplicity, accuracy, and scalability of the incremental data processing process.

[0092] Please see the attached Figure 1 In a preferred embodiment of the present invention, S2, constructing a directed acyclic graph of data lineage relationships includes:

[0093] Define data nodes, including table-level and field-level data entities. The establishment of node hierarchical relationships enables the graph to display high-level relationships between tables and describe in detail the specific dependencies between fields, ensuring the granularity and hierarchy of lineage information.

[0094] Defining table-level and field-level nodes enables multi-level coverage of kinship descriptions, provides a comprehensive perspective of fields, and lays the foundation for accurate modeling and visualization of data flows.

[0095] Define data flow relationships, including input and output relationships. These relationships together constitute a complete description of the data flow. By accurately recording the source and destination of data, the full-link tracking capability of lineage information is ensured.

[0096] Defining data flow relationships enables full tracking and transparency of the data processing process, providing complete data flow information for subsequent dependency analysis and conflict detection;

[0097] The node and edge sets of the graph are established based on data dependencies. The construction of the node and edge sets essentially maps the lineage information into a graph structure. The graph provides a panoramic view of data dependencies, which facilitates subsequent modeling and analysis.

[0098] The establishment of node sets and edge sets forms the core data structure of the bloodline graph, which can efficiently support the storage, query and visualization of bloodline relationships;

[0099] Ensure that the constructed graph structure meets the acyclic property. The data lineage graph needs to meet the characteristics of a directed acyclic graph (DAG) to ensure the logical consistency and parsability of data dependencies. Loop detection and removal ensure that the graph remains correct during dynamic updates.

[0100] By detecting and removing loops, the acyclic nature of the bloodline graph is ensured, uncertainty in data flow is avoided, and the stability and availability of the graph are improved.

[0101] Please see the attached Figure 1 In a preferred embodiment of the present invention, S3, performing hierarchical modeling on the data lineage relationship graph includes:

[0102] The nodes in the graph are divided into levels through topological sorting. Topological sorting algorithm: a topological sorting algorithm based on depth-first search (DFS) is used to sort the data lineage relationship graph. The nodes are divided into levels. The specific steps include:

[0103] Initialization: for all nodes 、 、 Set unvisited status;

[0104] Recursive search: Starting from any unvisited node Departure, press Point to recursively traverse its successor node;

[0105] Add node: After completing the node After traversing the successor nodes of Add to the result list;

[0106] Arrange in reverse order: the final list is the result of topological sorting;

[0107] Level division: For each node in the topological sorting result list, the level is calculated according to the number of its predecessor nodes;

[0108]

[0109] in For nodes level, Representation node The set of predecessor nodes of ;

[0110] Topological sorting leverages the characteristics of directed acyclic graphs to resolve node dependencies layer by layer, ensuring the correct dependency order when graph updates occur. This hierarchical division provides the foundation for subsequent priority calculations and hierarchical modeling.

[0111] Topological sorting enables dependency resolution and hierarchical division of data nodes, ensuring the logical order of node updates and laying the foundation for accurate execution of iterative updates.

[0112] Node priority is calculated based on the node's access frequency and change frequency. Node priority takes into account both the node's business importance (reflected by access frequency) and data stability (reflected by change frequency). The weight parameter provides flexible adjustment capabilities to adapt to different scenario requirements.

[0113] Priority calculation enables ranking of data nodes by importance, providing an accurate basis for the division of core and edge layers, and ensuring optimal resource allocation and update strategies.

[0114] Nodes with higher priorities are divided into the core layer, and nodes with lower priorities are divided into the edge layer. Nodes in the core layer are usually data entities that are frequently accessed or frequently changed. Their priority update has a direct impact on the overall performance and stability of the system. Nodes in the edge layer can be updated later to save computing resources.

[0115] By dividing the core layer and the edge layer, we can achieve hierarchical update strategies and priority allocation, significantly improving the efficiency of iterative updates and ensuring the timeliness of key data.

[0116] Mark the dependencies of nodes at each layer to ensure that iterative updates are executed in layers first. This strategy ensures that key data nodes are updated first and avoids data inconsistencies caused by incorrect update order of dependency paths.

[0117] The marking of hierarchical dependencies and the priority update strategy optimize the execution logic of the update sequence, reduce the risk of update conflicts, and improve the update efficiency and stability of the system.

[0118] Please see the attached Figure 1 In a preferred embodiment of the present invention, S4, performing iterative updates on the bloodline map based on the hierarchical modeling results, includes:

[0119] Incremental updates are performed in real time on core nodes. Real-time updates of core nodes ensure the synchronization of key data entities in the lineage graph, enabling timely reflection of high-frequency changes and supporting query and analysis requirements for real-time data lineage.

[0120] Real-time incremental updates for core nodes significantly improve the timeliness of the lineage graph, providing technical support for business scenarios with high real-time requirements (such as anomaly tracking and data traceability).

[0121] For edge layer nodes, incremental update operations are performed in batches according to time windows. The time window mechanism batches low-frequency update operations at the edge layer, avoiding the consumption of system resources by frequent update requests while maintaining the availability and consistency of the data lineage map.

[0122] Incremental updates of edge nodes are performed in batches according to time windows, balancing system performance and update timeliness, and reducing the interference of low-priority node updates on high-priority nodes.

[0123] During incremental updates, data dependency conflicts are detected and conflicting nodes and paths are marked. Data dependency conflicts are usually caused by asynchronous node updates or changes in incremental data dependencies. The dependency conflict detection mechanism ensures the correctness of the lineage relationship through hash checksums and version comparisons.

[0124] The data dependency conflict detection and marking mechanism avoids logical errors caused by inconsistent updates of the lineage graph, improving the integrity of data relationships and system stability;

[0125] Call the rule engine to automatically repair conflicting nodes and paths. The rule engine quickly resolves dependency conflicts through predefined rules and automated operations, ensuring the consistency and integrity of the lineage map during the update process.

[0126] The rule engine is called to automatically repair conflicting nodes and paths, which greatly reduces manual intervention, improves the automation level and maintenance efficiency of bloodline map updates, and enhances the reliability of the system.

[0127] Please see the attached Figure 1 In a preferred embodiment of the present invention, S5, optimizing the processing and storage of incremental data through distributed computing, includes:

[0128] Use a distributed graph database to store the lineage graph. By storing the nodes and edges of the lineage graph on different physical servers, the distributed graph database improves storage capacity and concurrency performance while reducing the risk of single point failure.

[0129] Using a distributed graph database to store bloodline graphs supports the storage needs of large-scale bloodline graphs while improving query efficiency and system fault tolerance.

[0130] The data lineage graph is partitioned and stored according to the node's access frequency and priority. Partitioned storage reduces access latency by storing frequently accessed core nodes in higher-performance partitions. Partitioned storage of infrequently accessed nodes improves overall storage resource utilization efficiency.

[0131] Partitioning storage based on node priority optimizes the storage structure of the lineage graph, increases access speed to high-priority nodes, and reduces the overall storage and access costs of the system.

[0132] Use a distributed computing framework to perform parallel processing on incremental data. This framework significantly improves computing efficiency by sharding large-scale incremental data and utilizing the computing power of multiple nodes to parallelly calculate lineage relationships.

[0133] The use of a distributed computing framework for parallel processing of incremental data significantly improves the computing performance of large-scale data lineage graph updates, supporting real-time and high-frequency update scenarios.

[0134] The processing results are synchronized to the graph database through the batch write interface. Batch write reduces the number of write operations to the database by merging write requests for updated data, thereby reducing write latency and improving database processing performance.

[0135] By synchronizing incremental data through the batch write interface, the update speed of the bloodline map is optimized, the stability and consistency of the update operation are ensured, and the writing requirements in high-concurrency scenarios are met.

[0136] Please see the attached Figure 1 In a preferred embodiment of the present invention, S6, using stream processing technology to achieve real-time bloodline map update includes:

[0137] The stream processing engine monitors data change events. The core of the stream processing engine is to capture real-time change events. By monitoring the change log of the data source, the changed content is transmitted to the lineage map update process in real time, achieving real-time response to data changes.

[0138] By monitoring data change events through the stream processing engine, data changes can be captured with millisecond latency, meeting the needs of high-real-time scenarios for dynamic updates of data lineage maps;

[0139] Parse the captured event stream into incremental update requests. By parsing the event stream, convert low-level log change information into high-level lineage graph update instructions, ensuring that the data lineage graph remains synchronized with the actual data status.

[0140] Capturing event streams and parsing them into incremental update requests enables standardized and modular processing of data lineage graph updates, improving the scalability and compatibility of the system.

[0141] Analyze data dependencies in real-time event streams and generate incremental relationships. This dependency analysis ensures the integrity and correctness of the data lineage graph. By dynamically analyzing upstream and downstream node relationships, it resolves the problem of dependency link breakage caused by data changes.

[0142] Real-time analysis of dependency relationships in event streams to generate incremental relationships effectively ensures the logical consistency of the lineage graph and provides technical support for lineage maintenance in dynamic data environments.

[0143] The kinship graph in the graph database is updated in real time based on the analysis results. The real-time update strategy based on the analysis results ensures the dynamic response of the kinship graph to the event stream, and the transaction management mechanism enhances the reliability of incremental updates.

[0144] The bloodline graph in the graph database is updated in real time based on the analysis results, which improves the efficiency of dynamic maintenance of the graph and supports real-time query and analysis needs in high-frequency change scenarios.

[0145] Please see the attached Figure 1 In a preferred embodiment of the present invention, S7, predicting changes in bloodline pathways using a machine learning model, includes:

[0146] Extract the characteristics of historical lineage change data, including access frequency, dependency relationships, and change frequency. By extracting the node's access behavior, dependency structure, and historical change records, a feature vector is constructed to comprehensively describe the node's dynamic behavior, providing high-quality input data for machine learning models.

[0147] Extracting the characteristics of historical bloodline change data provides detailed input for predicting the changing trends of bloodline paths, ensuring the accuracy and interpretability of the prediction results;

[0148] Use a time series model to predict data lineage change paths. The LSTM model can capture long-term and short-term dependencies in time series features. By analyzing historical change trends, it predicts future change paths, improving the accuracy and timeliness of prediction results.

[0149] Using time series models to predict data lineage change paths effectively improves the system's ability to perceive future changes and provides a scientific basis for proactively adjusting update strategies.

[0150] Generate priority incremental data requests based on prediction results. By combining prediction probability and node priority, dynamically adjust the execution order of incremental update tasks to ensure that nodes with high influence or high change probability are processed first, optimizing system resource allocation.

[0151] Prioritized incremental data requests are generated based on the prediction results, enabling intelligent sorting of update tasks and significantly improving the efficiency of incremental updates and the timeliness of important nodes.

[0152] Please see the attached Figure 1 In a preferred embodiment of the present invention, S8, detecting and repairing abnormal nodes includes:

[0153] Use anomaly detection algorithms to identify abnormal nodes in the bloodline graph. The anomaly detection algorithm analyzes the structure and attribute characteristics of the nodes in the bloodline graph to identify abnormal nodes that may cause data relationship errors or invalidation.

[0154] Anomaly detection algorithms are used to automatically identify abnormal nodes in the lineage graph, improving the efficiency of problem discovery and reducing the time and cost of manual troubleshooting.

[0155] Generate repair strategies for detected abnormal nodes. Based on the abnormal characteristics of the nodes and predefined rules, the repair strategies generate automated repair instructions to ensure that the problem repair is targeted and complete.

[0156] Generating repair strategies provides standardized exception repair solutions, significantly reducing the complexity of manual intervention while improving the accuracy and efficiency of repairs.

[0157] Execute the repair strategy to rebuild the dependencies of abnormal nodes. The execution of the repair strategy restores the logical integrity and availability of the graph by dynamically adjusting the node attributes and dependencies of the lineage graph.

[0158] Executing the repair strategy enables automated anomaly repair operations, ensuring the integrity and correctness of the lineage map and preventing abnormal nodes from affecting data flow analysis.

[0159] Record repair operation logs for subsequent audits. Repair logs provide transparency and facilitate system maintenance personnel to analyze abnormal causes and verify the correctness of repair results.

[0160] Recording repair operation logs provides data support for system operation and maintenance and auditing, which helps to further optimize anomaly detection and repair strategies.

[0161] Please see the attached Figure 1 In a preferred embodiment of the present invention, S9, the incremental data capture method includes:

[0162] For large-scale distributed data sources, incremental data is captured in parallel by data sharding. This sharding and parallel capture method for large-scale distributed data sources improves the speed and scalability of data capture and supports real-time extraction of incremental data in massive data scenarios.

[0163] During the capture process, the consistency of cross-system data sources is checked to identify possible synchronization errors. Cross-system consistency detection quickly identifies shard data anomalies caused by synchronization delays, task interruptions, or configuration errors by comparing data snapshots and verification.

[0164] During the capture process, the consistency of cross-system data sources is detected, and possible synchronization errors are identified in a timely manner, ensuring the accuracy and integrity of incremental data.

[0165] For identified erroneous data, fault-tolerance processing is performed, including retry mechanisms or data compensation strategies. The retry mechanisms and data compensation strategies ensure the reliability and integrity of data capture tasks by dynamically recovering and retrieving lost data shards.

[0166] The captured incremental data is marked as traceable batch tasks for subsequent iterative updates. The batch task marking mechanism associates the incremental data with the task metadata, achieving traceability of the incremental data and transparent management of the task status.

[0167] Marking the captured incremental data as traceable batch tasks simplifies the subsequent incremental data management process and improves the transparency and operability of data processing.

[0168] In order to better understand the present invention, the above contents are described in detail below in conjunction with specific embodiments.

[0169] Example 1: Iterative update method of bloodline graph without using stream processing technology

[0170] Specific implementation

[0171] Use batch processing to capture incremental data and identify data changes by regularly generating data snapshots for comparison.

[0172] Graph updates are performed in batch mode, and a full topological sort is performed each time to ensure the correctness of dependencies.

[0173] Data anomaly nodes are completed through manual analysis and manual repair.

[0174] Effect

[0175] It is suitable for static data environments or low-frequency data update scenarios, but the update delay is high and it is difficult to support dynamic data environments with high real-time requirements.

[0176] Lack of automated support for anomaly detection and repair leads to high maintenance costs.

[0177] Example 2: Real-time bloodline graph update method using stream processing technology

[0178] Specific implementation

[0179] Use stream processing technology (such as Apache Kafka and Apache Flink) to capture data change events, real-time parse incremental data and update bloodline map.

[0180] Optimize the update sequence of the map by hierarchical modeling, real-time update of core layer nodes, and batch update of edge layer nodes according to time windows.

[0181] Automatically detect and mark abnormal nodes during incremental update, and manually repair abnormal paths.

[0182] Effects

[0183] Improves real-time performance and can quickly respond to high-frequency data change scenarios.

[0184] The automatic anomaly detection part improves problem identification efficiency, but repair still relies on manual intervention, resulting in maintenance costs that cannot be further reduced.

[0185] Embodiment 3: Intelligent bloodline map update method combining machine learning and automatic repair mechanism

[0186] Specific implementation

[0187] Combine the stream processing technology of embodiment 2, predict data bloodline path changes through a machine learning model, dynamically adjust the update sequence, and update critical nodes first.

[0188] Call the rule engine to automatically repair the detected abnormal nodes, and complete the repair operation through dependency reconstruction and attribute adjustment.

[0189] Record repair operation logs for subsequent auditing and system optimization.

[0190] Effects

[0191] Achieves efficient real-time update, while reducing maintenance costs through path prediction and automatic anomaly repair mechanisms.

[0192] Has higher intelligent level and adaptive ability, suitable for high dynamicity, multi-source heterogeneous data scenarios.

[0193] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. An iterative updating method for a data lineage graph, characterized in that: The following steps are involved: Capture data increments and metadata changes; Construct a directed acyclic graph of data lineage relationships; Perform hierarchical modeling on the data lineage relationship graph; Iteratively update the bloodline map based on the hierarchical modeling results; Optimize the processing and storage of incremental data through distributed computing; Use stream processing technology to achieve real-time updates of bloodline graphs; Use machine learning models to predict changes in lineage paths and detect and repair abnormal nodes; The hierarchical modeling of the data lineage relationship graph includes: Divide the nodes in the graph into levels through topological sorting; Calculate node priority based on node access frequency and change frequency; Nodes with higher priority are divided into the core layer, and nodes with lower priority are divided into the edge layer; Mark the dependencies of nodes at each layer to ensure priority execution by layer during iterative updates; The hierarchical modeling results are used to iteratively update the bloodline map, including: For core layer nodes, incremental update operations are performed in real time; For edge layer nodes, incremental update operations are performed in batches according to time windows; Detect data dependency conflicts during incremental updates and mark conflicting nodes and paths; Call the rule engine to automatically repair conflicting nodes and paths; Optimizing the processing and storage of incremental data through distributed computing includes: Use a distributed graph database to store bloodline graphs; Partition and store the data lineage graph according to the access frequency and priority of the nodes; Use a distributed computing framework to perform parallel processing on incremental data; Synchronize the processing results to the graph database through the batch write interface; The method of using stream processing technology to implement real-time kinship graph updates includes: Monitor data change events through the stream processing engine; Parse the captured event stream into incremental update requests; Parse data dependencies in real-time event streams and generate incremental relationships; Update the bloodline graph in the graph database in real time based on the analysis results; The use of a machine learning model to predict changes in bloodline pathways includes: Extract the characteristics of historical lineage change data, including access frequency, dependency, and change frequency; Use time series models to predict data lineage change paths; Generates priority update incremental data requests based on prediction results.

2. The iterative update method for data lineage graph according to claim 1, characterized in that: The capture of data increments and metadata changes includes: Get data snapshots, including new and old snapshots of data; Identify incremental data by comparing snapshots; Perform timestamp filtering, hash value verification, and version comparison on incremental data to extract new, changed, and deleted data; Capture metadata changes, including additions, modifications, and deletions at the table and field levels; Normalize the recognition results into incremental data event format for subsequent processing.

3. The iterative updating method for data lineage graph according to claim 1, characterized in that: The directed acyclic graph for constructing data lineage relationships includes: Define data nodes, including table-level and field-level data entities; Define data flow relationships, including input and output relationships; Build a graph of node sets and edge sets based on data dependencies; Ensure that the constructed graph structure satisfies the acyclic property.

4. The iterative updating method for data lineage graph according to claim 1, characterized in that: The detecting and repairing of abnormal nodes includes: Use anomaly detection algorithms to identify abnormal nodes in the lineage graph; Generate repair strategies for detected abnormal nodes; Execute the repair strategy and rebuild the dependency relationships of abnormal nodes; Record the repair operation log for subsequent audit.

5. The iterative update method for data lineage graph according to claim 2, characterized in that: The incremental data capture method includes: For large-scale distributed data sources, incremental data is captured in parallel by data sharding; Check the consistency of data sources across systems during the capture process and identify possible synchronization errors; Implement fault-tolerance processing for identified erroneous data, including retry mechanisms or data compensation strategies; The captured incremental data is marked as traceable batch tasks for subsequent iterative updates.

Images