Big data-based data bloodline tracking and data asset management system

Abstract

This invention discloses a data lineage tracking and data asset management system based on big data, relating to the fields of big data governance and data asset management. The invention employs a data lineage graph construction module to collect data asset metadata and construct a data lineage graph containing nodes from the field level to the API level. A digital twin modeling module constructs a three-layer digital twin model based on the graph, consisting of a static structure layer, a dynamic operation layer, and a rule constraint layer. A change simulation module receives change plans and uses discrete event simulation to propagate the impact of changes from upstream to downstream along the lineage graph. It generates a list of affected assets and risk scores using a risk scoring formula. A solution evaluation and generation module calculates a comprehensive score for multiple What-if solutions and automatically generates rollback and data compensation solutions. A visualization and adaptive optimization module displays a risk heatmap and compares the solution results, updating simulation parameters online using gradient descent based on actual change feedback.

Description

Technical Field

[0001] This invention relates to the field of big data governance and data asset management technology, specifically to a data lineage tracing and data asset management system based on big data. Background Technology

[0002] In the data platforms of large internet companies, data assets exist in the form of tables, fields, tasks, reports, and APIs, numbering in the tens of thousands. The data processing chain spans multiple systems and teams. For example, user behavior logs are processed by multiple ETL tasks to generate a wide user profile table. This wide table is used simultaneously by dozens of downstream reports and recommendation models, with complex and frequently changing field-level dependencies. When the upstream table structure changes, the calculation logic is adjusted, data is delayed, or data goes offline, the impact of the change propagates cascading along the lineage. Failure to properly assess any link can lead to incorrect downstream report results, invalid model features, or abnormal API services, directly affecting business decisions and user experience. Due to the large scale of data assets and the dynamic changes in dependencies, change management faces a high degree of uncertainty.

[0003] Traditional data lineage tools primarily provide static dependency visualizations, failing to simulate the propagation of changes within the operational environment. Impact assessments still rely on manual review of lineage diagrams and experience-based judgment, resulting in assessments lacking quantifiable risk scoring and easily overlooking hidden cross-system dependencies. Furthermore, existing technologies lack the ability to compare multiple "what-if" solutions, making it difficult to select the optimal solution from various change strategies. Rollback scripts and data compensation tasks require manual writing, which is time-consuming and error-prone. In addition, simulation model parameters often use fixed rules, failing to automatically adjust based on actual change results. This leads to a cumulative discrepancy between the digital twin and the real production environment over time, gradually decreasing predictive accuracy. These shortcomings mean that data changes remain primarily reactive, resulting in persistently high incident rates and business impacts.

[0004] In summary, existing technologies cannot meet the needs for pre-emptive simulation, quantitative assessment, and continuous optimization of the impact of large-scale data asset changes. There is a need for a data asset digital twin change impact simulation system based on data lineage, which can simulate the impact, quantify risks, select the best support solutions, and adaptively optimize simulation parameters before changes occur, so as to reduce the accident rate and business impact caused by changes in the data production environment. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a data lineage tracking and data asset management system based on big data. It can collect data asset metadata and construct a data lineage graph containing nodes from the field level to the API level through a data lineage graph construction module. A digital twin modeling module constructs a three-layer digital twin model based on the graph, consisting of a static structure layer, a dynamic operation layer, and a rule constraint layer. A change simulation module receives change plans and uses discrete event simulation to propagate the impact of changes from upstream to downstream along the lineage graph. It generates a list of affected assets and risk scores using a risk scoring formula. A solution evaluation and generation module calculates a comprehensive score for multiple What-if solutions and automatically generates rollback and data compensation solutions. A visualization and adaptive optimization module displays a risk heatmap and compares the solution results, and updates simulation parameters online using gradient descent based on actual change feedback. This achieves pre-change simulation, risk quantification, solution selection, and continuous optimization, reducing the accident rate caused by changes in the data production environment.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a data lineage tracking and data asset management system based on big data, which includes the following components: Data lineage graph construction module: used to collect data asset metadata and construct a data lineage graph, which includes field-level, table-level, task-level, report-level and API-level nodes and dependency edges between nodes; Digital twin modeling module: used to construct a digital twin model of data assets based on the data lineage graph. The digital twin model mirrors the static structure and dynamic running state of fields, tables, tasks and dependencies. Change simulation module: used to receive change plans and simulate the execution of the change plans in the digital twin model, propagate the impact of the change from upstream to downstream along the lineage graph, and generate a list of affected assets and a risk score for each affected asset; The scheme evaluation and generation module is used to compare and analyze multiple simulation schemes, generate different simulation schemes by adjusting and changing parameters, and output the risk score and list of affected assets for each scheme. It also generates rollback schemes and data compensation schemes based on the selected simulation scheme. The visualization and adaptive optimization module is used to display the list of affected assets, risk scores, and scheme comparison results, and to adjust simulation parameters based on the deviation between the actual change results and the simulation prediction results.

[0007] Furthermore, the data lineage graph construction module extracts metadata from the data warehouse, scheduling system, BI reporting tools, and API gateway through a unified metadata collection interface. The extraction scope includes table structure, field comments, task scheduling configuration, report query SQL statements and data source configuration, and API interface definitions. The extracted metadata is standardized and stored in the metadata warehouse. A hybrid parsing engine is used to construct the data lineage graph. The hybrid parsing engine consists of a deterministic syntax parser and a code semantic parser. The deterministic syntax parser performs field-level lineage extraction on standard SQL statements and ETL component logic. The code semantic parser performs semantic analysis on stored procedures, dynamic SQL, and black-box ETL scripts through abstract syntax tree parsing and rule matching, generates candidate lineage relationships, and labels their confidence levels. The deterministic parsing results are fused with the candidate lineage relationships to form a complete data lineage graph. The nodes of the data lineage graph include field nodes, table nodes, task nodes, report nodes, and API nodes. The edges between nodes include field-to-field, table-to-task, and task-to-report dependency edges. Each edge records the lineage type, parsing source, confidence level, and update timestamp.

[0008] Furthermore, the digital twin modeling module constructs a three-layer digital twin model consisting of a static structure layer, a dynamic operation layer, and a rule constraint layer. The static structure layer stores metadata and lineage dependencies for fields, tables, and tasks. In the dynamic operation layer, task execution status is collected at 1-second intervals by acquisition probes deployed in the data task. Data latency, data quality score, and SLA compliance status are written when the task completion event is triggered. The rule constraint layer defines data quality rules, dependency strength thresholds, and risk weight parameters. The digital twin model adopts an incremental update mechanism, triggering local reconstruction only at the changed nodes when metadata or execution status changes, avoiding the performance loss caused by full reconstruction. Task execution status includes CPU utilization, memory usage, input / output throughput, and task execution duration. Data latency is the difference between the actual output time and the planned output time. Data quality score is calculated based on the data quality rule verification results. SLA compliance status is a Boolean value indicating whether the actual completion time meets the agreed SLA time. The data is written into the digital twin model after the task is completed.

[0009] Furthermore, when the change simulation module receives a change plan, it provides change type templates for the user to select. Change types include data structure changes, computational logic adjustments, data delays, and data shutdowns. The data structure change template requires the user to fill in the changed field name, the data type before the change, the data type after the change, and whether data loss is allowed. The computational logic adjustment template requires the user to upload the SQL logic or script files before and after the change. The data delay template requires the user to fill in the delay duration and the start time of the impact. The data shutdown template requires the user to fill in the shutdown object identifier and the shutdown time. The change plan also includes the change time window, the person responsible for the change, and the change priority. The system matches the field node identifier in the metadata repository according to the changed field name, identifies the modified operator node through the difference comparison of the abstract syntax tree of the SQL statement, and parses the change plan into a structured change event containing the change node identifier, change parameters, and change time, which serves as the trigger source for digital twin simulation.

[0010] Furthermore, the change simulation module employs a discrete event simulation engine, using change events as trigger sources to propagate the impact of changes downstream along the dependency edges of the lineage graph. The propagation calculation rules are as follows: when an upstream node changes, the impact dimensions of the change type on downstream nodes are determined. These impact dimensions include field structure compatibility, data timeliness delay, data quality degradation, and downstream task failure probability. Field structure compatibility is determined based on the data types and mapping relationships of fields before and after the change. If the input fields of downstream nodes do not exist after the change and have no default values, it is determined to be incompatible and triggers downstream task failure. If the data type changes but an automatic conversion rule exists, it is determined to be compatible and the conversion cost is recorded. Data timeliness delay is the delay duration of the upstream task along the dependency edges. The latency of the downstream task itself is added after accumulation; the data quality degradation is the change in the upstream data quality score multiplied by the downstream's dependence weight on the upstream; the failure probability of the downstream task is calculated using a logistic regression model. The training method of this logistic regression model is as follows: using the field structure incompatibility flag, data quality degradation, and historical change failure rate in historical change events as input features, and using whether the change caused the downstream task to fail as the label, collecting no less than 200 historical change records to form a training sample set, and using maximum likelihood estimation to train the model parameters. After training, the model outputs a probability value in the range of 0-1; the impact result of each downstream node is recorded and continues to propagate downstream until it propagates to the terminal node in the lineage graph with an out-degree of zero and a node type of report or API.

[0011] Furthermore, the change simulation module generates a risk score for each affected asset using the following formula: ,in To define the scope of impact, the value is the proportion of the number of affected assets to the total number of assets. ∈[0,1]; The asset criticality is calculated by dividing the sum of the asset's out-degree and in-degree in the lineage graph by the maximum node degree, and then multiplying this by the business importance weight. The business importance weight is configured by the user and ranges from 0 to 10. ∈[0,10]; The data quality sensitivity is determined by multiplying the number of data quality rules associated with the asset by the sum of the rule severity weights, dividing by the maximum number of rules for a single asset in the system, and then normalizing to [0,1]. If the maximum number of rules for a single asset in the system is less than 1, it is treated as 1. The SLA urgency level is determined by the ratio of the remaining SLA time to the total agreed SLA duration. ∈[0,1]; The historical change failure rate is calculated by dividing the number of failures caused by changes to the asset within the past 30 days by the total number of changes. If the total number of changes is 0, then... Set to 0; , , , , , These are the weighting coefficients. , , , , , , , The weight coefficients were obtained by normalizing historical fault samples after ranking them by feature importance using a random forest. Historical fault samples consist of actual fault records after each real change, labeled as whether a fault occurred, and the features are... , , , , Corresponding historical values and These are empirical constants; The output range is [0,1], and the higher the risk score, the greater the risk of change.

[0012] Furthermore, the scheme evaluation and generation module calculates a comprehensive score for each simulation scheme. The formula is: ,in This represents the average risk score of all affected assets under this plan. ∈[0,1); For the standard deviation of the risk score, ≥0; To assess the feasibility of rolling back the proposed solution, =0.4×Rollback script availability + 0.3×Data snapshot integrity + 0.3×Dependency task pause / recovery strategy completeness. Each sub-item has a value of 0-1. Rollback script availability is determined by whether the rollback script covers all affected tables. Data snapshot integrity is determined by whether the snapshot contains data integrity verification information before the change. Dependency task pause / recovery strategy completeness is determined by whether pause and recovery instructions for all dependent tasks have been generated. This is the normalized value of the total execution time of the plan. ,in The total execution time of the plan is expressed in hours. This is a preset reference duration; To calculate the normalized value of the resource costs required for the solution, ,in Calculate the resource costs for the proposed solution. To pre-set a reference cost; the costs of each scheme will be... After the values ​​are calculated, a comparison matrix is ​​output. The comparison matrix is ​​then sorted according to... The values ​​are sorted in descending order, and the advantages of each solution are marked. These advantages include the lowest risk, fastest execution, lowest cost, or most convenient rollback.

[0013] Furthermore, when the scheme evaluation and generation module generates a rollback scheme, it automatically generates a data snapshot recovery script based on the metadata snapshot before the change. The script includes structure rollback statements and data recovery statements for the changed tables. The structure rollback statements are automatically generated based on the table structure definition before the change, and the data recovery statements are generated through database flashback technology or snapshot data import. A task rollback version identifier is generated for each affected task and recorded in the rollback list. Task pause and recovery instructions are generated according to the lineage dependency order based on the task pause and recovery strategy. When generating a data compensation scheme, the module compares the data writing times in the task execution logs before and after the change. The system identifies missing data intervals based on the interstamp range and automatically generates data recalculation tasks based on these intervals. The scheduling time window for the recalculation task is calculated backwards from the downstream SLA requirements. The backwards calculation formula is: Recalculation window = Downstream SLA deadline - Estimated execution time of the recalculation task - Safety buffer time. The safety buffer time is determined by multiplying the historical execution time volatility by the average execution time. The downstream cache refresh strategy generates cache invalidation instructions or proactive refresh instructions based on the affected report and API caching mechanisms. The rollback plan and data compensation plan are packaged into an executable change plan, which includes the execution order, dependency checkpoints, and rollback trigger conditions.

[0014] Furthermore, after the change simulation module and the scheme evaluation and generation module are executed, the visualization and adaptive optimization module maps the list of affected assets and risk scores into a risk heatmap. The risk heatmap uses segmented color mapping: nodes with risk scores below 0.3 are displayed in green, nodes with risk scores between 0.3 and 0.7 are displayed in yellow, and nodes with risk scores above 0.7 are displayed in red. The color transition uses RGB linear interpolation. The impact propagation path is displayed in an animated manner along the edge of the lineage graph, highlighting it level by level. Next to each affected node, the risk score and key impact dimensions are displayed. The key impact dimensions are the top two impact dimensions that lead to the highest risk scores, including field structure incompatibility, data latency, data quality degradation, or task failure probability. The What-if scheme comparison results are displayed in the form of a comparison matrix and a radar chart. The rows of the comparison matrix represent different schemes, and the columns represent evaluation indicators, including average risk, execution time, resource cost, and rollback feasibility.

[0015] Furthermore, the change simulation module in the visualization and adaptive optimization module receives feedback on the actual change results after each real change is executed. It compares the actual list of faulty assets, the actual fault duration, and the actual data quality degradation with the simulation prediction results. The simulation deviation is calculated as: |Actual number of faulty assets - Predicted number of faulty assets| / Actual number of faulty assets + |Actual fault duration - Predicted fault duration| / Actual fault duration + |Actual quality degradation - Predicted quality degradation| / Actual quality degradation, with a preset threshold of 0.2. When the simulation deviation exceeds the preset threshold for three consecutive independent change events, adaptive adjustment of the simulation parameters is triggered. The adjustment targets include the risk score weight coefficient. , , , , , and the propagation computation rules and Parameters; the adjustment method uses online gradient descent, and the loss function is... ,in For the first The simulation deviation is calculated after the actual change is executed. To determine the number of historical change events involved in parameter updates, take the partial derivative of the weight coefficients and update them. and The parameters are reselected each time they are triggered by a grid search. The updated parameters are backtested using the last 5 historical change data. If the adjusted parameters make the average simulation deviation lower than the original parameters, they are deemed qualified and written into the digital twin model. Unverified parameters are not effective, thus ensuring that the digital twin model continuously approximates the real data production environment.

[0016] Compared with existing technologies, this big data-based data lineage tracking and data asset management system has the following beneficial effects: I. This invention constructs a digital twin model of data assets, and performs discrete event simulation on upstream data structure changes, computational logic adjustments, data delays or offline events in the digital twin environment. It automatically propagates the impact of changes along the lineage graph and generates a list of affected assets and a quantitative risk score. This achieves an upgrade of the operation and maintenance mode from "firefighting after changes" to "simulation before changes", avoiding omissions in manual investigation and hidden cross-system dependency risks, and significantly reducing the accident rate and business impact caused by changes in the data production environment.

[0017] Second, this invention uses a What-if multi-scheme comparison mechanism to calculate a comprehensive score for multiple change strategies and automatically generate rollback and data compensation schemes, providing quantifiable and executable pre-plans for change decisions. At the same time, the adaptive optimization module updates the risk score weight coefficients and propagation calculation parameters online using online gradient descent based on the actual fault asset list, fault duration, and quality degradation after the actual change is implemented, so that the digital twin model continuously approximates the real data production environment, improving the long-term accuracy of change impact prediction and system reliability.

[0018] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 This is a flowchart of the overall process of a data lineage tracking and data asset management system based on big data. Figure 2 A flowchart illustrating the simulation propagation of discrete events affecting changes in a data lineage tracing and data asset management system based on big data. Figure 3 A flowchart for the comprehensive evaluation and rollback compensation of a data lineage tracking and data asset management system based on big data. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0022] In business scenarios where enterprise big data warehouse platforms undergo iterative changes to data table structures, this embodiment discloses a specific implementation method for a big data-based data lineage tracking and data asset management system.

[0023] This embodiment is applied to the daily iterative operation and maintenance scenario of an offline data warehouse of a large enterprise. The enterprise's data warehouse contains hundreds of business data tables, hundreds of ETL scheduling tasks, dozens of business BI reports, and data API interfaces that provide services to the outside world. Daily operations frequently involve table structure changes such as adding data table fields and modifying field types. Existing technologies rely solely on operations personnel manually identifying upstream and downstream dependent assets when performing table structure changes. This is highly prone to overlooking affected downstream tasks, reports, and APIs, leading to production failures such as online report errors, abnormal data returns from external APIs, and failure to meet business SLAs. Furthermore, it lacks the ability to perform risk pre-simulation before changes, automatically generate rollback and data compensation contingency plans, and has a long handling cycle after change failures occur. This embodiment uses a big data-based data lineage tracing and data asset management system to complete the full-process simulation, risk assessment, and contingency plan output for table structure changes.

[0024] After system deployment, the initialization build process is executed first. The data lineage graph construction module calls a unified metadata collection interface, connecting to four types of components: data warehouse, scheduling system, BI reporting tools, and API gateway, to extract metadata in batches. The extracted metadata includes raw information such as data table metadata, field attributes, ETL task scripts, scheduling dependencies, report field mapping configurations, and API input / output parameter field mapping relationships. Internally, the module performs standardization processing on the raw metadata, unifying metadata field identifiers and object type encodings, and persistently stores the standardized metadata in the metadata repository.

[0025] The hybrid parsing engine works collaboratively internally. The deterministic parser specifically parses standard SQL statements and business logic generated by drag-and-drop visual ETL components, directly extracting field-level lineage dependencies. This parsing result is marked with 100% confidence. The code semantic parser handles stored procedures, dynamic SQL, and black-box ETL scripts, converting the scripts into an abstract syntax tree (AST). It traverses the AST nodes to identify field reading, transformation, and output logic, and generates candidate lineage relationships based on preset business rules. Simultaneously, it sets confidence levels for candidate lineages based on the script's parsing completeness; for example, 85% confidence is set when the script can completely parse all field mappings, and 60% confidence is set when some branch logic cannot be parsed. The system merges the deterministic parser output with the candidate lineage relationships marked with confidence. Specifically, when the deterministic parsing result and the candidate lineage relationship point to the same upstream and downstream field pairs, the one with higher confidence is used; when they point to different field pairs, the candidate lineage relationship with a confidence level higher than 50% is retained as a supplement; lineage relationships that cannot be confirmed by a single parsing source are marked for manual review. The final result is a complete data lineage graph, which contains field-level, table-level, task-level, report-level, and API-level nodes. It also records the dependency edges between nodes, with each dependency edge carrying a dependency weight. The dependency weight is set based on the frequency of downstream nodes' use of upstream fields and the importance of the business.

[0026] The digital twin modeling module builds a three-layer architecture for the data asset digital twin model based on the aforementioned data lineage graph. The static structure layer fully mirrors the metadata of all fields, data tables, and task objects, as well as all lineage dependency relationships. The dynamic execution layer deploys data collection probes on each data task execution node, collecting task execution status indicators at 1-second intervals. These indicators include CPU utilization, memory usage, input / output throughput, and task execution time. When each data task completes and triggers a task completion event, the system writes data latency, data quality score, and SLA compliance status to the dynamic execution layer. Data latency is defined as the difference between the actual task output time and the planned output time. The data quality score is calculated from the results of data quality rule verification; for example, passing all integrity, uniqueness, and validity checks earns 100 points, and failing any check results in a deduction based on preset criteria. The SLA compliance status is a Boolean flag: true if the actual task completion time is earlier than or equal to the agreed SLA time, and false otherwise. The rule constraint layer stores pre-configured data quality rules, dependency strength thresholds, and risk weight parameters. The dependency strength threshold is set based on the tightness of business coupling between business objects. If more than 70% of the input fields of a downstream task originate from the same upstream node, it is considered a strong dependency. The digital twin model enables an incremental update mechanism. When metadata is modified or task running status indicators change, only the changed node and its directly related neighbor nodes are partially reconstructed, without a full model recalculation, reducing system computational overhead.

[0027] The operations and maintenance personnel submit a change plan. This change pertains to a data structure change, modifying the user age field in the user information table of the business data table. Previously, the field's data type was tinyint; after the change, it's int. Data loss is not permitted. The system invokes the change simulation module, which retrieves a data structure change type template. The operations and maintenance personnel fill in the template with the changed field name, the original and new data types, whether data loss is permissible, and also add information such as the change time window, the person responsible for the change, and the change priority, forming a complete change plan. The system matches and retrieves the corresponding field node identifier from the metadata repository, parsing the change plan into a structured change event. This change event includes the change node identifier, change parameters, and change time. This structured change event serves as the simulation trigger source for the discrete event simulation engine.

[0028] After receiving a change event, the discrete event simulation engine calculates the propagation of the change's impact level by level from upstream to downstream nodes along the dependency edges of the lineage graph. This embodiment sets four impact dimensions: field structure compatibility, data timeliness delay, data quality degradation, and downstream task failure probability. The field structure compatibility judgment logic compares the data types of the fields before and after the change, as well as the mapping relationship between upstream and downstream fields. In this case, the conversion from tinyint to int has an automatic database conversion rule, so the judgment result is compatible, and the conversion cost corresponding to this conversion is recorded. If a downstream input field disappears directly after the change and no default value is configured, it is directly judged as incompatible, and the downstream task is marked as having a failure risk. The data timeliness delay is calculated by accumulating the upstream task delay along the dependency edges, and then adding the inherent delay of the downstream task itself. The data quality degradation is equal to the change in the upstream data quality score multiplied by the downstream's dependency weight configured on the dependency edge. The downstream task failure probability input features include historical change failure rate, field structure incompatibility flag, and data quality degradation, which are input to the trained logistic regression model, outputting a task failure probability value in the range of 0-1. After each downstream node's impact calculation is completed, the node's impact result is saved and continued to be passed downstream until it reaches the terminal nodes of report and API types.

[0029] The simulation process calls the risk scoring formula to calculate the risk score for each affected asset. The weighting coefficients within the formula , , , , , and empirical constants , Pre-configured in the rule constraint layer of the digital twin model; weight coefficients are set based on the contribution ratio of various risk factors to business failures, using empirical constants. , The settings should be based on the distribution range of the business assets. The value represents the scope of impact and is taken as the total number of all assets accessible downstream of the change node. The criticality of an asset is determined by assigning a graded value based on whether the asset is used for external business and the frequency of calls to core business systems. This represents data quality sensitivity, set according to the business's tolerance for data errors. This represents the urgency level of the SLA, calculated from the remaining allowable timeout time stipulated in the SLA; This represents the historical change failure rate of the asset, taken from the historical change failure statistics recorded in the metadata repository. A list of affected assets is output after all affected assets have completed risk scoring calculations.

[0030] The scheme evaluation and generation module can adjust and change parameters to generate multiple different simulation schemes. For each simulation scheme, a comprehensive score is calculated using a comprehensive scoring formula. . This is the average of the risk scores for all affected assets in the plan. The standard deviation of the risk score; Stab represents the feasibility score of the rollback plan, with a value range of 0-100. The higher the value, the easier it is to perform the rollback operation. The value after normalizing the total execution time of the plan; The normalized values ​​are used to represent the computational resource costs required for the proposed solution. Overall score. The higher the score, the better the overall effect of the change plan. After the operations and maintenance personnel select the target simulation plan based on the comprehensive score of multiple plans, the system automatically generates the rollback plan and the data compensation plan.

[0031] Rollback plan generation logic: Retrieve a metadata snapshot before the change occurs, automatically generate table structure rollback statements, and simultaneously generate data recovery statements using database flashback technology; generate a task rollback version identifier for each affected task and write it to the rollback list; generate task pause and resume instructions according to dependency lineage to ensure correct dependency order during rollback. Data compensation plan: Compare the timestamps of task execution logs before and after the change to identify the data missing time interval, automatically generate data recalculation tasks based on the missing interval, and determine the scheduling time window of the recalculation task by working backward from the downstream SLA requirements; for affected reports and APIs, generate cache invalidation instructions or active refresh instructions based on their caching mechanisms. The system packages the rollback plan and data compensation plan into an executable change plan, specifying the execution order, dependency checkpoints, and rollback trigger conditions. When the rollback trigger conditions are triggered during the change execution process, the plan can be executed directly.

[0032] The visualization and adaptive optimization module visualizes the simulation results, mapping affected assets and their corresponding risk scores to generate a risk heatmap using segmented color mapping, such as red for high risk, yellow for medium risk, and green for low risk. The propagation path of change impacts is animated and highlighted level by level along the dependency edges of the lineage graph, with risk scores and the two key impact dimensions with the highest contribution displayed next to each affected node. Multiple What-if simulation schemes are displayed using a comparison matrix and radar chart, with rows of the comparison matrix corresponding to different simulation schemes and columns including evaluation indicators such as average risk, execution time, resource cost, and rollback feasibility.

[0033] Once the actual change operation is completed, the system receives feedback on the actual change results and compares the actual list of faulty assets, actual fault duration, and actual data quality degradation with the simulation prediction results. When the simulation deviation of three consecutive independent change events exceeds a preset threshold, adaptive adjustment of simulation parameters is triggered. The preset threshold is set based on the business-acceptable level of simulation error, and the adjustment includes the various weight coefficients of the risk score. , , , , , and empirical constants , The weights are updated using an online gradient descent algorithm, and the loss function is... , Representing the Simulation deviation of the second change. The number of historical change events that participated in the update; , A grid search method was used to reselect the parameters. After the parameters were updated, the data from the last 5 historical changes were retrieved for backtesting. Only when the average simulation deviation after backtesting was lower than the average simulation deviation of the original parameters would the updated parameters be written into the digital twin model and officially take effect. Parameters that failed the backtesting were discarded and not included in subsequent simulation calculations.

[0034] In summary, this embodiment completes multi-source metadata collection and full-level lineage analysis through a data lineage graph construction module, builds a three-layer digital twin model of data assets, parses change plans into structured change events, and uses a discrete event simulation engine to complete the propagation of upstream and downstream impacts along the lineage chain. It also calculates risk scores for affected assets, comprehensively evaluates multiple simulation schemes, automatically outputs executable rollback and data compensation plans, visualizes risks and propagation paths, and adaptively iterates simulation parameters based on real change feedback. This embodiment fully covers the entire process of data table structure changes on big data platforms, enabling risk pre-assessment before formal change execution, outputting implementable contingency plans, and continuously optimizing the simulation model after the change. This effectively avoids omissions caused by manual review, reduces the probability of online failures caused by data asset changes on big data platforms, and ensures the stable operation of business reports and external API services.

[0035] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A data lineage tracking and data asset management system based on big data, characterized in that: The system includes the following components: Data lineage graph construction module: used to collect data asset metadata and construct a data lineage graph, which includes field-level, table-level, task-level, report-level and API-level nodes and dependency edges between nodes; Digital twin modeling module: used to construct a digital twin model of data assets based on the data lineage graph. The digital twin model mirrors the static structure and dynamic running state of fields, tables, tasks and dependencies. Change simulation module: used to receive change plans and simulate the execution of the change plans in the digital twin model, propagate the impact of the change from upstream to downstream along the lineage graph, and generate a list of affected assets and a risk score for each affected asset; The scheme evaluation and generation module is used to compare and analyze multiple simulation schemes, generate different simulation schemes by adjusting and changing parameters, and output the risk score and list of affected assets for each scheme. It also generates rollback schemes and data compensation schemes based on the selected simulation scheme. The visualization and adaptive optimization module is used to display the list of affected assets, risk scores, and scheme comparison results, and to adjust simulation parameters based on the deviation between the actual change results and the simulation prediction results.

2. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, The data lineage graph construction module extracts metadata from the data warehouse, scheduling system, BI reporting tools and API gateway through a unified metadata collection interface, and stores the extracted metadata in the metadata warehouse after standardization. A hybrid parsing engine is used to construct the data lineage graph. The hybrid parsing engine consists of a deterministic syntax parser and a code semantic parser. The deterministic syntax parser extracts field-level lineage from standard SQL statements and ETL component logic. The code semantic parser performs semantic analysis on stored procedures, dynamic SQL, and black-box ETL scripts through abstract syntax tree parsing and rule matching, generates candidate lineage relationships and labels their confidence levels. The deterministic parsing results are then fused with the candidate lineage relationships to form a complete data lineage graph.

3. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, The digital twin modeling module constructs a three-layer digital twin model consisting of a static structure layer, a dynamic operation layer, and a rule constraint layer. The static structure layer stores metadata and lineage dependencies for fields, tables, and tasks. In the dynamic operation layer, task execution status is collected at 1-second intervals by acquisition probes deployed in the data task. Data latency, data quality score, and SLA compliance status are written when the task completion event is triggered. The rule constraint layer defines data quality rules, dependency strength thresholds, and risk weight parameters. The digital twin model adopts an incremental update mechanism, triggering local reconstruction only at the changed nodes when metadata or execution status changes. Task execution status includes CPU utilization, memory usage, input / output throughput, and task execution duration. Data latency is the difference between the actual output time and the planned output time. The data quality score is calculated based on the data quality rule verification results. SLA compliance status is a Boolean value indicating whether the actual completion time meets the agreed SLA time. The data is written into the digital twin model after the task is completed.

4. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, When the change simulation module receives a change plan, it provides change type templates for the user to select. Change types include data structure changes, computational logic adjustments, data delays, and data shutdowns. The data structure change template requires the user to fill in the changed field name, the data type before the change, the data type after the change, and whether data loss is allowed. The computational logic adjustment template requires the user to upload the SQL logic or script files before and after the change. The data delay template requires the user to fill in the delay duration and the start time of the impact. The data shutdown template requires the user to fill in the shutdown object identifier and the shutdown time. The change plan also includes the change time window, the person responsible for the change, and the change priority. The system matches the field node identifier in the metadata repository based on the changed field name, identifies the modified operator node through the difference comparison of the abstract syntax tree of the SQL statement, and parses the change plan into a structured change event containing the change node identifier, change parameters, and change time, which serves as the trigger source for digital twin simulation.

5. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, The change simulation module employs a discrete event simulation engine, using change events as trigger sources to propagate the impact of changes downstream level by level along the dependency edges of the lineage graph. The propagation calculation rules are as follows: when an upstream node changes, the impact dimensions of the change type on downstream nodes are determined. These impact dimensions include field structure compatibility, data timeliness delay, data quality degradation, and downstream task failure probability. Field structure compatibility is determined based on the data types and mapping relationships of fields before and after the change. If the input fields of a downstream node do not exist after the change and have no default values, it is determined to be incompatible and triggers downstream task failure. If the data type changes but an automatic conversion rule exists, it is determined to be compatible and the conversion cost is recorded. Data timeliness delay is the sum of the upstream task delay along the dependency edges and the downstream task's own delay. Data quality degradation is the change in upstream data quality score multiplied by the downstream's dependency weight on the upstream. The failure probability of downstream tasks is calculated using a logistic regression model. The training method of this logistic regression model is as follows: the incompatibility flag of field structure in historical change events, the magnitude of data quality degradation, and the historical change failure rate are used as input features, and whether the change causes the downstream task to fail is used as the label. At least 200 historical change records are collected to form a training sample set. Maximum likelihood estimation is used to train the model parameters. After training, the model outputs a probability value in the range of 0-1. The impact result of each downstream node is recorded and continues to propagate downstream until it propagates to the terminal node in the lineage graph with an out-degree of zero and a node type of report or API.

6. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, The change simulation module generates a risk score for each affected asset using the following formula: ,in The scope of influence; For asset criticality; Sensitivity to data quality; SLA urgency level; Historical change failure rate; , , , , , These are the weighting coefficients. and This is an empirical constant.

7. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, The scheme evaluation and generation module calculates a comprehensive score for each simulation scheme. The formula is: ,in This is the average risk score of all affected assets under this plan; The standard deviation of the risk score; Assess the feasibility of rolling back the proposed solution. This is the normalized value of the total execution time of the plan; This is the normalized value of the resource cost required for the solution.

8. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, When the scheme evaluation and generation module generates a rollback scheme, it automatically generates a data snapshot recovery script based on the metadata snapshot before the change. The script includes structure rollback statements and data recovery statements for the changed table. The structure rollback statements are automatically generated based on the table structure definition before the change, and the data recovery statements are generated through database flashback technology or snapshot data import. A task rollback version identifier is generated for each affected task and recorded in the rollback list. The task pause and recovery instructions are generated according to the bloodline dependency order based on the task pause and recovery strategy. When generating a data compensation scheme, the missing data interval is identified by comparing the data write timestamp range in the task execution logs before and after the change. A data recalculation task is automatically generated based on the missing interval. The scheduling time window of the recalculation task is calculated backward based on the downstream SLA requirements. The downstream cache refresh strategy generates cache invalidation instructions or active refresh instructions based on the affected report and API caching mechanisms; the rollback plan and data compensation plan are packaged into an executable change plan, which includes the execution order, dependency checkpoints and rollback trigger conditions.

9. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, After the change simulation module and the scheme evaluation and generation module are executed, the visualization and adaptive optimization module maps the list of affected assets and risk scores into a risk heat map, which uses segmented color mapping. The impact propagation path is animated and highlighted along the edges of the lineage graph. Each affected node displays a risk score and key impact dimensions. The key impact dimensions are the top two impact dimensions that lead to the highest risk score, including field structure incompatibility, data latency, data quality degradation, or task failure probability. The What-if solution comparison results are displayed in the form of a comparison matrix and radar chart. The rows of the comparison matrix represent different solutions, and the columns represent evaluation indicators, including average risk, execution time, resource cost, and rollback feasibility.

10. The data lineage tracking and data asset management system based on big data according to claim 1, characterized in that, The visualization and adaptive optimization module's change simulation module receives feedback on the actual change results after each real change is executed, comparing the actual faulty asset list, actual fault duration, and actual data quality degradation with the simulation prediction results. When the simulation deviation exceeds a preset threshold for three consecutive independent change events, adaptive adjustment of simulation parameters is triggered, with the adjustment objects including the risk score weight coefficient. , , , , , and the propagation computation rules and Parameters; the adjustment method uses online gradient descent, and the loss function is... ,in For the first The simulation deviation is calculated after the actual change is executed. To determine the number of historical change events involved in parameter updates, take the partial derivative of the weight coefficients and update them. and The parameters are reselected each time they are triggered via grid search; the updated parameters are backtested using the most recent 5 historical change data. If the adjusted parameters make the average simulation deviation lower than the original parameters, they are deemed qualified and written into the digital twin model. Unverified parameters are not effective.

Images