A metadata-based business data bloodline tracking method, system and terminal

Abstract

The application relates to a metadata-based service data blood relationship tracking method, a system and a terminal, and belongs to the technical field of service data processing. The service data blood relationship tracking method comprises the following steps: listening to metadata change events of a cloud platform during service processing; querying a logical mapping table according to the metadata change events, and locating an affected logical ID; performing blood relationship edge deduction from a constructed blood relationship graph according to the affected logical ID; and according to a deduction result, marking blood relationship edges directly dependent on the affected logical ID as invalid, and marking downstream blood relationship edges indirectly dependent on the affected logical ID as suspicious. The application has the beneficial effects of improving data management efficiency and the timeliness of blood relationship tracking.

Description

Technical Field

[0001] This invention relates to the technical field of business data processing, and in particular to a method, system, and terminal for business data lineage tracing based on metadata. Background Technology

[0002] As digital transformation deepens, enterprise business systems are becoming increasingly complex, with data flowing across cloud platforms, distributed systems, and multi-source heterogeneous environments, forming a vast data ecosystem network. Data lineage, as a core tool describing the entire data chain from generation and processing to consumption, is becoming increasingly important—not only supporting key scenarios such as data quality monitoring, fault tracing, and compliance auditing, but also serving as the foundation for enterprises to achieve data asset management and value extraction. Especially in sectors like finance and e-commerce, where data consistency and reliability are extremely critical, lineage tracing capabilities directly impact the accuracy of business decisions and risk control capabilities.

[0003] Currently, mainstream data lineage tracing methods mainly fall into two categories: one is based on static configuration files, which extracts dependencies between tables and fields and constructs a lineage graph by parsing ETL scripts, SQL statements, or configuration files of data integration tools; the other is based on runtime log collection, which dynamically generates lineage chains by recording the input and output information of data processing tasks. For example, some tools analyze the data flow relationships between tasks by monitoring job logs of computing engines such as Spark and Flink; other technologies embed probes in the data pipeline to collect data transmission logs in real time to construct lineage relationships. These methods mostly rely on the direct parsing of physical layer metadata (such as database table names and field names), and the updates to lineage relationships usually lag behind actual data changes.

[0004] On the one hand, static configuration parsing requires manual triggering of updates, and runtime log collection has a delay of seconds to minutes, which cannot meet the real-time tracking requirements of high-frequency changes in metadata in the dynamic data environment of the cloud platform, and is prone to causing the lineage relationship to be disconnected from the actual data link. On the other hand, the lineage relationship directly built based on physical metadata is difficult to isolate the impact of underlying storage changes (such as table name modification, partition adjustment) on the business logic layer. When the upstream data structure is slightly adjusted, the downstream dependent lineage link may be misjudged as invalid as a whole, requiring manual intervention to verify a large number of invalid dependencies, resulting in low data governance efficiency. Summary of the Invention

[0005] To improve data governance efficiency and the timeliness of lineage tracing, this invention provides a business data lineage tracing method, system, and terminal based on metadata.

[0006] In a first aspect, the present invention provides a method for tracing the lineage of business data based on metadata, employing the following technical solution:

[0007] A metadata-based method for tracing the lineage of business data includes:

[0008] Monitor metadata change events during business processing on the cloud platform;

[0009] Based on the metadata change event, query the logical mapping table and locate the affected logical ID;

[0010] Based on the affected logical ID, kinship edges are deduced from the constructed kinship graph;

[0011] Based on the derivation results, the bloodline edges that are directly dependent on the affected logical ID are marked as invalid, and the downstream bloodline edges that are indirectly dependent on the affected logical ID are marked as suspicious.

[0012] By adopting the above technical solution and monitoring metadata change events on the cloud platform, changes in data structure or attributes can be captured in real time without manual triggering of updates. This avoids the delays associated with traditional post-audit methods and ensures the timeliness of lineage updates. Based on the logical mapping table, affected logical IDs are located, and dependencies are deduced using the lineage graph. This accurately identifies downstream data with direct and indirect dependencies, reducing the error rate of manual investigation. Marking direct dependency edges as "failed" and indirect dependencies as "suspicious" enables refined management of lineage status, helping users quickly differentiate risk levels and prioritize critical issues. When data anomalies occur, the root cause (such as upstream metadata change points) can be quickly located through lineage relationships, shortening the problem investigation cycle and improving the efficiency of resolving data quality issues, thereby enhancing data governance efficiency.

[0013] Optionally, the specific steps for deriving lineage edges from the constructed lineage graph based on the affected logical ID include:

[0014] From the constructed lineage graph, obtain the dependency lineage edges of the direct and indirect dependency fields associated with the affected logical ID;

[0015] The influence score of the dependent bloodline edge is calculated based on the bloodline path characteristics. The bloodline path characteristics include dependency depth weight and business weight. Dependency depth includes direct dependency and indirect dependency. The business weight is dynamically weighted based on the field business attributes.

[0016] Filter out dependent bloodline edges whose influence on the score is greater than a set score value;

[0017] Based on the historical impact on business operations, and combined with the primary impact score, the secondary impact score of the selected dependent bloodline edges is calculated;

[0018] Dependency edges whose secondary influence scores are greater than a score threshold are selected as target dependency edges; the target dependency edges include direct dependency edges and indirect dependency edges.

[0019] By employing the aforementioned technical solution and obtaining the dependency lineage edges of the direct and indirect dependent fields associated with the affected logical ID, all relevant dependencies involving that logical ID can be comprehensively identified. Both direct and multi-layered indirect dependencies are considered, helping to accurately pinpoint the potentially affected scope and avoid overlooking potential influencing factors. Calculating the primary impact score of dependency lineage edges using lineage path characteristics transforms abstract dependencies into concrete quantitative values, making the impact levels of different dependency lineage edges comparable. Filtering out dependency lineage edges with primary impact scores greater than a set value quickly eliminates dependencies with minor impact, focusing attention on key influencing factors, improving processing efficiency, and avoiding wasting time and resources on irrelevant dependencies. Based on historical business impact data, a secondary impact score is calculated using the primary impact score, taking into account past experience and actual circumstances, making the impact assessment more accurate and reliable.

[0020] Optionally, based on the derivation results, the steps following the marking of bloodline edges directly dependent on the affected logical ID as invalid include:

[0021] Get the urgency level of the business process;

[0022] Based on the urgency level, determine whether the affected logical ID needs to be repaired;

[0023] If so, then parse the problem type of the affected logical ID;

[0024] Determine the repair plan based on the type of problem described;

[0025] After repairing the affected logical IDs, update the bloodline network and record the repair information.

[0026] By adopting the above technical solution, after marking the bloodline edge that directly depends on the affected logical ID as invalid, the steps of obtaining the business urgency, judging the necessity of repair, determining the problem type and solution, updating the bloodline network and recording information can achieve precise risk control, rational resource allocation and closed-loop management of system maintenance, significantly improve the efficiency of fault handling and business continuity assurance capabilities, and realize the transformation of "fault handling from passive response to proactive prevention".

[0027] Optionally, the steps to determine the urgency of a business process include:

[0028] Obtain the scope and impact score of the core business affected by the affected logical ID;

[0029] Obtain the depth impact score of the affected logical ID on the business depth;

[0030] Obtain the urgency score for the need for repair;

[0031] The urgency of the business process is calculated based on the scope impact score, the depth impact score, and the urgency impact score.

[0032] By adopting the above technical solution, the abstract "business urgency" is transformed into a quantifiable indicator through the decomposition and calculation of three dimensions: scope impact score, depth impact score, and urgency impact score. This avoids decision-making bias caused by a single dimension judgment (such as relying solely on the business scope) and makes the urgency assessment more in line with the complex needs of actual business scenarios.

[0033] Optionally, the step of querying the logical mapping table and locating the affected logical ID based on the metadata change event includes:

[0034] Retrieve the change attributes of metadata change events;

[0035] Determine whether the changed attribute is a regular change;

[0036] If so, the standard logic mapping table is invoked, and the affected logic ID is located from the standard logic mapping table;

[0037] If not, then based on the current business scenario, a temporary logical mapping table is built, and the affected logical ID is located from the temporary logical mapping table.

[0038] By adopting the above technical solution, the core content of the change can be directly identified by obtaining the change attributes of metadata change events, providing a clear target for subsequent logical ID positioning. Calling the standard logical mapping table is suitable for standardized, high-frequency change scenarios. Millisecond-level positioning can be achieved through predefined mapping relationships, avoiding the need for redundant development of temporary solutions. Building a temporary logical mapping table is for non-routine changes, allowing for flexible definition of mapping rules based on the current scenario. Through a closed-loop design of "attribute extraction - type judgment - differentiated mapping - precise positioning," rapid, accurate, and scenario-based positioning of the impact scope of metadata changes is achieved, ensuring efficient processing of routine changes while also considering flexibility for special scenarios.

[0039] Optionally, depending on the current business scenario, the steps to build a temporary logical mapping table include:

[0040] Collect business requirements for the current business scenario;

[0041] Based on the business requirements, calculate the scenario score for the current business scenario;

[0042] Retrieve the historical logical mapping table that matches the scene segment;

[0043] Extract mapping logic that matches the business requirements from the matching historical logic mapping table to construct a mapping logic set;

[0044] The mapping logic in the mapping logic set is split and reorganized to generate a temporary logic mapping table.

[0045] By adopting the above technical solution, and collecting the business requirements of the current business scenario, it ensures that the temporary logic mapping table is highly matched with the actual business objectives, avoiding the introduction of redundant or irrelevant mapping logic. By calculating scenario scores and retrieving matching historical logic mapping tables, experience from similar past scenarios can be reused. Mapping logic matching business requirements is extracted from the matching historical logic mapping tables, and validated mature logic is directly included in the candidate set, improving construction efficiency while reducing error risks. By splitting and reorganizing the mapping logic set, different logic units can be flexibly combined to cope with complex business needs.

[0046] Optionally, based on the derivation results, the steps following the marking of downstream lineage edges indirectly dependent on the affected logical ID as suspicious include:

[0047] The duration of the suspicious flag is obtained, and the duration of the suspicious flag is greater than the repair duration of the affected logical ID;

[0048] Before the suspicious marker duration reaches the set alarm duration, determine whether the downstream bloodline edge that is indirectly dependent on the affected logical ID is normal;

[0049] If so, remove the suspicious flag;

[0050] If not, a manual review will be triggered, and an alarm message will be output.

[0051] By adopting the above technical solution, and setting the marking duration to be longer than the normal duration for repairing the affected logic ID, it is ensured that the suspicious status of downstream bloodline edges will not immediately trigger alarms within a reasonable repair cycle. During the validity period of the suspicious marking, the operating status of downstream bloodline edges is monitored in real time to automatically determine whether they have returned to normal. If the downstream bloodline edge remains abnormal during the observation period, a manual review process is immediately triggered and an alarm message is output. By setting the suspicious marking duration to be longer than the repair duration and the alarm triggering threshold, a risk control logic of "buffer observation period + graded response" is constructed. This avoids false alarms caused by temporary fluctuations and allows for timely intervention when anomalies persist, achieving refined monitoring of indirectly dependent links.

[0052] Secondly, this invention provides a business data lineage tracing system based on metadata, employing the following technical solution:

[0053] A metadata-based business data lineage tracing system includes:

[0054] The data monitoring module is used to monitor metadata change events during business processing on the cloud platform;

[0055] The data lookup module is used to query the logical mapping table and locate the affected logical ID based on the metadata change event;

[0056] The data analysis module is used to deduce bloodline edges from the constructed bloodline graph based on the affected logical IDs;

[0057] The data processing module is used to mark the bloodline edges that are directly dependent on the affected logical ID as invalid, and mark the downstream bloodline edges that are indirectly dependent on the affected logical ID as suspicious, based on the derivation results.

[0058] Thirdly, the present invention provides a terminal, which adopts the following technical solution:

[0059] A terminal, comprising:

[0060] The storage unit contains metadata-based business data lineage tracing programs.

[0061] A processor is used to execute a program stored in the memory to implement the steps of the above-described metadata-based business data lineage tracing method.

[0062] In summary, the present invention has at least the following beneficial effects:

[0063] By monitoring metadata change events on the cloud platform, changes to data structures or attributes can be captured in real time without manual updates, avoiding the delays of traditional post-audit methods and ensuring the timeliness of lineage updates. Based on the logical mapping table, affected logical IDs can be located, and dependencies can be deduced using the lineage graph, accurately identifying directly and indirectly dependent downstream data and reducing the error rate of manual investigation. Marking direct dependency edges as "failed" and indirect dependencies as "suspicious" enables refined management of lineage status, helping users quickly distinguish risk levels and prioritize critical issues. When data anomalies occur, the root cause (such as upstream metadata change points) can be quickly located through lineage relationships, shortening the problem investigation cycle and improving the efficiency of resolving data quality issues, thereby enhancing data governance efficiency. Attached Figure Description

[0064] Figure 1 This is a first flowchart of an embodiment of the present invention;

[0065] Figure 2 This is a second flowchart of an embodiment of the present invention;

[0066] Figure 3 This is a third flowchart of an embodiment of the present invention;

[0067] Figure 4 This is the fourth flowchart of an embodiment of the present invention;

[0068] Figure 5 This is the fifth flowchart of an embodiment of the present invention;

[0069] Figure 6 This is the sixth flowchart of an embodiment of the present invention;

[0070] Figure 7 This is the seventh flowchart of an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will be described in conjunction with the appendices in the embodiments of the present invention. Figure 1 - Appendix Figure 7 The technical solutions in the embodiments of the present invention are clearly and completely described herein. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] The first embodiment of this invention discloses a method for tracing the lineage of business data based on metadata. (Refer to...) Figure 1 The business data lineage tracing method may include S110-S140:

[0073] S110 monitors metadata change events during business processing on the cloud platform;

[0074] S120: Based on the metadata change event, query the logical mapping table and locate the affected logical ID;

[0075] S130, based on the affected logical ID, derive the bloodline edges from the constructed bloodline graph;

[0076] S140, based on the derivation results, mark the bloodline edges that are directly dependent on the affected logical ID as invalid, and mark the downstream bloodline edges that are indirectly dependent on the affected logical ID as suspicious.

[0077] Specifically, the Flink CDC component can be used to parse the Write-Ahead Log (WAL) file of the database, and a Flink CDC task can be configured to capture DDL change events of the database metadata tables. These change events can then be converted into a uniform format (such as JSON) and pushed to the metadata_events topic of a message queue (such as Kafka).

[0078] Reference Figure 2 Specifically, for S120, the steps of querying the logical mapping table and locating the affected logical ID based on the metadata change event can include S210-S240:

[0079] S210, retrieve the change attributes of the metadata change event;

[0080] S220, determine whether the change attribute is a regular change;

[0081] S230, if so, then call the standard logic mapping table and locate the affected logic ID from the standard logic mapping table;

[0082] S240. If not, then build a temporary logic mapping table based on the current business scenario, and locate the affected logic ID from the temporary logic mapping table.

[0083] Specifically, the ANTLR4 parser generation tool can be used to construct parsing rules for database DDL syntax, extracting from change events: change type (e.g., ADD / ALTER / DROP), affected objects (e.g., table name, field name), attribute details (e.g., data type, constraints), etc.; then, key fields are matched using regular expressions, and the results are output. The output is then matched against the rule base for pattern matching, and the Drools rule engine is used to perform logical judgments. The rule base defines regular and non-regular changes; regular changes can represent modifications to predefined fields (e.g., changes to field comments, adjustments to non-key indexes), while non-regular changes can represent structural changes (e.g., adding fields, modifying primary keys).

[0084] The logical mapping table is structured as {physical metadata ID: logical ID}. Extract the identifiers of the changed physical metadata (such as table name and field name), and then obtain the table-level and field-level logical IDs simultaneously via HMGET.

[0085] Furthermore, referring to Figure 3 Based on the current business scenario, the steps for building a temporary logical mapping table include S310-S350:

[0086] S310 collects the business requirements of the current business scenario;

[0087] S320, calculates the scenario score of the current business scenario based on business needs;

[0088] S330, retrieve the historical logic mapping table that matches the scene score;

[0089] S340: Extract the mapping logic that matches the business requirements from the matched historical logic mapping table to construct a set of mapping logic;

[0090] S350, splits and reorganizes the mapping logic in the mapping logic set to generate a temporary logic mapping table.

[0091] Specifically, Kafka Streams can be used to capture business operation events (such as order creation / user behavior) in real time, extract key parameters (such as tenant ID, operation type, and data version), deploy an OpenAPI specification collector, automatically parse the business system interface documents, extract field mapping relationships and constraints, and then output a structured requirement document to a document database. Based on the structured requirement document, a scoring dimension matrix is ​​constructed, and Flink Stateful Functions is used to maintain the scenario state, dynamically calculating the scenario score according to the formula: Scenario Score = ∑(Dimension Weight × Normalized Value). The scoring dimension matrix includes dimensions, dimension weights, and dimension calculation methods. For example, a dimension representing complexity has a dimension weight of 0.4 and is calculated as the number of fields × nesting level × 1.5; a dimension representing timeliness requirements has a dimension weight of 0.3 and is calculated as the SLA latency requirement (ms) / 100; a dimension representing historical call frequency has a dimension weight of 0.3 and is calculated as log10(number of calls in the last 7 days + 1) × 10.

[0092] The historical mapping table is abstracted as feature vectors, and an index is built using the Milvus vector database. The current scene vector is input, and a similarity query is performed, prioritizing historical mapping tables with similarity greater than a set threshold. The Drools rule engine loads the historical mapping table, filtering historical mapping logic according to filtering rules (such as DRL syntax). For multiple mapping logics of the same field, a confidence score is calculated: confidence score = historical call success rate × usage frequency weight. Logics with a confidence score > 85% are retained. The mapping logic is modeled as a directed graph (nodes = fields, edges = transformation rules). The Tarjan algorithm is used to identify strongly connected components, decomposing them into independent subgraphs. High-confidence logic branches in the independent subgraphs are merged, and redundant transformation paths are eliminated.

[0093] Reference Figure 4 For S130, the steps for deriving lineage edges from the constructed lineage graph based on the affected logical ID specifically include S410-S450:

[0094] S410, from the constructed lineage graph, obtain the dependency lineage edges of the direct and indirect dependency fields associated with the affected logical ID;

[0095] S420, calculate the first-order influence score of dependent bloodline edges based on bloodline path characteristics;

[0096] S430, filter out dependent bloodline edges whose influence on the score is greater than the set score value;

[0097] S440, based on historical impact on business and combined with primary impact score, calculates secondary impact score for the selected dependent bloodline edges;

[0098] S450: Filter dependent bloodline edges whose secondary influence score is greater than the score threshold, and use them as target dependent bloodline edges.

[0099] Specifically, a lineage graph is a graphical representation used to describe the dependencies between data or entities. Nodes in the graph represent logical IDs (which can be data objects, tasks, etc.), and edges represent the dependencies between them. Target dependency lineage edges include direct dependency lineage edges and indirect dependency lineage edges.

[0100] Directly dependent on bloodline edges: Perform a first-level neighbor query in the bloodline graph: Input the affected logical ID as the starting point, and retrieve all directly connected outgoing edges (i.e., downstream dependencies) and incoming edges (i.e., upstream dependencies).

[0101] Indirect dependency edges: Based on graph traversal algorithms (such as Breadth-First Search), a depth threshold (such as 5 levels) is set, and multi-level downstream dependency edges are recursively obtained. Path analysis is used to distinguish between "strong associations" (such as direct data connections) and "weak associations" (such as those through intermediate tables).

[0102] The lineage path features include dependency depth weight and business weight. Dependency depth includes direct and indirect dependencies. Business weight is dynamically weighted based on field business attributes: for example, core transaction fields (e.g., amount / user ID): weight × 1.5; analytical fields (e.g., log tags): weight × 0.7. Weight values ​​are sourced from field importance tags in the metadata management system. Direct dependency weight can be 0.8, level 1 indirect dependency weight can be 0.5, and level ≥2 indirect dependency weight can be 0.3. First-order impact score = (dependency depth weight × 0.6) + (business weight × 0.4). The first-order impact score of lineage edges is sharded by time (e.g., 5-second window) using Flink window functions, and the set of edges with a first-order impact score > the set score value is output to a Kafka topic. The set score value can be dynamically adjusted based on the business scenario, based on a base threshold; for example, during peak business periods, the set score value is lowered from the base threshold. Calculate the secondary impact score: Determine the historical data source, which can include historical failure frequency and business impact coefficient. Historical failure frequency refers to the failure records of related edges within the past 3 months (such as data delays and quality alarms). The business impact coefficient is the rate of decrease in access volume for the corresponding report / API in the business log. Overlay calculation logic: Secondary impact score = Primary impact score × (1 + Historical failure frequency × 0.2 + Business impact coefficient). Business impact coefficient: If the daily access volume of the related report > 100,000: +0.3; if the related API call chain involves the payment core: +0.5.

[0103] The secondary impact scores are sorted in descending order, and edges with scores greater than the threshold are automatically selected as "highly sensitive target edges". For edges with the same score, the shallower edge is selected first (direct dependency > indirect dependency).

[0104] Reference Figure 5 For S140, based on the derivation results, the steps after marking the bloodline edge directly dependent on the affected logical ID as invalid specifically include S510-S550:

[0105] S510, obtain the urgency of business processing;

[0106] S520 determines whether the affected logical IDs need to be repaired based on their urgency.

[0107] S530, if so, then parse the problem type of the affected logical ID;

[0108] S540, determine the repair plan based on the problem type;

[0109] After repairing the affected logical IDs, the S550 updates the lineage network and records the repair information.

[0110] Specifically, after obtaining the urgency of the business processing, it is determined whether the urgency exceeds the urgency threshold. If so, the affected logical ID is determined to need repair. System logs are reviewed to record error information, exception stacks, etc., related to the affected logical ID. For example, system logs may show database connection errors, code logic errors, etc. Based on log analysis results, the problem type is categorized into database problems, code logic problems, configuration problems, etc. A mapping relationship between problem types and repair solutions is established. For example, for database problems, the repair solution may include checking database connection configuration and repairing database table structure or data; for code logic problems, the repair solution may be modifying code logic and adding exception handling mechanisms; for configuration problems, the repair solution may be updating configuration files. A data lineage management tool is used to update the relationship between the repaired logical ID and related data and business processes to ensure the accuracy of the lineage network. The repair time, repair solution, and status before and after repair are recorded and stored in a dedicated repair record database for subsequent auditing and traceability. If it is determined that no repair is needed, manual verification information can be output.

[0111] Furthermore, referring to Figure 6 For S510, the specific steps for obtaining the urgency of a service processing request include S610-S640:

[0112] S610, obtain the scope of impact on core business from the affected logical ID;

[0113] S620, obtain the depth impact score of the affected logical ID on the business depth;

[0114] S630, obtain the urgent impact score for the urgency of repair;

[0115] S640 calculates the urgency of business processing based on the scope impact score, depth impact score, and urgency impact score.

[0116] Specifically, by analyzing business tables in a data warehouse or database, identify the core business processes and data associated with the affected logical ID. For example, in an e-commerce system, a logical ID might be linked to product information, order information, and user information. Count the number of affected core business processes, the number of data tables involved, or the number of business modules involved. Based on the statistical results, establish scoring rules. If 1-2 core business processes or data tables are involved, the impact is scored as 1 point; 3-5 are scored as 2 points; and more than 5 are scored as 3 points. Break down the core business processes related to the affected logical ID in detail, analyzing the degree of impact of the logical ID on each business step. For example, in the order processing flow, the logical ID might affect order creation, payment, and shipping. Score the impact based on the degree of impact. If the logical ID only affects the surface information display of the business process, the deep impact is scored as 1 point; if it affects a critical step in the business process, such as the payment step, the deep impact is scored as 2 points; if it causes the business process to malfunction, the deep impact is scored as 3 points. Analyze whether the affected business processes are time-sensitive. For example, during e-commerce promotional events, the logical IDs related to order processing are affected, making their repair urgency high; while for logical IDs related to routine data statistics, the repair urgency is relatively low. Scoring rules are established based on business time sensitivity. If the impact occurs within a critical business time window, the urgency score is 3 points; if it occurs within a non-critical business time window but affects the normal operation of some business activities, the urgency score is 2 points; if the impact on the current business operation is minor, the urgency score is 1 point. Different weights are assigned to the scope impact score, depth impact score, and urgency impact score, such as a weight of 0.3 for the scope impact score, 0.4 for the depth impact score, and 0.3 for the urgency score. Business processing urgency = Scope impact score × 0.3 + Depth impact score × 0.4 + Urgent impact score × 0.3.

[0117] Reference Figure 7 In S140, based on the derivation results, the steps following the marking of downstream bloodline edges indirectly dependent on the affected logical ID as suspicious specifically include S710-S740:

[0118] S710, obtain the duration of the suspicious flag, the duration of the suspicious flag is greater than the repair duration of the affected logical ID;

[0119] S720, before the suspicious tag duration reaches the set alarm duration, determines whether the downstream bloodline edge that is indirectly dependent on the affected logical ID is normal;

[0120] S730, if so, remove the suspicious flag;

[0121] S740, if not, trigger manual review and output alarm information.

[0122] Specifically, while marking suspicious edges, a timestamp `start_time` is recorded, and a scheduled task (such as CronJob) is started to periodically check the difference between the current time and `start_time`. The historical average repair duration `avg_fix_time` for this logical ID is extracted from system logs or monitoring tools (such as Prometheus) (e.g., by analyzing the repair records of historical work orders). When (current_time - start_time) > (avg_fix_time × safety factor (e.g., 1.5)), it is determined that the suspicious marking duration has exceeded the limit.

[0123] Probes can be deployed on indirectly dependent downstream nodes to check whether data throughput (e.g., Kafka consumer latency) and data quality (e.g., field non-null rate) meet standards. The downstream output is compared with preset business rules (e.g., order amount > 0). An anomaly is identified if the failure rate exceeds 5%, and downstream services are required to send heartbeat signals every minute; three consecutive heartbeats trigger an anomaly. For example, after five consecutive normal health probe tests, a status update is automatically triggered, suspicious markers are cleared, the edge status in the lineage graph is updated to "Verified," and related systems are notified to update their caches via a message queue (e.g., RabbitMQ). Alarm information includes anomaly trend charts for suspicious edges (e.g., Grafana screenshots), the impact scope of related business modules (e.g., order payment failure rate), and remediation solutions for similar historical issues.

[0124] The implementation scenario for this embodiment is as follows:

[0125] The system monitors metadata change events during business processing on the cloud platform, obtains the change attributes of the metadata change events, and determines whether the change attributes are regular changes. If so, it calls the standard logic mapping table and locates the affected logic ID from the standard logic mapping table. Otherwise, it collects the business requirements of the current business scenario, calculates the scenario score of the current business scenario based on the business requirements, and retrieves the historical logic mapping table that matches the scenario score. It extracts the mapping logic that matches the business requirements from the matching historical logic mapping table to construct a mapping logic set. It then splits and reorganizes the mapping logic in the mapping logic set to generate a temporary logic mapping table and locates the affected logic ID from the temporary logic mapping table.

[0126] From the constructed lineage graph, obtain the dependency lineage edges of the direct and indirect dependency fields associated with the affected logical ID. Calculate the primary impact score of the dependency lineage edges based on lineage path characteristics. Filter out dependency lineage edges with a primary impact score greater than a set score value. Based on historical impact on business operations and the primary impact score, calculate the secondary impact score of the filtered dependency lineage edges. Filter out dependency lineage edges with a secondary impact score greater than a score threshold as target dependency lineage edges. Mark the target dependency lineage edges that are directly dependent on the affected logical ID as invalid, and mark the downstream lineage edges that are indirectly dependent on the affected logical ID as suspicious.

[0127] For failed lineage edges: Obtain the scope impact score, depth impact score, and urgency impact score of the affected logical IDs affecting core business, and calculate the urgency of business processing based on the scope impact score, depth impact score, and urgency impact score; then, based on the urgency, determine whether the affected logical IDs need to be repaired. If so, parse the problem type of the affected logical IDs and determine the repair plan based on the problem type; after repairing the affected logical IDs, update the lineage relationship network and record the repair information.

[0128] For suspicious bloodline edges: obtain the suspicious marking duration. Before the suspicious marking duration reaches the set alarm duration, determine whether the downstream bloodline edge that is indirectly dependent on the affected logical ID is normal. If it is, cancel the suspicious marking. If not, trigger manual review and output alarm information.

[0129] Based on the above method embodiments, the second embodiment of the present invention discloses a metadata-based business data lineage tracing system. The metadata-based business data lineage tracing system of this embodiment can implement any of the above-described metadata-based business data lineage tracing methods, and the specific working process of each module in the metadata-based business data lineage tracing system can be referred to the corresponding process in the above method embodiments.

[0130] For ease of understanding, an example is as follows: A metadata-based business data lineage tracing system includes:

[0131] The data monitoring module is used to monitor metadata change events during business processing on the cloud platform;

[0132] The data lookup module is used to query the logical mapping table and locate the affected logical IDs based on metadata change events;

[0133] The data analysis module is used to deduce lineage edges from the constructed lineage graph based on the affected logical IDs;

[0134] The data processing module is used to mark the bloodline edges that are directly dependent on the affected logical ID as invalid, and mark the downstream bloodline edges that are indirectly dependent on the affected logical ID as suspicious, based on the derivation results.

[0135] A third embodiment of the present invention provides a terminal. As one implementation of the terminal, the terminal may include: a memory and a processor; wherein,

[0136] The memory is used to store metadata-based business data lineage tracing procedures;

[0137] The processor is used to execute programs stored in memory to implement the steps of the above-described metadata-based business data lineage tracing method.

[0138] The memory can communicate with the processor via a communication bus, which can be an address bus, a data bus, a control bus, etc.

[0139] Additionally, the memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device.

[0140] Furthermore, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0141] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A method for tracing the lineage of business data based on metadata, characterized in that, include: Monitor metadata change events during business processing on the cloud platform; Based on the metadata change event, query the logical mapping table and locate the affected logical ID; Based on the affected logical ID, the lineage edges are derived from the constructed lineage graph; the lineage graph is a graphical representation used to describe the dependencies between data or entities, where nodes represent logical IDs and edges represent dependencies. Based on the derivation results, the bloodline edges that are directly dependent on the affected logical ID are marked as invalid, and the downstream bloodline edges that are indirectly dependent on the affected logical ID are marked as suspicious. Based on the affected logical ID, the specific steps for deriving lineage edges from the constructed lineage graph include: From the constructed lineage graph, obtain the dependency lineage edges of the direct and indirect dependency fields associated with the affected logical ID; The influence score of the dependent bloodline edge is calculated based on the bloodline path characteristics. The bloodline path characteristics include dependency depth weight and business weight. Dependency depth includes direct dependency and indirect dependency. The business weight is dynamically weighted based on the field business attributes. Filter out dependent bloodline edges whose influence on the score is greater than a set score value; Based on the historical impact on business operations, and combined with the primary impact score, the secondary impact score of the selected dependent bloodline edges is calculated; Dependency edges with secondary influence scores greater than a score threshold are selected as target dependency edges; the target dependency edges include direct dependency edges and indirect dependency edges. Based on the derivation results, the steps following the marking of bloodline edges directly dependent on the affected logical ID as invalid include: Get the urgency level of the business process; Based on the urgency level, determine whether the affected logical ID needs to be repaired; If so, then parse the problem type of the affected logical ID; Determine the repair plan based on the type of problem described; After repairing the affected logical IDs, update the bloodline network and record the repair information; Based on the derivation results, the steps following the marking of downstream lineage edges indirectly dependent on the affected logical ID as suspicious include: The duration of the suspicious flag is obtained, and the duration of the suspicious flag is greater than the repair duration of the affected logical ID; Before the suspicious marker duration reaches the set alarm duration, determine whether the downstream bloodline edge that is indirectly dependent on the affected logical ID is normal; If so, remove the suspicious flag; If not, a manual review will be triggered, and an alarm message will be output.

2. The method for tracing the lineage of business data based on metadata according to claim 1, characterized in that, The steps to determine the urgency of a business process include: Obtain the scope and impact score of the core business affected by the affected logical ID; Obtain the depth impact score of the affected logical ID on the business depth; Obtain the urgency score for repair; The urgency of the business process is calculated based on the scope impact score, the depth impact score, and the urgency impact score.

3. The method for tracing the lineage of business data based on metadata according to claim 1, characterized in that, The steps of querying the logical mapping table and locating the affected logical ID based on the metadata change event include: Retrieve the change attributes of metadata change events; Determine whether the changed attribute is a regular change; If so, the standard logic mapping table is invoked, and the affected logic ID is located from the standard logic mapping table; If not, then based on the current business scenario, a temporary logical mapping table is built, and the affected logical ID is located from the temporary logical mapping table.

4. The method for tracing the lineage of business data based on metadata according to claim 3, characterized in that, Based on the current business scenario, the steps to build a temporary logical mapping table include: Collect business requirements for the current business scenario; Based on the business requirements, calculate the scenario score for the current business scenario; Retrieve the historical logical mapping table that matches the scene segment; Extract mapping logic that matches the business requirements from the matching historical logic mapping table to construct a mapping logic set; The mapping logic in the mapping logic set is split and reorganized to generate a temporary logic mapping table.

5. A business data lineage tracing system based on metadata, characterized in that, Performing the metadata-based business data lineage tracing method as described in any one of claims 1-4 includes: The data monitoring module is used to monitor metadata change events during business processing on the cloud platform; The data lookup module is used to query the logical mapping table and locate the affected logical ID based on the metadata change event; The data analysis module is used to deduce bloodline edges from the constructed bloodline graph based on the affected logical IDs; The data processing module is used to mark the bloodline edges that are directly dependent on the affected logical ID as invalid, and mark the downstream bloodline edges that are indirectly dependent on the affected logical ID as suspicious, based on the derivation results.

6. A terminal, characterized in that, include: The storage unit contains metadata-based business data lineage tracing programs. A processor for executing a program stored in the memory to implement the metadata-based business data lineage tracing method as described in any one of claims 1-4.

Images