Node anomaly detection method and device, electronic equipment, medium and product

Abstract

The application discloses a node anomaly detection method and device, electronic equipment, medium and product. The method comprises the following steps: when a suspected abnormal node is detected, performing breadth-first search based on a blood relation directed graph with the suspected abnormal node as a starting point to obtain a plurality of upstream nodes; obtaining a plurality of detection snapshot results corresponding to the plurality of upstream nodes in a current scheduling period; calculating a comprehensive confidence of the suspected abnormal node according to the plurality of detection snapshot results; and obtaining an abnormal confidence level of the suspected abnormal node according to the comprehensive confidence. The method can upgrade single-point isolated time sequence detection to multi-node joint inference based on data blood relation, and effectively distinguish between normal business fluctuations and real data faults by introducing upstream detection evidence for cross-validation, thereby significantly reducing the false positive rate and improving the operation and maintenance efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a node anomaly detection method, apparatus, electronic device, readable storage medium, and computer program product. Background Technology

[0002] As the scale of offline scheduling tasks in enterprise data platforms continues to expand and data links become increasingly complex, automatic data quality detection has become a critical link in ensuring the security of downstream businesses. Existing business node anomaly detection methods typically extract historical time-series sequences independently for each node and use time-series anomaly detection algorithms to determine whether the current period's data value deviates from the normal range. If it does, it is judged as an anomaly and an alarm is triggered. However, this single-node independent detection mechanism relies solely on the node's own historical patterns for anomaly judgment, lacking joint analysis of the correlation between upstream and downstream nodes. It cannot distinguish whether the current node's anomaly is caused by upstream fault propagation or by its own business fluctuations, making it highly susceptible to misjudging upstream data changes as node-specific anomalies. This results in a high false alarm rate and reduces the accuracy of anomaly detection. Summary of the Invention

[0003] In view of the above problems, this application provides a node anomaly detection method, apparatus, electronic device, readable storage medium and computer program product, which can solve the problem of high false alarm rate caused by lack of upstream and downstream correlation analysis in single node detection.

[0004] Firstly, this application provides a node anomaly detection method, including: When a suspected abnormal node is detected, a breadth-first search is performed based on the bloodline directed graph, starting from the suspected abnormal node, to obtain multiple upstream nodes. Obtain multiple detection snapshot results corresponding to the multiple upstream nodes within the current scheduling period; The overall confidence level of the suspected abnormal node is calculated based on the results of the multiple detection snapshots; The anomaly confidence level of the suspected abnormal node is obtained based on the comprehensive confidence level.

[0005] In the above technical solution, this method upgrades traditional single-point isolated time-series detection to multi-node joint inference based on data lineage. It efficiently collects detection snapshot results of upstream nodes along the lineage directed graph through breadth-first search and calculates the comprehensive confidence score accordingly. This introduces "external evidence" from the entire upstream of the chain for cross-validation of suspected abnormal nodes, thereby effectively distinguishing between normal business fluctuations and real data failures, and significantly reducing the high false alarm rate caused by single-point time-series detection. At the same time, classifying the anomaly confidence level based on the comprehensive confidence score makes alarm decisions more interpretable and hierarchically operable. This avoids the ineffective interference of low-confidence anomalies to operation and maintenance personnel and provides a quantitative basis for rapid response to high-confidence failures, thereby improving the detection accuracy and operation and maintenance efficiency in the scenario of automatic data quality discovery.

[0006] In some implementations, when a suspected abnormal node is detected, a breadth-first search is performed based on the kinship directed graph, starting from the suspected abnormal node, to obtain multiple upstream nodes, including: When a suspected abnormal node is detected, it is determined whether the suspected abnormal node has a reachable upstream node in the bloodline directed graph; If it exists, then starting from the suspected abnormal node, perform a breadth-first search based on the bloodline directed graph to obtain multiple upstream nodes; The breadth-first search is equipped with a maximum backtracking depth limit and a loop prevention mechanism for the set of visited nodes.

[0007] In the above technical solution, this method can provide stable and reliable upstream detection evidence for comprehensive confidence calculation while ensuring that the kinship tracing process is controllable and efficient, thereby balancing the improvement of detection accuracy and the stability of system performance.

[0008] In some embodiments, the method further includes: If the suspected abnormal node has no reachable upstream node in the kinship directed graph, then the self-abnormality confidence score of the suspected abnormal node is obtained, and the self-abnormality confidence score is normalized to obtain the comprehensive confidence score of the suspected abnormal node.

[0009] In the above technical solution, the method can automatically downgrade to single-node detection mode in scenarios where lineage information is missing or there is no reachable upstream node, ensuring that the detection capability does not fail due to incomplete lineage data, thereby guaranteeing the engineering robustness and reliability of the overall solution under different data environments.

[0010] In some implementations, calculating the overall confidence level of the suspected abnormal node based on the multiple detection snapshot results includes: Based on the multiple detection snapshot results, obtain multiple fluctuation amplitude factors, multiple detection index type coefficients, and multiple anomaly confidence scores corresponding to the multiple upstream nodes; Obtain multiple lineage distance attenuation factors between the multiple upstream nodes and the suspected abnormal nodes; Based on the multiple fluctuation amplitude factors, the multiple detection index type coefficients, and the multiple bloodline distance attenuation factors, calculate multiple initial weights that correspond one-to-one with the multiple upstream nodes; The initial weights are normalized to obtain a set of final weights that correspond one-to-one with the upstream nodes. Obtain the self-abnormal confidence score and self-weight of the suspected abnormal node within the current scheduling period; The initial confidence level of the suspected abnormal node is calculated based on its own weight, its own anomaly confidence score, multiple final weights, and multiple anomaly confidence scores. The initial confidence level is then corrected to obtain the overall confidence level.

[0011] In the above technical solution, the method can dynamically balance the contributions of upstream and downstream detection evidence by integrating the weighted calculation of multi-dimensional factors and the adaptive correction mechanism, thereby significantly improving the accuracy and reliability of anomaly judgment under complex lineage topology.

[0012] In some implementations, obtaining multiple fluctuation amplitude factors, multiple detection index type coefficients, and multiple anomaly confidence scores corresponding one-to-one with the multiple upstream nodes based on the multiple detection snapshot results includes: Based on the multiple detection snapshot results, obtain multiple indicator fluctuation amplitudes, multiple detection indicator types, and multiple anomaly confidence scores corresponding to the multiple upstream nodes; Based on the fluctuation range of the multiple indicators, a piecewise function is used to calculate multiple fluctuation range factors that correspond one-to-one with the multiple upstream nodes; Based on the multiple detection index types and type coefficient calibration table, obtain multiple detection index type coefficients that correspond one-to-one with the multiple upstream nodes; The type coefficient calibration table is obtained by backtesting and optimizing the detection effect of historical abnormal samples.

[0013] In the above technical solution, the method can ensure that the values ​​of the fluctuation amplitude factor and the type coefficient are based on evidence and closely resemble the abnormal diagnosis rules in real scenarios, thereby ensuring the rationality of the weight calculation and providing a reliable basis for the accuracy of the comprehensive confidence level.

[0014] In some implementations, obtaining multiple lineage distance attenuation factors between the plurality of upstream nodes and the suspected abnormal node includes: Obtain the number of lineage layers between the multiple upstream nodes and the suspected abnormal nodes; Based on the number of bloodline layers, an exponential decay function is used to calculate multiple bloodline distance decay factors that correspond one-to-one with the multiple upstream nodes.

[0015] In the above technical solution, this method enables upstream nodes with closer bloodline distance to contribute more to the current anomaly judgment, thus making it more in line with the actual law of fault propagation decaying with the link, thereby improving the rationality of upstream node weight allocation and the credibility of comprehensive confidence calculation.

[0016] In some implementations, obtaining the suspected abnormal node's own anomaly confidence score and its own weight within the current scheduling period includes: Obtain the total number of upstream nodes and the self-detection results of the suspected abnormal nodes within the current scheduling period; Determine whether the total number of nodes has reached a preset sufficient threshold; If so, the preset weight lower limit value is determined as the weight of the suspected abnormal node itself; If not, select the weight corresponding to the total number of nodes from the weight configuration table as the weight of the suspected abnormal node; The self-abnormality confidence score of the suspected abnormal node is obtained based on the self-detection results; The weights in the weight configuration table decrease as the total number of nodes increases, and the weight of a node itself is complementary to the sum of the multiple final weights.

[0017] In the above technical solution, the method can dynamically adjust the ratio of its own weight to the upstream weight according to the number of upstream nodes, so that the calculation of the comprehensive confidence score can always maintain a reasonable weight distribution in scenarios with different levels of information sufficiency between upstream and downstream. This avoids the one-sidedness of its own judgment being overly dominant when upstream information is abundant, while avoiding the risk of misjudgment caused by the excessively high weight of joint inference when upstream information is scarce. In this way, the scenario adaptability and overall robustness of the comprehensive confidence score calculation are improved.

[0018] In some implementations, the step of correcting the initial confidence level to obtain a comprehensive confidence level includes: Among the multiple upstream nodes, nodes with extreme fluctuation characteristics that cannot be traced further upstream are marked as root cause candidate nodes; wherein, the extreme fluctuation characteristics include at least one of the following: the fluctuation amplitude of the detection index reaches a preset extreme threshold, the entire table is empty, and the entire column is empty. Obtain the lineage propagation chain of the root cause candidate nodes; The initial confidence level is corrected based on the bloodline transmission chain to obtain the comprehensive confidence level.

[0019] In the above technical solution, the method can use the propagation chain of root cause candidate nodes to make targeted corrections to the initial confidence level, thereby enhancing the confidence level of downstream nodes affected by the real source and suppressing isolated fluctuations that lack source support, so that the comprehensive confidence level can more accurately reflect the real propagation status of data problems.

[0020] In some embodiments, the step of correcting the initial confidence level based on the bloodline transmission chain to obtain a comprehensive confidence level includes: Based on the bloodline transmission chain, determine whether the suspected abnormal node is an affected downstream node; If the suspected abnormal node is an affected downstream node, the initial confidence level of the suspected abnormal node is positively corrected according to the positive correction coefficient to obtain the comprehensive confidence level. If the suspected abnormal node is not an affected downstream node, then the bloodline propagation chain is used to determine whether the suspected abnormal node is a slightly fluctuating node without upstream support. If the suspected anomalous node is the slightly fluctuating node, then the initial confidence level of the suspected anomalous node is negatively corrected according to the negative correction coefficient to obtain the comprehensive confidence level. The affected downstream node is a node located downstream of the root cause candidate node that has been detected as potentially abnormal. The positive correction coefficient is negatively correlated with the kinship distance between the suspected abnormal node and the root cause candidate node.

[0021] In the above technical solution, the method can automatically match the corresponding correction strategy according to the positional relationship between the suspected abnormal node and the root cause candidate node, so that the comprehensive confidence level can more accurately reflect the true fault state of the node.

[0022] In some implementations, after obtaining the anomaly confidence level of the suspected anomaly node based on the comprehensive confidence level, the method further includes: Among the multiple upstream nodes, nodes with extreme fluctuation characteristics and unable to be traced further upstream are marked as root cause candidate nodes; Determine whether there is a downstream node to be alarmed in the lineage propagation chain of the root cause candidate node; If there are downstream nodes to be alarmed, obtain the node list of the downstream nodes to be alarmed, the first alarm information of the downstream nodes to be alarmed, and the second alarm information of the root cause candidate nodes; Using the root cause candidate node as the aggregation key, the first alarm information and the second alarm information are aggregated to obtain aggregated alarm information; Based on the aggregated alarm information and the node list of downstream nodes to be alarmed, output abnormal alarm information; The downstream node to be alarmed is a node whose abnormal confidence level meets the preset alarm level. The abnormal alarm information includes the identifier of the root cause candidate node, the fault type of each downstream node to be alarmed, the node list of the downstream nodes to be alarmed, and the overall confidence level of the downstream nodes to be alarmed.

[0023] In the above technical solution, the method can aggregate multiple downstream alarms caused by the same root cause into a main alarm, so that the on-duty personnel do not have to deal with a large number of repetitive and scattered alarm information, thereby directly locating the source of the fault and grasping the complete scope of impact, thus significantly improving the efficiency of fault response and operation and maintenance experience.

[0024] Secondly, this application provides a node anomaly detection device, comprising: The search unit is used to perform a breadth-first search based on the kinship directed graph, starting from the suspected abnormal node, when a suspected abnormal node is detected, to obtain multiple upstream nodes. The first acquisition unit is used to acquire multiple detection snapshot results corresponding to the multiple upstream nodes in the current scheduling period; A calculation unit is used to calculate the overall confidence level of the suspected abnormal node based on the results of the multiple detection snapshots; The second acquisition unit is used to acquire the anomaly confidence level of the suspected abnormal node based on the comprehensive confidence level.

[0025] In the above technical solution, the device can upgrade traditional single-point isolated time series detection to multi-node joint inference based on data lineage. It efficiently collects detection snapshot results of upstream nodes along the lineage directed graph through breadth-first search, and calculates the comprehensive confidence score accordingly. This introduces "external evidence" from the entire upstream of the link for cross-validation of suspected abnormal nodes, thereby effectively distinguishing between normal business fluctuations and real data failures, and significantly reducing the high false alarm rate caused by single-point time series detection. At the same time, classifying the anomaly confidence level based on the comprehensive confidence score can also make alarm decisions more interpretable and hierarchically operable. This avoids the ineffective interference of low-confidence anomalies to operation and maintenance personnel, and provides a quantitative basis for rapid response to high-confidence failures. In this way, the detection accuracy and operation and maintenance efficiency in the automatic data quality discovery scenario are improved as a whole.

[0026] Thirdly, this application provides an electronic device, the electronic device including a memory and a processor, the memory for storing a computer program, the processor running the computer program to cause the electronic device to perform the node anomaly detection method described in any one of the first aspects.

[0027] Fourthly, this application provides a readable storage medium storing a computer program, which, when executed by a processor, performs the node anomaly detection method described in any one of the first aspects.

[0028] Fifthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, performs the node anomaly detection method described in any one of the first aspects.

[0029] The beneficial effects of this application are as follows: By using multi-node joint inference and comprehensive confidence calculation, the alarm confidence of the current node is automatically reduced when the upstream is normal, thereby effectively distinguishing between normal business fluctuations and real data failures, significantly reducing the false alarm rate and improving alarm credibility; at the same time, during the lineage tracing process, candidate nodes for the root cause of the fault are automatically identified and the root cause and propagation link are directly marked in the alarm, so that on-duty personnel do not need to manually check layer by layer, thereby greatly shortening the fault location time; in addition, by making full use of the high confidence characteristics of extreme anomalies of row number type through differentiated detection indicator type weight coefficients, and being compatible with multiple detection indicators such as field type and business type, the detection coverage is wider and the comprehensive judgment is more accurate; when the lineage information is missing or the upstream is not connected to the detection, it automatically degrades to the single-node detection mode, which can also ensure that the detection capability does not fail and has good engineering robustness; finally, by aggregating multiple downstream alarms driven by the same root cause into one, alarm noise can be significantly reduced, thereby reducing the burden of invalid processing, while clearly showing the scope of the fault impact and improving the overall operation and maintenance experience. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the node anomaly detection method in some embodiments of this application; Figure 2 This is a schematic diagram of the node anomaly detection device in some embodiments of this application; Figure 3This is a schematic diagram of the structure of an electronic device in some embodiments of this application. Detailed Implementation

[0032] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0034] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more (including two), similarly, "multiple sets" refers to two or more sets (including two sets), and "multiple pieces" refers to two or more pieces (including two pieces) unless otherwise explicitly defined.

[0035] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0037] Existing technologies primarily employ single-node time-series detection or static threshold rules to independently determine anomalies for each data task, failing to utilize the detection status of upstream and downstream data lines for cross-validation. When normal business fluctuations (such as major promotions or events) cause changes in data metrics, false alarms are highly likely to occur. Furthermore, when a real data failure occurs, the same source of failure can propagate downstream along the data line, triggering numerous duplicate alarms. Moreover, on-duty personnel must manually investigate layer by layer to locate the root cause, resulting in low response efficiency.

[0038] To address the aforementioned technical issues, this application provides a node anomaly detection method. This method aims to: examine the anomaly judgment of a single node within the upstream and downstream relationships formed by data lineage; collect the detection status of upstream nodes as external verification evidence by tracing the source; dynamically adjust the calculation basis of comprehensive confidence based on the presence and quantity of upstream information; and combine root cause identification and propagation chain aggregation to converge scattered alarms into information that can directly target the source, thereby transforming anomaly detection from isolated decision-making into joint inference with a global perspective.

[0039] like Figure 1 As shown, some embodiments of this application provide a node anomaly detection method, which includes: S100. When a suspected abnormal node is detected, a breadth-first search is performed based on the bloodline directed graph, starting from the suspected abnormal node, to obtain multiple upstream nodes.

[0040] In this embodiment, the suspected abnormal node is the data output task node that is initially determined by the time-series anomaly detection algorithm to have data quality problems, and serves as the starting point for triggering the lineage tracing process.

[0041] In this embodiment, the starting point is the suspected abnormal node itself, that is, the starting node of the lineage backtracking traversal.

[0042] In this embodiment, the lineage directed graph is a directed graph that records the causal relationship between data production and data consumption, used to describe which upstream tasks produce data tables or fields and which downstream tasks consume them.

[0043] In this embodiment, breadth-first search (BFS) is a strategy that traverses the directed graph of lineage layer by layer from the starting point, expanding layer by layer rather than going deep along a single path, in order to collect detection snapshot results of all reachable upstream nodes.

[0044] In this embodiment, the upstream node is a task node located upstream of the current node in the kinship directed graph, whose data output is directly or indirectly consumed by the current node.

[0045] In this embodiment, the method can upgrade isolated single-node judgment to multi-node joint inference on the bloodline skeleton, so that the anomaly judgment has a global perspective.

[0046] As an optional implementation, a breadth-first search is performed based on the lineage directed graph to obtain multiple upstream nodes. This also includes: obtaining the node type of the suspected abnormal node in the lineage directed graph, where the node type includes any one of the following: source layer node, detail layer node, summary layer node, and application layer node; reading the corresponding maximum backtracking depth upper limit from the configuration table according to the node type; during the breadth-first search traversal, for each upstream node on the current branch, stopping the upward traversal of that branch when any of the following conditions are met: the current traversal depth reaches the maximum backtracking depth upper limit corresponding to the node type; the lineage distance decay factor of the current upstream node is lower than a preset truncation threshold; the abnormality confidence score of the current upstream node is lower than a preset no-abnormality threshold and the current traversal depth exceeds a preset trusted upstream distance.

[0047] In the above implementation, the lineage depth of large-scale data links can reach tens or even hundreds of layers. If all upstream nodes are traversed without restriction, unacceptable computational overhead and storage pressure will be generated. At the same time, the marginal contribution of upstream nodes with excessively large lineage distances to the anomaly detection of the current node approaches zero, making continued backtracking extremely cost-effective. This implementation, through triple control of depth upper limit, attenuation truncation, and branch pruning, can effectively constrain the backtracking range while ensuring detection accuracy, avoiding excessive consumption of computational resources, and enabling the scheme to maintain stable engineering performance under complex lineage topologies.

[0048] As an optional implementation, breadth-first search based on a directed graph of lineage further includes: maintaining a set of visited nodes; when a node already existing in the set of visited nodes is encountered during the breadth-first search traversal, the node is marked as a loop node, and the lineage loop in which the loop node is located is recorded; in the process of calculating the overall confidence, for the upstream node marked as a loop node, a basic lineage distance attenuation factor is calculated based on the number of lineage layers between the loop node and the suspected abnormal node, and then the basic lineage distance attenuation factor is multiplied by a preset loop attenuation coefficient, and the product is used as the final lineage distance attenuation factor of the loop node, wherein the loop attenuation coefficient is less than 1.

[0049] In the above implementation, circular dependencies may exist in the data lineage graph. If only the set of visited nodes is used for loop prevention, although infinite traversal can be prevented, the lineage layer parameter of nodes in the loop will have non-unique values. Directly using the distorted lineage layer to calculate the weight will lead to bias in the confidence judgment. This implementation, while preventing infinite traversal, reduces the weight contribution of loop nodes to a reasonable level through a combination of loop marking and additional attenuation coefficients. This avoids artificially high or low confidence due to distortion of the lineage distance parameter, and improves the detection accuracy in scenarios with loop-containing lineage graphs.

[0050] S200: Obtain multiple detection snapshot results corresponding to multiple upstream nodes within the current scheduling period.

[0051] In this embodiment, the current scheduling period is the business date (bizDate) range of the data task corresponding to the suspected abnormal node, which is used to ensure that the detection results collected by all upstream nodes are in the same time dimension.

[0052] In this embodiment, one-to-one correspondence means that each upstream node has a uniquely matched detection snapshot result, and there is no situation where an upstream node has no detection result or multiple upstream nodes share the same detection result.

[0053] In this embodiment, the detection snapshot result is a complete record of the detection results, abnormal confidence scores and indicator types of all detection indicators of a certain node on a specified business date, which serves as the basic data for collecting upstream information during the lineage tracing process.

[0054] In this embodiment, the method can collect the detection snapshot results of upstream nodes on the same business date to provide a data foundation for subsequent multi-node joint verification.

[0055] In this embodiment, the snapshot results are pre-calculated and persistently stored by the data quality detection module in each scheduling cycle. Each snapshot record includes at least one or more of the following: node identifier, business date, anomaly confidence score of each detection indicator, indicator fluctuation range of each detection indicator, and detection indicator type. Among them, the indicator fluctuation range is generated synchronously by the data quality detection module during detection calculation and stored along with the snapshot for direct reading during lineage tracing.

[0056] As an optional implementation, obtaining multiple detection snapshot results corresponding to each of the multiple upstream nodes within the current scheduling period further includes: obtaining the instance identifier of the current task instance of the suspected abnormal node, where the instance identifier is a unique number that distinguishes multiple running instances of the data task under the same business date within the current scheduling period; using the instance identifier as the search key, searching along the lineage to find the corresponding instance identifier of each upstream node that is under the same data version link as the current task instance; querying the detection snapshot repository using the corresponding instance identifier to obtain the detection snapshot results of each upstream node within the current scheduling period; if there is no detection snapshot record for an upstream node under the current business date, automatically reverting to the most recent available detection snapshot of that upstream node in the historical scheduling period as the alternative detection snapshot; when calculating the comprehensive confidence score, applying a time decay correction factor to the abnormal confidence score corresponding to the alternative detection snapshot, where the time decay correction factor is calculated according to β^Δt, where Δt is the number of days the business date of the alternative detection snapshot is reverted from the current business date, and β is a preset time decay coefficient and β∈(0,1).

[0057] In the above implementation, upstream nodes and the current node may have multiple physical running instances on the same business date due to scheduling delays, task reruns, etc. If the detection snapshot is directly matched according to the business date, the detection results may be misaligned with the actual data version of the current node. In addition, some upstream nodes may not have a detection snapshot record in the current period due to incomplete dependencies or task failures. If it is directly determined that there is no upstream information, valuable detection evidence will be lost. This implementation traces the corresponding instances under the same data version along the lineage link by instance identifier, ensuring that the collected upstream detection snapshots are in the same data version link as the current node, eliminating the snapshot mismatch problem caused by task reruns and scheduling delays. At the same time, the snapshot rollback mechanism automatically selects the most recent available historical snapshot as a substitute when there is no current snapshot, and introduces time decay correction so that the confidence contribution of the rollback snapshot decreases exponentially with the increase of rollback days, ensuring a reasonable balance between the availability and timeliness of upstream detection information, and improving the robustness and detection coverage of the solution in the scenario of unstable scheduling.

[0058] S300. Calculate the overall confidence level of suspected abnormal nodes based on the results of multiple detection snapshots.

[0059] In this embodiment, the overall confidence score is the final anomaly confidence score calculated by combining the current node's own detection results with the detection results of all upstream nodes, and is used to drive alarm decisions.

[0060] In this embodiment, the method can use a three-dimensional factor-weighted comprehensive confidence score calculation method to comprehensively consider factors such as bloodline distance attenuation, upstream fluctuation amplitude, and detection index type weights, and dynamically balance the influence of self-detection and upstream detection to achieve accurate multi-node joint inference.

[0061] S400. Obtain the anomaly confidence level of suspected abnormal nodes based on the comprehensive confidence level.

[0062] In this embodiment, the abnormal confidence level is an alarm priority level divided according to the threshold range of the comprehensive confidence level, which is used to drive the hierarchical alarm decision.

[0063] In this embodiment, a tiered alarm decision is made based on the overall confidence level: When there is high confidence (S≥high threshold), an alarm signal is generated or an alarm process is triggered, and root cause candidate nodes and propagation links are marked. An alarm is triggered and marked as requiring manual verification when the threshold is medium confidence (medium threshold ≤ S < high threshold); No alarm is triggered when there is low confidence (S < low threshold), which is considered false alarm suppression. The threshold parameter can be configured uniformly by the platform, and can be set differently according to business domain or table level.

[0064] Wherein, S is the comprehensive confidence score, which is the final anomaly confidence score calculated by weighting and summing the current node's own detection results with the detection results of all upstream nodes. The value range is normalized to the interval [0,1] and is used to drive alarm decisions.

[0065] Meanwhile, the threshold parameters for high, medium, and low thresholds can be uniformly configured by the platform, which supports differentiated settings based on business domain or table level.

[0066] In some embodiments, obtaining the anomaly confidence level of a suspected abnormal node based on a comprehensive confidence score includes: obtaining the business domain identifier of the business domain to which the suspected abnormal node belongs; querying a domain configuration table based on the business domain identifier to obtain a first confidence threshold and a second confidence threshold corresponding to the business domain, wherein the first confidence threshold is higher than the second confidence threshold; comparing the comprehensive confidence score with the first confidence threshold and the second confidence threshold to determine the anomaly confidence level; wherein the first confidence threshold and the second confidence threshold are configured independently for each business domain; the first confidence threshold for core business domains is lower than the first confidence threshold for non-core business domains, the core business domain is a pre-configured set of business domain identifiers involving financial settlement, regulatory reporting, or core user transactions, and the non-core business domains are other business domains besides the core business domains.

[0067] In the above implementation, different business domains exhibit fundamentally different fault tolerance capabilities and business impacts regarding data quality failures. Core business domains (such as financial settlement and regulatory reporting) are constrained by compliance audits and have extremely low tolerance for missed alarms. Failure to detect data anomalies in a timely manner can lead to serious business risks or compliance issues. In contrast, exploratory analytics or internal operations domains have a higher tolerance for occasional false alarms, and frequent low-value alerts can distract operations personnel. Using a globally uniform threshold configuration would fail to simultaneously address the dual needs of "strictly preventing missed alarms in core domains" and "suppressing false alarms in non-core domains." This implementation addresses this by configuring independent thresholds at the business domain level, enabling precise matching of alarm decisions with business importance: core business domains are configured with lower high-confidence thresholds to reduce the risk of missed alarms and ensure timely detection of anomalies in critical data links; non-core business domains are configured with higher confidence thresholds to suppress invalid alarm noise and reduce unnecessary occupation of operations resources. This allows for a global optimization of the balance between alarm effectiveness and operations resource allocation efficiency.

[0068] In the above embodiments, this method upgrades traditional single-point isolated time-series detection to multi-node joint inference based on data lineage. It efficiently collects detection snapshot results of upstream nodes along the lineage directed graph through breadth-first search and calculates the comprehensive confidence score accordingly. This introduces "external evidence" from the entire upstream of the chain for cross-validation of suspected abnormal nodes, thereby effectively distinguishing between normal business fluctuations and real data failures, and significantly reducing the high false alarm rate caused by single-point time-series detection. At the same time, classifying anomaly confidence levels based on comprehensive confidence scores makes alarm decisions more interpretable and hierarchically operable. This avoids the ineffective interference of low-confidence anomalies to maintenance personnel and provides a quantitative basis for rapid response to high-confidence failures, thereby improving the detection accuracy and maintenance efficiency in the scenario of automatic data quality discovery.

[0069] In some embodiments, step S100 may include: S110. When a suspected abnormal node is detected, determine whether the suspected abnormal node has a reachable upstream node in the bloodline directed graph. If yes, proceed to step S120; otherwise, proceed to step S130.

[0070] In this embodiment, the reachable upstream node is the data production task node that can be traced from the current node along the upstream direction through one or more directed edges in the bloodline directed graph.

[0071] S120. Starting from the suspected abnormal node, perform a breadth-first search based on the kinship directed graph to obtain multiple upstream nodes; the breadth-first search is equipped with a maximum backtracking depth limit and a loop prevention mechanism for the set of visited nodes.

[0072] In this embodiment, the maximum backtracking depth is limited to the maximum number of upstream traversals set during the breadth-first search process. This is used to control the traversal range and avoid excessive computational overhead in extremely deep lineage links.

[0073] In this embodiment, the anti-loop mechanism for the visited node set is to maintain the visited node set during the breadth-first search process, and to prevent visited nodes from being added to the queue again, in order to prevent infinite traversal caused by loops in the lineage graph.

[0074] In this embodiment, the lineage backtracking traversal adopts a breadth-first search (BFS) strategy, starting from the trigger node and traversing upstream layer by layer along the directed graph of data lineage. The maximum backtracking depth limit is used to prevent excessive computational overhead in extremely deep lineage links, and the loop prevention mechanism of the visited node set is used to prevent infinite traversal caused by loops in the lineage graph. This ensures that the search process can stably and reliably complete the upstream node collection under the premise of controllable computational overhead.

[0075] S130. Obtain the self-abnormal confidence score of the suspected abnormal node, and normalize the self-abnormal confidence score to obtain the comprehensive confidence level of the suspected abnormal node.

[0076] In this embodiment, the self-abnormal confidence is divided into an anomaly confidence score that relies solely on the current node's own detection indicators output by the time-series anomaly detection algorithm, without introducing any detection information from upstream nodes.

[0077] In this embodiment, the normalization process is to convert the abnormal confidence score of itself to a uniform dimension score in the [0,1] interval according to a preset mapping rule, so that the confidence scores of different detection indicators and different nodes are comparable and used for subsequent calculation of comprehensive confidence.

[0078] In this embodiment, if the current node has no reachable upstream node in the lineage graph, the lineage backtracking step is skipped, and the overall confidence level degrades to relying solely on its own detection results. This ensures that the detection capability does not fail when lineage information is missing or upstream nodes are not connected to the detection, thus exhibiting good engineering robustness.

[0079] In the above embodiments, this method can provide stable and reliable upstream detection evidence for comprehensive confidence calculation while ensuring that the pedigree tracing process is controllable and efficient, thereby balancing the improvement of detection accuracy and the stability of system performance. In scenarios where pedigree information is missing or there is no reachable upstream node, it automatically degrades to a single-node detection mode to ensure that the detection capability does not fail due to incomplete pedigree data, thus guaranteeing the engineering robustness and reliability of the overall solution under different data environments.

[0080] In some embodiments, step S300 may include: S310. Based on the results of multiple detection snapshots, obtain multiple fluctuation amplitude factors, multiple detection index type coefficients, and multiple anomaly confidence scores that correspond one-to-one with multiple upstream nodes.

[0081] In this embodiment, the fluctuation amplitude factor is a weighted coefficient obtained by mapping the fluctuation of each detection indicator of the upstream node through a piecewise function. The greater the fluctuation amplitude, the stronger the explanatory power of the upstream node for the current node's anomaly, and the higher the factor value.

[0082] In this embodiment, the detection indicator type coefficient is a preset weighting coefficient based on the confidence contribution of different detection indicator types to anomaly diagnosis. The confidence contribution of row count indicators is the highest, followed by field indicators, and business indicators vary depending on the rules.

[0083] To ensure the reasonableness of the weight calculation, step S310 may also include: S311. Based on the results of multiple detection snapshots, obtain the fluctuation range of multiple indicators, the types of multiple detection indicators, and the confidence scores of multiple anomalies that correspond one-to-one with multiple upstream nodes.

[0084] S312. Based on the fluctuation range of multiple indicators, use a piecewise function to calculate multiple fluctuation range factors that correspond one-to-one with multiple upstream nodes.

[0085] In this embodiment, a piecewise function is used to calculate the fluctuation amplitude factor: When the fluctuation range of the indicator reaches an extreme threshold (such as when the entire table is empty or the entire column is full of empty values), the factor takes the maximum value of 1.0. For moderate amplitudes, linear or nonlinear mappings are applied to the (0,1) interval; Take the lower value when there are slight fluctuations.

[0086] Among them, the greater the fluctuation amplitude, the stronger the explanatory power of the upstream node for the current node's anomaly, and the higher the weight.

[0087] As an optional implementation, the piecewise function includes extreme thresholds and medium thresholds: When the fluctuation range of the indicator is greater than or equal to the extreme threshold, the fluctuation range factor takes the maximum value. When the fluctuation range of the indicator is less than the extreme threshold and greater than or equal to the medium threshold, the fluctuation range factor increases monotonically as the fluctuation range of the indicator increases. When the fluctuation range of the indicator is less than the medium threshold, the fluctuation range factor takes the preset lower limit value.

[0088] S313. Based on multiple detection index types and type coefficient calibration tables, obtain multiple detection index type coefficients that correspond one-to-one with multiple upstream nodes; wherein, the type coefficient calibration table is obtained by backtesting and optimizing the detection effect of historical abnormal samples.

[0089] In this embodiment, the type coefficient calibration table is a preset configuration table that records the mapping relationship between each detection index type and its corresponding weight coefficient. It is used to quickly query the type coefficient of a node based on its detection index type. The weight coefficients corresponding to each detection index type in the type coefficient calibration table are obtained by applying different candidate coefficient combinations to historical abnormal samples for backtracking detection and iteratively adjusting them with the detection accuracy as the optimization objective.

[0090] In this embodiment, the historical abnormal samples are a set of node detection records that have been confirmed as real data quality faults in the platform's historical operation, which are used as the data basis for type coefficient calibration.

[0091] In this embodiment, the backtesting optimization of the detection effect is to apply different types of coefficient combinations to historical abnormal samples for retrospective detection, and to iteratively adjust the values ​​of each type of coefficient with the detection accuracy as the optimization goal, so that the coefficient configuration in the type coefficient calibration table is close to the abnormal diagnosis rules in real scenarios.

[0092] In this embodiment, the method can use a differentiated weight configuration mechanism for multiple detection indicator types. The confidence contribution of row count indicators (such as the entire table being empty or a sudden drop in the number of rows) is the highest, and its coefficient is taken as the maximum value; the confidence contribution of field indicators (such as null value rate, empty character rate, and mean deviation) is the second highest, and its coefficient is moderate; the confidence contribution of business indicators (custom business rules) varies depending on the rule, and the default coefficient is low.

[0093] In this embodiment, the coefficients of each type are calibrated through backtesting using historical data.

[0094] S320. Obtain multiple lineage distance decay factors between multiple upstream nodes and suspected abnormal nodes.

[0095] In this embodiment, the bloodline distance attenuation factor is a weighting coefficient calculated by using an exponential attenuation function based on the number of bloodline layers between the upstream node and the current node. It is used to quantify the degree of attenuation of the detection result of the upstream node with a greater bloodline distance being weaker than that of the current node.

[0096] To better reflect the actual pattern of fault propagation with link attenuation, step S320 may further include: S321. Obtain the number of lineage layers between multiple upstream nodes and suspected abnormal nodes.

[0097] In this embodiment, the number of bloodline layers is the number of directed edges traversed from the current node to an upstream node in the directed bloodline graph, which is used to quantify the bloodline distance between the upstream node and the current node.

[0098] S322. Based on the number of bloodline layers, use the exponential decay function to calculate multiple bloodline distance decay factors that correspond one-to-one with multiple upstream nodes.

[0099] In this embodiment, the further the blood relationship of the upstream node, the weaker the correlation between its detection result and the current node, and the weight is correspondingly reduced.

[0100] For example, the bloodline distance attenuation factor is calculated using an exponential attenuation function α^depth.

[0101] Where depth is the number of bloodline layers between the upstream node and the current node, and α is the attenuation coefficient (α∈(0,1), which can be uniformly configured by the platform). The greater the depth, the smaller the attenuation factor. The further the upstream node is from the current node, the weaker the correlation between its detection result and the current node, and the weight is correspondingly reduced.

[0102] S330. Based on multiple fluctuation amplitude factors, multiple detection index type coefficients, and multiple bloodline distance attenuation factors, calculate multiple initial weights that correspond one-to-one with multiple upstream nodes.

[0103] In this embodiment, calculation refers to multiplying the fluctuation amplitude factor, detection index type coefficient, and lineage distance attenuation factor of a single upstream node to obtain the initial weight of the upstream node.

[0104] In this embodiment, the initial weight is the original weight value obtained by multiplying the fluctuation amplitude factor of a single upstream node, the detection index type coefficient, and the bloodline distance attenuation factor, which is used for subsequent normalization processing.

[0105] In some implementations, calculating the initial weights corresponding to each of the multiple upstream nodes further includes: determining whether the fluctuation range of the indicators of each upstream node reaches a preset extreme threshold; when the fluctuation range of the indicators of a certain upstream node reaches the preset extreme threshold, adjusting the bloodline distance attenuation factor of the upstream node upward according to a preset enhancement coefficient, and the adjusted bloodline distance attenuation factor does not exceed 1.0; wherein, the value range of the preset enhancement coefficient is greater than 1 and less than or equal to 2.

[0106] In the above implementation, when an upstream node experiences extreme fluctuations such as an entire table being empty or an entire column containing null values, the transmission of this anomaly to downstream data quality is highly deterministic. Even if the node is far from the lineage of the current suspected anomaly node, its anomaly signal still has strong cross-layer transmission characteristics and should not be excessively attenuated due to increased lineage distance. If a uniform exponential decay mechanism is adopted, the attenuation rate of extreme fluctuation nodes and ordinary fluctuation nodes is the same with increasing lineage distance, which will cause the extreme anomaly signal at a greater distance to be weakened to the same order of magnitude as the ordinary fluctuation signal, thereby increasing the risk of missed detection in root cause identification. This implementation applies an additional boosting coefficient to the distance attenuation factor of extreme fluctuation nodes, enabling them to obtain a higher weight contribution at the same lineage distance. This effectively preserves the anomaly signal strength even when the root cause node is far from the current node, reducing the risk of extreme root cause anomalies being overwhelmed by excessive distance attenuation.

[0107] S340. Normalize the multiple initial weights to obtain multiple final weights that correspond one-to-one with the multiple upstream nodes.

[0108] In this embodiment, the method can perform weight normalization processing on all upstream nodes involved in the inference, dividing the original weight of each node by the sum of the original weights of all upstream nodes, so that the sum of the weights of each upstream node is 1, ensuring that the overall confidence level is comparable under a unified dimension.

[0109] S350: Obtain the self-abnormal confidence score and self-weight of the suspected abnormal node in the current scheduling period.

[0110] In this embodiment, the self-weight is the weight ratio coefficient of the current node's own detection result in the comprehensive confidence calculation, which is complementary to the sum of the final weights of all upstream nodes (the sum of the two is 1), and is used to dynamically balance the contribution of self-detection and upstream joint inference to the final confidence.

[0111] In some embodiments, obtaining the self-abnormal confidence score and self-weight of the suspected abnormal node within the current scheduling period further includes: obtaining the number of high-confidence upstream nodes whose abnormal confidence scores exceed a preset high-confidence threshold, and the number of low-confidence upstream nodes whose abnormal confidence scores are lower than a preset low-confidence threshold; calculating the ratio of the number of high-confidence upstream nodes to the total number of the multiple upstream nodes as the high-confidence percentage; calculating the ratio of the number of low-confidence upstream nodes to the total number of the multiple upstream nodes as the low-confidence percentage; when the high-confidence percentage exceeds a preset high-confidence percentage threshold, setting the self-weight of the suspected abnormal node to a first weight lower limit; when the low-confidence percentage exceeds a preset low-confidence percentage threshold, setting the self-weight of the suspected abnormal node to a first weight upper limit; wherein the first weight upper limit is greater than the first weight lower limit.

[0112] In the above implementation, if the self-weight is adjusted solely based on the number of upstream nodes, the same self-weight will be output in two scenarios where the number of high-confidence upstream nodes and low-confidence upstream nodes are the same but their quality differs greatly. This fails to reflect the actual impact of the quality of upstream detection information on the overall confidence judgment. The large presence of high-confidence upstream nodes implies sufficient and reliable evidence for joint inference, and the overall contribution of upstream weights should be further amplified. Conversely, the large presence of low-confidence upstream nodes implies weak evidence for joint inference or even the introduction of interference. In this case, over-reliance on upstream nodes would reduce the accuracy of the judgment. This implementation introduces two quality dimensions, high-confidence percentage and low-confidence percentage, which, in addition to the sufficiency of quantity, also consider the credibility of information. This allows the allocation of self-weights and upstream weights to be simultaneously regulated by both the "quantity" and "quality" of upstream information. In scenarios where the proportion of low-quality upstream information is high, it can effectively suppress the negative impact of interference signals on the overall confidence, improving the adaptability and robustness of the joint inference mechanism under different upstream quality distributions.

[0113] To avoid the bias of over-reliance on one's own judgment when upstream information is abundant, and to mitigate the risk of misjudgment caused by excessively high weights in joint inference when upstream information is scarce, step S350 may also include: S351. Obtain the total number of upstream nodes and the self-detection results of suspected abnormal nodes within the current scheduling cycle.

[0114] In this embodiment, the self-detection result is the abnormal status and corresponding abnormal confidence score of each detection indicator of the current node in the current scheduling period, output by the time-series anomaly detection algorithm.

[0115] S352. Determine whether the total number of nodes has reached the preset sufficient threshold. If yes, proceed to step S353; otherwise, proceed to steps S354~S355.

[0116] In this embodiment, the total number of nodes is the number of all reachable upstream nodes collected during the kinship tracing process.

[0117] In this embodiment, the preset sufficiency threshold is a preset threshold value used to determine whether the number of upstream nodes is sufficient to support multi-node joint inference. When the total number of nodes reaches the threshold, the upstream information is considered sufficient; otherwise, the upstream information is considered insufficient.

[0118] In this embodiment, the preset sufficient threshold can be set by the platform operation and maintenance personnel or algorithm engineers based on their experience with the actual distribution of the lineage map.

[0119] S353. Set the preset weight lower limit as the weight of the suspected abnormal node.

[0120] In this embodiment, the preset weight lower limit is the minimum value that the weight can take. When there are enough upstream nodes, the weight is fixed at this lower limit value to ensure that the upstream joint inference contributes enough to the overall confidence and to avoid the self-detection result from overly dominating the judgment.

[0121] In this embodiment, when there are sufficient upstream nodes, the total weight of the upstream nodes is appropriately increased so that the joint judgment conclusion of the upstream nodes contributes more to the overall confidence. When there are insufficient upstream nodes, their own weight is close to 1, and the overall confidence degrades to a result close to that of a single node detection, thus achieving graceful degradation without loss of detection capability.

[0122] S354. Select the weights corresponding to the total number of nodes from the weight configuration table and use them as the weights of the suspected abnormal nodes.

[0123] In this embodiment, the weight configuration table is a preset mapping table that records the weight values ​​that should be taken for each upstream node under different total number of upstream nodes. It is used to quickly determine the weight of itself based on the actual number of upstream nodes collected when there are insufficient upstream nodes.

[0124] S355. Obtain the self-abnormal confidence score of suspected abnormal nodes based on their own detection results; wherein, the weight in the weight configuration table decreases as the total number of nodes increases, and the self-weight is complementary to the sum of multiple final weights.

[0125] S360. Calculate the initial confidence level of the suspected abnormal node based on its own weight, its own anomaly confidence score, multiple final weights, and multiple anomaly confidence scores.

[0126] In this embodiment, the initial confidence score is the uncorrected anomaly confidence score obtained by weighting and summing the detection results of the self and all upstream nodes according to their respective weights, and it serves as the basic input for subsequent correction processing.

[0127] In this embodiment, the formula for calculating the initial confidence level is: S=W_self*score_self+Σ(w_i*score_i); Where score_self is the current node's own anomaly confidence score; score_i is the anomaly confidence score of the i-th upstream node; w_i is the final weight of the i-th upstream node after normalization; W_self is the weight of the current node itself, and it is complementary to the sum of the final weights of all upstream nodes (the sum of the two is 1).

[0128] In this embodiment, when the current node is a suspected abnormal node, the calculated S is the initial confidence level of the suspected abnormal node.

[0129] In this embodiment, when there are enough upstream nodes, the total weight of the upstream nodes 1-W_self is appropriately increased so that the joint judgment conclusion of the upstream nodes contributes more to the overall confidence. When there are insufficient upstream nodes, W_self approaches 1, and the overall confidence level degrades to a result close to that of a single-node detection, thus achieving graceful degradation.

[0130] Finally, the overall confidence level S is normalized to the [0,1] interval for use in alarm decision-making.

[0131] S370. The initial confidence level is corrected to obtain the comprehensive confidence level.

[0132] To ensure that the overall confidence level more accurately reflects the true propagation status of the data issue, step S370 may further include: S371. Mark the nodes with extreme fluctuation characteristics among multiple upstream nodes that cannot be traced further upstream as root cause candidate nodes.

[0133] In this embodiment, the criteria for determining root cause candidate nodes include: Extreme fluctuation characteristics, that is, the node detection indicators show extreme anomalies such as the entire table being empty or the entire column being empty, and the fluctuation range of the indicators reaches the extreme threshold; And there are abnormal upstream nodes with no further tracing, meaning that none of the current node's direct upstream nodes were marked as abnormal on the same business date, or the current node is already the top-level node in the lineage graph. If either condition is met, it is marked as a root cause candidate node, realizing automatic identification and rapid location of root cause candidate nodes in the fault chain.

[0134] In this embodiment, during the BFS backtracking traversal, when an upstream node is captured, the system queries the detection snapshots of all its direct upstream nodes. If none of the direct upstream nodes have any anomaly detection records or their anomaly confidence scores are lower than a preset anomaly judgment threshold, then the system determines that the anomaly of that node cannot be explained by the upstream nodes. This determination is performed in real time during the BFS traversal, without requiring an additional one-time full-graph scan.

[0135] S372. Obtain the lineage propagation chain of root cause candidate nodes.

[0136] In this embodiment, the “propagation chain” has been explained in S371. Here, the “bloodline propagation chain” is the same concept (i.e., the complete fault path that spreads downstream from the root cause candidate node through the data bloodline).

[0137] S373. The initial confidence level is corrected according to the bloodline transmission chain to obtain the comprehensive confidence level.

[0138] In this embodiment, the correction process involves adjusting the initial confidence level by positively adding or negatively attenuating it based on the positional relationship between the suspected abnormal node and the root cause candidate node.

[0139] In this embodiment, when a node is marked as a root cause candidate node, all downstream nodes in its propagation chain give a positive weight to the root cause candidate node in the comprehensive confidence calculation, thereby increasing the comprehensive confidence and reducing false alarms, downgrades, and misjudgments. For nodes that are found to be normal in terms of upstream nodes after backtracking and have only slight fluctuations in their own indicators, a negative correction is made to reduce the comprehensive confidence and reduce false alarms related to business fluctuations.

[0140] As an optional implementation, step S373 includes: Determine whether a suspected abnormal node is an affected downstream node based on the bloodline transmission chain; If the suspected anomalous node is an affected downstream node, the initial confidence level of the suspected anomalous node is positively corrected according to the positive correction coefficient to obtain the comprehensive confidence level. If the suspected abnormal node is not an affected downstream node, then determine whether the suspected abnormal node is a slightly fluctuating node without upstream support based on the bloodline propagation chain. If the suspected anomalous node is a node with slight fluctuations, the initial confidence level of the suspected anomalous node is negatively corrected according to the negative correction coefficient to obtain the comprehensive confidence level. Among them, the affected downstream nodes are those located downstream of the root cause candidate nodes and those that have been detected as potentially abnormal. The positive correction coefficient is negatively correlated with the kinship distance between the suspected anomalous node and the root cause candidate node.

[0141] In the above embodiments, the method can dynamically balance the contributions of upstream and downstream detection evidence by integrating the weighted calculation of multi-dimensional factors and the adaptive correction mechanism, thereby significantly improving the accuracy and reliability of anomaly judgment under complex lineage topology.

[0142] In some embodiments, after step S400, the method may further include: S510. Mark the nodes with extreme fluctuation characteristics among multiple upstream nodes that cannot be traced further upstream as root cause candidate nodes.

[0143] S520. Determine whether there is a downstream node to be alarmed in the lineage propagation chain of the root cause candidate node. If yes, proceed to step S530. If no, directly output the abnormal alarm information with the second alarm information of the root cause candidate node itself, without the need for subsequent aggregation processing.

[0144] In this embodiment, the downstream node to be alarmed is a node located downstream of the root cause candidate node lineage propagation chain and whose abnormal confidence level meets the preset alarm level.

[0145] S530: Obtain the list of downstream nodes to be alarmed, the first alarm information of the downstream nodes to be alarmed, and the second alarm information of the root cause candidate nodes.

[0146] In this embodiment, the node list is a complete list of downstream nodes to be alarmed, which is used to clearly display the range of all downstream nodes affected by the same root cause in the aggregated alarm.

[0147] S540. Using the root cause candidate node as the aggregation key, aggregate the first alarm information and the second alarm information to obtain aggregated alarm information.

[0148] In this embodiment, the aggregation key is an associated field used to merge multiple alarms into the same main alarm, with the unique identifier of the root cause candidate node as the key value.

[0149] In this embodiment, the aggregation process is the operation of merging multiple alarms under the same aggregation key into a single main alarm.

[0150] In this embodiment, the aggregated alarm information is an alarm data structure generated by merging the first alarm information and the second alarm information by an aggregation key, which includes information such as root cause candidate node identifier, list of affected downstream nodes and fault type.

[0151] S550: Output abnormal alarm information based on the aggregated alarm information and the node list of downstream nodes to be alarmed.

[0152] In this embodiment, the downstream node to be alarmed is a node whose abnormal confidence level meets the preset alarm level; the abnormal alarm information includes the identifier of the root cause candidate node, the fault type of each downstream node to be alarmed, the node list of the downstream nodes to be alarmed, and the comprehensive confidence level of the downstream nodes to be alarmed.

[0153] In this embodiment, the abnormal alarm information is the final output alarm content, including the identifier of the root cause candidate node, the fault type of each downstream node to be alarmed, the node list of downstream nodes to be alarmed and the comprehensive confidence level, which makes it easier for on-duty personnel to quickly locate the source of the fault and grasp the complete scope of impact.

[0154] In this embodiment, the method can use root cause candidate nodes as aggregation keys to collect all associated downstream alarms, aggregate multiple downstream alarms under the same root cause candidate node into a main alarm and list the affected downstream nodes, thereby achieving alarm noise reduction and aggregation based on the bloodline propagation chain.

[0155] In the above embodiments, the method can aggregate multiple downstream alarms caused by the same root cause into a single main alarm, so that on-duty personnel do not have to deal with a large number of repetitive and scattered alarm messages, thereby directly locating the source of the fault and grasping the complete scope of impact, thus significantly improving the efficiency of fault response and the operation and maintenance experience.

[0156] In some embodiments, since over 85% of data quality issues can be traced back to extreme fluctuations in an upstream root cause candidate node, and all intermediate nodes in the complete propagation chain from the root cause candidate node to the target node are typically simultaneously marked as anomalies by time-series detection, this application, based on the above principles, achieves a systematic improvement in detection accuracy without relying on additional manually labeled data.

[0157] Meanwhile, this application differs from existing time-series enhancement schemes (which introduce features such as holiday factors and business calendars) in that the latter involve vertical in-depth analysis (modeling by enriching the time-series features of a single node), while this application involves horizontal correlation verification (multi-dimensional verification and correction of detection conclusions through cross-node lineage joint inference). Therefore, these two approaches are logically orthogonal and can be combined complementaryly.

[0158] In addition, this application relies on the platform's existing data lineage metadata service, data quality detection index snapshot storage service, and alarm delivery service, and can be incrementally extended on the existing platform infrastructure without introducing additional external components.

[0159] like Figure 2 As shown, some embodiments of this application provide a schematic diagram of a node anomaly detection device. It should be understood that this device is related to... Figure 1The method executed in the middle corresponds to the steps involved in the aforementioned method. The specific functions and effects of the device can be found in the description above. To avoid repetition, detailed descriptions are omitted here.

[0160] The node anomaly detection device includes: Search unit 610 is used to perform a breadth-first search based on the kinship directed graph, starting from the suspected abnormal node, when a suspected abnormal node is detected, to obtain multiple upstream nodes. The first acquisition unit 620 is used to acquire multiple detection snapshot results corresponding to multiple upstream nodes in the current scheduling period; The calculation unit 630 is used to calculate the comprehensive confidence level of suspected abnormal nodes based on the results of multiple detection snapshots; The second acquisition unit 640 is used to obtain the anomaly confidence level of the suspected abnormal node based on the comprehensive confidence level.

[0161] In this embodiment, the functions of each unit in the device correspond one-to-one with the specific implementation of the corresponding steps (S100 to S400) in the aforementioned method, and can complete the entire process from triggering suspected abnormal nodes, collecting upstream detection snapshots through lineage tracing, calculating comprehensive confidence level to obtaining abnormal confidence level, which will not be elaborated here.

[0162] like Figure 3 As shown, this application provides an electronic device 700, which includes a processor 701 and a memory 702. The processor 701 and the memory 702 are interconnected and communicate with each other through a communication bus 703 and / or other forms of connection mechanism (not shown). The memory 702 stores a computer program that can be executed by the processor 701. When the computer program is run, the processor 701 executes the computer program to perform the method in any of the aforementioned optional implementations.

[0163] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method in any of the aforementioned optional implementations.

[0164] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0165] This application provides a computer program product, which includes a computer program that, when run by a processor, executes the method in any of the aforementioned optional implementations.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for detecting node anomalies, characterized in that, include: When a suspected abnormal node is detected, a breadth-first search is performed based on the bloodline directed graph, starting from the suspected abnormal node, to obtain multiple upstream nodes. Obtain multiple detection snapshot results corresponding to the multiple upstream nodes within the current scheduling period; The overall confidence level of the suspected abnormal node is calculated based on the results of the multiple detection snapshots; The anomaly confidence level of the suspected abnormal node is obtained based on the comprehensive confidence level.

2. The node anomaly detection method according to claim 1, characterized in that, When a suspected abnormal node is detected, a breadth-first search is performed based on the kinship directed graph, starting from the suspected abnormal node, to obtain multiple upstream nodes, including: When a suspected abnormal node is detected, it is determined whether the suspected abnormal node has a reachable upstream node in the bloodline directed graph; If it exists, then starting from the suspected abnormal node, perform a breadth-first search based on the bloodline directed graph to obtain multiple upstream nodes; The breadth-first search is equipped with a maximum backtracking depth limit and a loop prevention mechanism for the set of visited nodes.

3. The node anomaly detection method according to claim 2, characterized in that, The method further includes: If the suspected abnormal node has no reachable upstream node in the kinship directed graph, then the self-abnormality confidence score of the suspected abnormal node is obtained, and the self-abnormality confidence score is normalized to obtain the comprehensive confidence score of the suspected abnormal node.

4. The node anomaly detection method according to claim 1, characterized in that, The step of calculating the overall confidence level of the suspected abnormal node based on the multiple detection snapshot results includes: Based on the multiple detection snapshot results, obtain multiple fluctuation amplitude factors, multiple detection index type coefficients, and multiple anomaly confidence scores corresponding to the multiple upstream nodes; Obtain multiple lineage distance attenuation factors between the multiple upstream nodes and the suspected abnormal nodes; Based on the multiple fluctuation amplitude factors, the multiple detection index type coefficients, and the multiple bloodline distance attenuation factors, calculate multiple initial weights that correspond one-to-one with the multiple upstream nodes; The initial weights are normalized to obtain a set of final weights that correspond one-to-one with the upstream nodes. Obtain the self-abnormal confidence score and self-weight of the suspected abnormal node within the current scheduling period; The initial confidence level of the suspected abnormal node is calculated based on its own weight, its own anomaly confidence score, multiple final weights, and multiple anomaly confidence scores. The initial confidence level is then corrected to obtain the overall confidence level.

5. The node anomaly detection method according to claim 4, characterized in that, The step of obtaining multiple fluctuation amplitude factors, multiple detection indicator type coefficients, and multiple anomaly confidence scores corresponding one-to-one with the multiple upstream nodes based on the multiple detection snapshot results includes: Based on the multiple detection snapshot results, obtain multiple indicator fluctuation amplitudes, multiple detection indicator types, and multiple anomaly confidence scores corresponding to the multiple upstream nodes; Based on the fluctuation range of the multiple indicators, a piecewise function is used to calculate multiple fluctuation range factors that correspond one-to-one with the multiple upstream nodes; Based on the multiple detection index types and type coefficient calibration table, obtain multiple detection index type coefficients that correspond one-to-one with the multiple upstream nodes; The type coefficient calibration table is obtained by backtesting and optimizing the detection effect of historical abnormal samples.

6. The node anomaly detection method according to claim 4, characterized in that, The step of obtaining multiple lineage distance attenuation factors between the multiple upstream nodes and the suspected abnormal nodes includes: Obtain the number of lineage layers between the multiple upstream nodes and the suspected abnormal nodes; Based on the number of bloodline layers, an exponential decay function is used to calculate multiple bloodline distance decay factors that correspond one-to-one with the multiple upstream nodes.

7. The node anomaly detection method according to claim 4, characterized in that, The step of obtaining the suspected abnormal node's own anomaly confidence score and its own weight within the current scheduling period includes: Obtain the total number of upstream nodes and the self-detection results of the suspected abnormal nodes within the current scheduling period; Determine whether the total number of nodes has reached a preset sufficient threshold; If so, the preset weight lower limit value is determined as the weight of the suspected abnormal node itself; If not, select the weight corresponding to the total number of nodes from the weight configuration table as the weight of the suspected abnormal node; The self-abnormality confidence score of the suspected abnormal node is obtained based on the self-detection results; The weights in the weight configuration table decrease as the total number of nodes increases, and the weight of a node itself is complementary to the sum of the multiple final weights.

8. The node anomaly detection method according to claim 4, characterized in that, The step of correcting the initial confidence level to obtain the comprehensive confidence level includes: Among the multiple upstream nodes, nodes with extreme fluctuation characteristics that cannot be traced further upstream are marked as root cause candidate nodes; wherein, the extreme fluctuation characteristics include at least one of the following: the fluctuation amplitude of the detection index reaches a preset extreme threshold, the entire table is empty, and the entire column is empty. Obtain the lineage propagation chain of the root cause candidate nodes; The initial confidence level is corrected based on the bloodline transmission chain to obtain the comprehensive confidence level.

9. The node anomaly detection method according to claim 8, characterized in that, The step of correcting the initial confidence level based on the bloodline transmission chain to obtain a comprehensive confidence level includes: Based on the bloodline transmission chain, determine whether the suspected abnormal node is an affected downstream node; If the suspected abnormal node is an affected downstream node, the initial confidence level of the suspected abnormal node is positively corrected according to the positive correction coefficient to obtain the comprehensive confidence level. If the suspected abnormal node is not an affected downstream node, then the bloodline propagation chain is used to determine whether the suspected abnormal node is a slightly fluctuating node without upstream support. If the suspected anomalous node is the slightly fluctuating node, then the initial confidence level of the suspected anomalous node is negatively corrected according to the negative correction coefficient to obtain the comprehensive confidence level. The affected downstream node is a node located downstream of the root cause candidate node that has been detected as potentially abnormal. The positive correction coefficient is negatively correlated with the kinship distance between the suspected abnormal node and the root cause candidate node.

10. The node anomaly detection method according to claim 1, characterized in that, After obtaining the anomaly confidence level of the suspected anomaly node based on the comprehensive confidence level, the method further includes: Among the multiple upstream nodes, nodes with extreme fluctuation characteristics and unable to be traced further upstream are marked as root cause candidate nodes; Determine whether there is a downstream node to be alarmed in the lineage propagation chain of the root cause candidate node; If there are downstream nodes to be alarmed, obtain the node list of the downstream nodes to be alarmed, the first alarm information of the downstream nodes to be alarmed, and the second alarm information of the root cause candidate nodes; Using the root cause candidate node as the aggregation key, the first alarm information and the second alarm information are aggregated to obtain aggregated alarm information; Based on the aggregated alarm information and the node list of downstream nodes to be alarmed, output abnormal alarm information; The downstream node to be alarmed is a node whose abnormal confidence level meets the preset alarm level. The abnormal alarm information includes the identifier of the root cause candidate node, the fault type of each downstream node to be alarmed, the node list of the downstream nodes to be alarmed, and the overall confidence level of the downstream nodes to be alarmed.

11. A node anomaly detection device, characterized in that, The node anomaly detection device includes: The search unit is used to perform a breadth-first search based on the kinship directed graph, starting from the suspected abnormal node, when a suspected abnormal node is detected, to obtain multiple upstream nodes. The first acquisition unit is used to acquire multiple detection snapshot results corresponding to the multiple upstream nodes in the current scheduling period; A calculation unit is used to calculate the overall confidence level of the suspected abnormal node based on the results of the multiple detection snapshots; The second acquisition unit is used to acquire the anomaly confidence level of the suspected abnormal node based on the comprehensive confidence level.

12. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the node anomaly detection method according to any one of claims 1 to 10.

13. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, performs the node anomaly detection method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the node anomaly detection method according to any one of claims 1 to 10.

Images