Automatic data exception root cause positioning and playback repair method and system and storage medium

Abstract

The invention provides an automatic data exception root cause positioning and playback repair method and system and related equipment. The method comprises the following steps: constructing a dynamic heterogeneous topological graph containing data logic nodes and physical resource nodes in real time, and establishing real-time directed dependency connection between the nodes; in response to a monitored abnormal signal, generating an anti-fact intervention set assuming that an upstream node is in a reference state by using a Do operator based on the dynamic heterogeneous topological graph; the anti-fact intervention set is substituted into a causal reasoning model for simulation, and the abnormal maintenance probability of the abnormal signal still existing under the condition that the anti-fact intervention set takes effect is obtained; and calculating a causal contribution degree according to the exception maintenance probability to lock a root cause node, and transmitting state data of the root cause node to a repair module to trigger playback repair. According to the invention, the accuracy and stability of abnormal root cause positioning are improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer information technology, and more specifically, to an automated method for locating and replayably repairing the root causes of data anomalies, an automated system for locating and replayably repairing the root causes of data anomalies, an electronic device, and a storage medium. Background Technology

[0002] Currently, with the widespread adoption of microservice architectures and large-scale data processing pipelines, business systems are often composed of numerous interdependent computing tasks, service instances, and underlying physical resources. To ensure stable system operation, the operations and maintenance (O&M) side typically deploys various monitoring and alerting mechanisms to collect operational metrics such as throughput, latency, error rate of each task, and CPU utilization and memory usage of the server, and records request chains and task execution information through a log system. When sudden changes in metrics or alarms are triggered, O&M personnel generally rely on log timestamps, monitoring curve patterns, and limited dependency information to troubleshoot and attempt to identify the root cause of the anomaly.

[0003] However, in existing technologies, the aforementioned root cause analysis process largely remains at the level of empirical judgment based on correlation. Specifically, on the one hand, the logical dependencies between data processing tasks and the operational status of underlying physical resources are usually recorded in different systems, lacking the ability to uniformly model data logical nodes and physical resource nodes. Operations personnel need to switch back and forth between multiple monitoring interfaces and log systems, manually comparing the fluctuations of indicators of different nodes within the same time period. On the other hand, when an abnormal signal occurs, multiple upstream task nodes and their dependent servers and containers may simultaneously experience increased latency or abnormal resource usage. Existing technologies often only perform rough sorting based on correlation clues such as simultaneous anomalies and similar curve shapes, making it difficult to distinguish which nodes are passively affected nodes in the anomaly propagation chain and which nodes are more likely to be key nodes that directly trigger the anomaly. This can easily lead to the mistaken identification of downstream affected nodes as root cause nodes.

[0004] Furthermore, existing technologies often fail to automatically and structurally organize the node's state data at the moment of the anomaly, such as configuration parameters, dependency library versions, runtime context, and input data locations. Instead, they rely on manual searching, copying, and recording of these states in different systems. Moreover, replay repairs typically involve manually writing scripts and manually loading historical data in test or pre-production environments for replay. This makes it difficult to reproduce the actual environment at the time of the anomaly in a timely and complete manner. It is also prone to errors due to missing key state data or selecting inconsistent runtime contexts, leading to discrepancies between the repair verification results and the real production environment. This can result in situations where problems cannot be reproduced or new anomalies are introduced after repair.

[0005] It should be noted that the statements in the background section above are only for providing background information related to this disclosure and do not necessarily constitute prior art. Summary of the Invention

[0006] To at least partially overcome the problems existing in the related technologies, embodiments of this disclosure provide an automated data anomaly root cause localization and replayable repair method, an automated data anomaly root cause localization and replayable repair system, an electronic device, and a storage medium.

[0007] According to one aspect of this disclosure, an automated method for locating the root cause of data anomalies and replayable repair is provided, comprising: A dynamic heterogeneous topology graph containing data logic nodes and physical resource nodes is constructed in real time, and real-time directed dependency connections are established between the nodes. In response to the detected abnormal signals, a set of counterfactual interventions assuming that the upstream nodes are in a baseline state is generated using the Do operator based on the dynamic heterogeneous topology graph. The counterfactual intervention set was substituted into the causal inference model for simulation, and the probability of the abnormal signal still existing under the condition that the counterfactual intervention set is effective was obtained. The root cause node is locked by calculating the causal contribution based on the anomaly maintenance probability, and the status data of the root cause node is transmitted to the repair module to trigger replay repair.

[0008] In one exemplary embodiment of this disclosure, the real-time construction of a dynamic heterogeneous topology graph including data logical nodes and physical resource nodes includes: Parse the task orchestration metadata of the data processing pipeline, extract the input / output patterns and code version information of each computing task, and generate a set of data logic nodes; Call the infrastructure monitoring interface to collect the CPU utilization, memory level and disk I / O metrics of the server container carrying the computing task, and generate a set of physical resource nodes; Based on the unique execution identifier of the computing task during runtime, the nodes in the physical resource node set are mapped and bound to the corresponding nodes in the data logic node set; The bound set of nodes is used as the vertices of the graph, and edge connections are established according to the data flow direction and resource call relationship to generate the dynamic heterogeneous topology graph.

[0009] In one exemplary embodiment of this disclosure, the step of mapping and binding nodes in the physical resource node set to corresponding nodes in the data logical node set includes: Extract the task start timestamp and task end timestamp of each node in the data logic node set; Based on the task start timestamp and task end timestamp, time window slicing is performed on the time series monitoring data of the physical resource node set; The physical resource state vector, after being processed by time window slicing, is written as an attribute feature into the corresponding logical node object, thus completing the binding of heterogeneous node attributes.

[0010] In one exemplary embodiment of this disclosure, calculating the causal contribution based on the anomaly maintenance probability includes: Obtain the baseline anomaly probability of the structured causal model under conditions without any counterfactual intervention; Calculate the probability difference between the anomaly maintenance probability and the baseline anomaly probability; The probability difference is input into a preset normalization function for weighted processing to obtain the causal contribution value of each upstream node to the current abnormal signal. A root cause candidate list is generated in descending order of the causal contribution values ​​for locking purposes.

[0011] In one exemplary embodiment of this disclosure, the triggering of playback repair includes: Read the configuration parameters, dependency library versions, and input data pointers of the locked root cause node at the time of the anomaly, and generate on-site environment metadata; Based on the aforementioned on-site environment metadata, a twin sandbox environment consistent with the production environment is launched in an isolated container cluster. The patch to be verified is injected into the twin sandbox environment, and the historical data stream is loaded according to the input data pointer; Perform deterministic replay in the twin sandbox environment and output the repair verification results after replay.

[0012] In one exemplary embodiment of this disclosure, the step of launching a twin sandbox environment consistent with the production environment in an isolated container cluster based on the on-site environment metadata includes: Identifier of the persistent storage volume mounted by the root cause node at the time of the anomaly; A storage snapshot of the persistent storage volume at the time of the anomaly is created using copy-on-write. Create a new container instance and mount the storage snapshot to the corresponding directory of the new container instance in read-only mode; The new container instance is launched as the basic operating unit of the twin sandbox environment.

[0013] In one exemplary embodiment of this disclosure, the triggering of playback repair includes: Extract the source code script corresponding to the locked root cause node and parse it into an abstract syntax tree; A preset mutation operator is applied to the condition judgment nodes and data transformation nodes in the abstract syntax tree to generate multiple candidate repair mutation trees; The multiple candidate repair mutation trees are compiled into executable code and deployed in parallel to multiple shadow processes; Abnormal sample data is injected into the multiple shadow processes simultaneously, and the shadow processes whose output results meet expectations are selected as the target repair solutions.

[0014] In one exemplary embodiment of this disclosure, locking the root cause node includes: Assign a unique dynamic taint label to the node with the highest causal contribution. In the dynamic heterogeneous topology graph, the system traverses downstream along the directed dependency connections and passes the dynamic taint label to all downstream nodes that directly or indirectly depend on the node. Identify all sets of nodes tagged with the dynamic taint label, and calculate the maximum propagation boundary of the node set in the graph; All nodes within the maximum propagation boundary are marked as affected ranges and output as part of the locking result.

[0015] According to one aspect of this disclosure, an automated data anomaly root cause localization and replayable repair system is provided, comprising: The topology graph construction module is used to construct a dynamic heterogeneous topology graph containing data logical nodes and physical resource nodes in real time, and to establish real-time directed dependency connections between the nodes. The counterfactual intervention module is used to respond to detected abnormal signals and generate a set of counterfactual interventions based on the dynamic heterogeneous topology graph using the Do operator, assuming that the upstream node is in the baseline state. An anomaly simulation module is used to substitute the counterfactual intervention set into the causal reasoning model for simulation, and to obtain the anomaly maintenance probability that the anomaly signal still exists under the condition that the counterfactual intervention set is effective. The replay repair module is used to calculate the causal contribution based on the anomaly maintenance probability to lock the root cause node, and transmit the status data of the root cause node to the repair module to trigger the replay repair.

[0016] According to one aspect of this disclosure, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to execute any of the methods described above by executing the executable instructions.

[0017] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0018] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, a dynamic heterogeneous topology graph containing data logic nodes and physical resource nodes is constructed in real time, and real-time directed dependency connections are established between the nodes. This unifies the operational relationships that were originally scattered across task orchestration metadata and infrastructure monitoring systems into topological constraints within the same graph structure. Based on these topological constraints, when an abnormal signal is detected, the data logic nodes related to the abnormal signal and their dependent physical resource nodes can be retrieved simultaneously within the same data structure, without repeatedly switching between multiple monitoring interfaces and log systems for comparison. This restricts the scope of investigation to the set of nodes that have an actual dependency relationship with the abnormal signal during the root cause analysis phase, reducing interference from irrelevant nodes and improving the focus and information utilization efficiency of the anomaly localization process. Furthermore, based on the dynamic heterogeneous topology graph, a counterfactual intervention set assuming upstream nodes are in a baseline state is generated using the Do operator. This counterfactual intervention set is then substituted into a causal inference model for simulation to obtain the anomaly maintenance probability of the abnormal signal under different counterfactual scenarios. This allows each upstream node to have a quantifiable impact on whether the current anomaly still occurs under the assumption that it remains normal. By combining the probability of maintaining the anomaly to calculate the causal contribution and thereby locking down the root cause node, it is possible to distinguish between downstream nodes that are passively affected by dependence and key nodes that have a higher causal contribution to the anomaly signal when there are multiple nodes fluctuating simultaneously. This reduces the probability of misjudgment caused by relying solely on experience-based judgments based on time correlation or curve shape similarity, thereby shortening the multi-round trial-and-error investigation process and improving the accuracy and stability of anomaly root cause localization.

[0019] Furthermore, by transmitting the root cause node's status data as structured data to the remediation module when the root cause node is identified to trigger replay remediation, key information such as configuration parameters, dependency library versions, and node runtime context relied upon in the root cause analysis phase can be completely and consistently integrated into the subsequent remediation process at the moment the anomaly occurs. Compared to relying on manual searching and recording of the aforementioned status information in different systems, this transmission mechanism reduces omissions and recording deviations during status acquisition. This facilitates the accurate reconstruction of a consistent on-site context with the production environment in an isolated environment during replay remediation, thereby reducing the risk of problems being unreproducible or remediation verification results not matching production reality due to inconsistent context. This improves the repeatability and reliability of the replay remediation process.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A system architecture diagram is shown that can be applied to the automated data anomaly root cause localization and replayable repair method according to embodiments of the present disclosure.

[0023] Figure 2 A flowchart illustrating an automated data anomaly root cause localization and replayable repair method according to an embodiment of this disclosure is shown.

[0024] Figure 3 A schematic diagram of a dynamic heterogeneous topology map is shown in an embodiment of this disclosure.

[0025] Figure 4 A schematic diagram of an automated data anomaly root cause localization and replayable repair system is shown in an embodiment of this disclosure.

[0026] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0027] Figure 6 A schematic diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown.

[0028] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0029] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] Figure 1 A schematic diagram of a system architecture for an automated data anomaly root cause localization and replayable repair method applicable to embodiments of this disclosure is shown.

[0032] like Figure 1 As shown, system architecture 100 may include one or more terminal devices such as desktop computer 101 and portable computer 102, network 104, and server 105. Network 104 is used as a medium to provide a communication link between the terminal devices and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0033] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0034] The automated data anomaly root cause localization and replayable repair method provided in this embodiment can be executed by a terminal device, and correspondingly, the automated data anomaly root cause localization and replayable repair system can be set in the terminal device. However, it is readily understood by those skilled in the art that the automated data anomaly root cause localization and replayable repair method provided in this embodiment can also be executed by a server 105, and correspondingly, the automated data anomaly root cause localization and replayable repair system can also be set in the server 105. This exemplary embodiment does not impose any special limitations on this.

[0035] This disclosure first provides an automated method for locating the root cause of data anomalies and for replayable repair. The method is then described below using server execution as an example.

[0036] Figure 2 The flowchart illustrating an automated data anomaly root cause localization and replayable repair method according to an embodiment of this disclosure is shown schematically. (Reference) Figure 2 As shown, the method may include steps S210 to S240: Step S210: Construct a dynamic heterogeneous topology graph containing data logic nodes and physical resource nodes in real time, and establish real-time directed dependency connections between the nodes; Step S220: In response to the detected abnormal signal, generate a set of counterfactual interventions assuming that the upstream node is in the baseline state based on the dynamic heterogeneous topology map using the Do operator; Step S230: Substitute the counterfactual intervention set into the causal reasoning model for simulation to obtain the probability of the abnormal signal still existing under the condition that the counterfactual intervention set is effective; Step S240: Calculate the causal contribution based on the anomaly maintenance probability to lock the root cause node, and transmit the status data of the root cause node to the repair module to trigger replay repair.

[0037] The following will provide a detailed description of the automated data anomaly root cause localization and replayable repair method in this example embodiment.

[0038] In step S210, a dynamic heterogeneous topology graph containing data logical nodes and physical resource nodes is constructed in real time, and real-time directed dependency connections are established between the nodes.

[0039] In this example implementation, the topological state at a certain moment can be represented as a directed graph. ,in This includes data logical nodes and physical resource nodes that are active or recently active at that moment, for example, reference Figure 3 As shown, the graph includes a data logic node layer 310 and a physical resource node layer 320; This is used to record the direction of data flow and resource call relationships. The aforementioned dynamic heterogeneous topology graph can be stored in one of three forms: adjacency list, adjacency matrix, or graph database. In the adjacency list implementation, each node maintains its outgoing and incoming edge lists, suitable for high-sparseness scenarios; in the adjacency matrix implementation, the connection relationships between nodes are encoded into a matrix. ,in This indicates the existence of a self-node. Pointing to node This method of directed edges is more advantageous when the node size is moderate and matrix operations are frequently required (e.g., path search, connectivity analysis).

[0040] More specifically, in this example embodiment, the real-time construction of a dynamic heterogeneous topology map including data logical nodes and physical resource nodes may include the following steps S211 to S214. Wherein: Step S211: Parse the task orchestration metadata of the data processing pipeline, extract the input / output patterns and code version information of each computing task, and generate a set of data logic nodes.

[0041] In this example implementation, task orchestration metadata can be stored by the scheduling system or workflow engine in the form of configuration files or metadata tables, and a unified task orchestration metadata abstraction model is predefined to abstract each computation task into a tuple: in, This represents a unique identifier for the task; and These represent the input and output patterns of the task, such as the identifiers of the datasets it depends on, the topic names, or the set of upstream task identifiers. This indicates the code version number or image tag corresponding to the task; This represents a hash digest of the configuration parameters associated with the task. During parsing, each task record in the task orchestration metadata can be traversed, and the aforementioned attributes can be extracted according to predefined field mapping rules.

[0042] After completing the metadata parsing, each task tuple is converted into a data logical node object, and a data logical node set is constructed. Each data logical node object can contain attribute fields such as task identifier, task type, input / output mode, code version, and configuration summary. To support subsequent binding with physical resource nodes, the data logical node object can also reserve fields for recording a list of runtime unique execution identifiers, task start timestamps, and task end timestamps. These fields are filled by runtime components when the task is actually scheduled for execution.

[0043] Step S212: Call the infrastructure monitoring interface to collect the CPU utilization, memory level and disk I / O indicators of the server container carrying the computing task, and generate a set of physical resource nodes.

[0044] In this example implementation, the infrastructure monitoring interface can be a monitoring API provided by a cloud vendor or an HTTP / GPRS interface exposed by a self-developed monitoring service. An infrastructure identifier can be assigned to each monitored server, virtual machine, or container. The system maintains a mapping between this identifier and the actual running entity (e.g., Kubernetes Pod name, container ID, hostname, etc.). During data collection, a fixed sampling period can be used. Pulling or receiving monitoring metrics, for each physical resource entity, at the sampling time Collect CPU utilization Memory usage Disk I / O read / write speed Equal values, and combine them into a time series. .

[0045] To facilitate association with data logic nodes, this example implementation defines physical resource node objects at the physical resource layer. Each physical resource node object corresponds to a physical resource entity or the aggregated state of a physical resource entity within a specific time interval. For example, a static physical resource node is created for each physical resource entity. The node attributes record static information such as resource type (e.g., container, physical machine, virtual machine), cluster affiliation, and node tag. Then, a time-series buffer for monitoring metrics is maintained within the node object, storing recent monitoring data through a circular buffer or time-indexed structure. (See reference...) Figure 3 As shown, node 321 can represent a container; node 322 can represent a storage volume, etc.

[0046] Step S213: Based on the unique execution identifier of the computing task during runtime, map and bind the nodes in the physical resource node set to the corresponding nodes in the data logical node set.

[0047] In this example implementation, the unique execution identifier is an identifier used to identify the execution of a specific task instance, such as a run ID assigned by the scheduling system for each task run, a workflow instance ID, or a TraceID in a cross-service call chain. For each data logical node... When its corresponding computing task is scheduled for execution, the runtime component can generate or obtain a unique execution identifier. And inject the identifier into the log context, monitoring tags, and call chain tracing information during the task's lifecycle.

[0048] More specifically, in an exemplary embodiment of this disclosure, the step of mapping and binding nodes in the physical resource node set to corresponding nodes in the data logical node set may include the following steps S2131 to S2133. Wherein: Step S2131: Extract the task start timestamp and task end timestamp of each node in the data logic node set.

[0049] In this example implementation, the start timestamp of the task instance can be recorded directly when the task is triggered. And record the end timestamp when the task instance ends or fails. Write these two timestamps along with the unique execution identifier into the run mapping table. Run the mapping table Each record in the middle is .

[0050] Step S2132: Based on the task start timestamp and task end timestamp, perform time window slicing on the time series monitoring data of the physical resource node set.

[0051] In this example implementation, a physical resource entity Monitoring time series For example, for a certain task instance, its execution time interval is... Then the resource time window corresponding to this task instance can be defined as... ,in and This is an optional buffer period used to cover the warm-up or cleanup phases before and after task startup. In practice, this can be achieved by performing an index range query on the monitored time series or by using a time index structure to quickly locate events falling within a given timeframe. All monitoring sampling points within .

[0052] Furthermore, in order to compress the resource status within this time window into attribute feature vectors that can be attached to logical node objects, statistical aggregation processing can be performed on the sample point set, such as calculating the average CPU utilization. Peak CPU utilization , No. quantile memory usage Average I / O throughput And so on, and thereby obtain the resource state vector. .

[0053] Step S2133: Write the physical resource state vector after time window slicing as an attribute feature into the corresponding logical node object to complete the binding of heterogeneous node attributes.

[0054] In this example implementation, for data logical nodes Upon identifying its relationship with a physical resource entity in a particular execution instance A binding relationship exists, and the resource state vector for this execution instance has been calculated. After that, you can Add or update a resource feature field to the attribute structure to record the observation results at the resource level for this execution. If the same logical node has multiple execution instances at different times, a set of resource features can be maintained for each execution instance, or the resource features of multiple executions can be further aggregated according to a set time window to form a comprehensive feature describing recent resource usage.

[0055] Step S214: The bound set of nodes is used as the vertices of the graph, and edge connections are established according to the data flow direction and resource call relationship to generate the dynamic heterogeneous topology graph.

[0056] In this example implementation, the direction of data flow can be determined based on the input / output pattern in the task orchestration metadata: if the task The output dataset is the task If declared as the input dataset, then from the graph... point to Establish a directed edge The edge attributes record information such as dataset identifier, data partitioning information, and data transmission protocol type. In some exemplary embodiments, to characterize the fine-grained nature of data dependencies, statistical information such as data magnitude and estimated latency can also be appended to the edge attributes, which also falls within the scope of this disclosure.

[0057] For resource call relationships, you can refer to the runtime mapping table. Extract the relationship between tasks and physical resource entities, if a certain data logical node Physical resource entities at a certain time slice The carrying capacity is found in the map from Points to the corresponding physical resource node Establish a directed edge This indicates that the data logical node invoked the physical resource during its operation. In some exemplary embodiments, a directed edge from the physical resource node to the data logical node may also be used, or a bidirectional edge may be established simultaneously. Furthermore, to ensure the real-time performance of the map, this example implementation triggers a local update each time a task is scheduled, completed, or a monitoring metric changes significantly. For example, when a new entry is added... When recording, it is possible to record only the corresponding and New binding edges are added, and the corresponding time window information is updated in the node attributes. When a task definition is modified or added, the corresponding data logic node and its dependency edges with upstream and downstream nodes are added incrementally. Thus, the above dynamic heterogeneous topology graph can maintain synchronization with the production environment's operating status with low overhead.

[0058] In step S220, in response to the detected abnormal signal, a set of counterfactual interventions assuming that the upstream node is in a baseline state is generated using the Do operator based on the dynamic heterogeneous topology map.

[0059] In this example implementation, the first step is to locate the target node corresponding to the abnormal signal in the map. The target node can be a data logical node or a physical resource node, and its identifier can be determined by a combination of the metric name that triggered the alarm, the task identifier, the resource identifier, and the timestamp. Upon receiving an abnormal alarm event, the relevant identifier is parsed from the abnormal event and added to the dynamic heterogeneous topology graph. A graph query is performed to find the target node object that corresponds one-to-one with the abnormal signal, and this target node is used as the structural anchor point for the outcome variable in subsequent causal inference. The target node and upstream node can be respectively as follows: Figure 3 Nodes 311 and 312 are shown in the diagram.

[0060] After locating the target node, a breadth-first or depth-first search approach is used to start from the target node. Starting from the beginning, traverse backwards along the incoming edge to collect all paths within the preset path length threshold. upstream node set within Furthermore, to avoid excessive computational overhead due to an overly large intervention set, this example implementation introduces two constraints during the traversal process: firstly, limiting the maximum path length. For example, only upstream nodes with a limit of a certain number of hops are considered; secondly, the types of upstream nodes are filtered, for example, only data logic nodes and key physical resource nodes are retained, while auxiliary nodes with low correlation to business anomalies are filtered out. Through the above constraints, this example implementation can control the size of the counterfactual intervention set while ensuring causal coverage.

[0061] After structurally determining the set of candidate upstream nodes, each upstream node can be included in the structured causal model. State variables are represented as random variables The result variable corresponding to the abnormal signal is represented as And represent the currently observed world state as a sample. In real-world observation scenarios, the state of the upstream node It has already been affected by various external factors and system noise. In this example implementation, the baseline state of the node before the anomaly occurred is used as the intervention target state. This baseline state can be obtained statistically within a reference time window prior to the time of the anomaly, such as by using the mean, median, or a certain quantile within the window. Thus, for each upstream node... An intervention of the form of a Do operator can be defined. In a causal model, this means that the state of the node is forced to be fixed as the baseline state, and is no longer determined by its parent node or external noise.

[0062] When constructing a counterfactual intervention set, it can be done for each upstream node. Construct a separate intervention scenario A set of single-node counterfactual interventions was obtained. Based on this, this example implementation can conveniently evaluate the impact of each upstream node individually recovering to its baseline state on the anomalous signal. The aforementioned counterfactual intervention set can be represented using both symbolic expressions and data structures. The symbolic expression form can be represented as follows: This is used for parsing in the causal reasoning engine; the data structure can be defined as a list of key-value pairs containing node identifiers and intervention values.

[0063] In step S230, the counterfactual intervention set is substituted into the causal reasoning model for simulation to obtain the probability of the abnormal signal still existing under the condition that the counterfactual intervention set is effective.

[0064] In this example implementation, given a structured causal model, each counterfactual intervention scenario constructed in step S220 is... The probability of the target anomalous signal occurring under this intervention condition is determined through model solving or simulation calculation. This probability is the probability of abnormal maintenance in the corresponding intervention scenario. This represents the result variable corresponding to the abnormal signal. This indicates the abnormal value corresponding to the abnormal signal, such as exceeding a threshold, triggering an error code, or having an abnormal status flag.

[0065] Specifically, this example implementation employs a structured causal model. The structured causal model consists of a directed acyclic graph. and a set of structural equations Composition, in which Represents a node In the figure The set of parent nodes in the middle, This represents independent noise variables. For the target variable... Its abnormal events can be represented as Without intervention, the probability of an anomaly is: in, This represents the union of all upstream variables.

[0066] Introducing intervention In this case, the corresponding equation is replaced with an externally given constant value to obtain a new causal model, and then the calculation is recalculated under the new causal model. .

[0067] Furthermore, to improve computational efficiency, this example implementation constructs a system that takes the upstream node state vector and topology code as input and the result variable as output. The probability distribution is the output depth model. During the training phase, historical data is used for fitting. The mapping relationship; in the intervention phase, for a given intervention scenario First, the state components of the specified upstream node are forcibly overwritten in the input vector by intervening semantics using the Do operator. Then, the modified input vector is input into the model. ,get Predicted distribution This will then identify the corresponding abnormal areas. The probability quality is used as the probability of maintaining the anomaly.

[0068] In step S240, the causal contribution is calculated based on the anomaly maintenance probability to lock the root cause node, and the status data of the root cause node is transmitted to the repair module to trigger replay repair.

[0069] In this example implementation, calculating the causal contribution based on the anomaly maintenance probability may include the following steps S2411 to S2414. Wherein: Step S2411: Obtain the baseline anomaly probability of the structured causal model under conditions without any counterfactual intervention.

[0070] In this example implementation, the situation without intervention is regarded as the standard world under the original structured causal model, and the corresponding anomaly probability is denoted as: in, For the result variables corresponding to the detected abnormal signals, This represents the set of outlier values ​​for the outcome variable, with a baseline outlier probability. It can be obtained by statistically analyzing historical observation samples or by performing a single, non-interventional inference in a structured causal model.

[0071] For each counterfactual intervention scenario constructed in the preceding steps The corresponding probability of maintaining the anomaly is calculated using a causal inference model: Step S2412: Calculate the probability difference between the anomaly maintenance probability and the baseline anomaly probability.

[0072] In this example implementation, the probability difference value of the above intervention scenario is defined as: in, Indicating the situation where intervention is applied Then, the reduction in the probability of anomalies compared to the uninterrupted state. If in a certain scenario... If the value is positive, the intervention can be considered to help alleviate the abnormality; if... If the value is close to zero, it indicates that the intervention has a limited impact on the anomaly; if... A negative value indicates that the intervention actually increases the probability of anomalies. Furthermore, to avoid incorrectly including scenarios that amplify anomalies in positive causal contributions, the probability difference can be truncated, resulting in: Furthermore, it is possible to retain only the portion that mitigates anomalies, providing non-negative input for subsequent node-level contribution aggregation.

[0073] In practical implementation, an upstream node typically corresponds to one or more counterfactual intervention scenarios. Therefore, for each upstream node... Maintain the related intervention scenario index set For example, both single-node intervention scenarios and multi-node joint intervention scenarios involving that node can be included in this set. To obtain the probability difference at the node level, the set can be... Aggregating the truncation probability differences in the data yields: Step S2413: Input the probability difference into a preset normalization function for weighted processing to obtain the causal contribution value of each upstream node to the current abnormal signal.

[0074] In this example implementation, linear normalization can be used, with the sum of the probability differences of all nodes as the normalization coefficient, to obtain: in, To prevent extremely small positive numbers with a denominator of zero; Indicates upstream node The causal contribution of a node can be numerically understood as the relative share of anomaly mitigation undertaken by that node among all upstream nodes.

[0075] Step S2414: Generate a root cause candidate list in descending order of the causal contribution values ​​for locking purposes.

[0076] In this example implementation, all nodes can be sorted in descending order of contribution to obtain an ordered sequence. Nodes with significantly higher contributions than other nodes are then prioritized. In one implementation, a contribution threshold can be set, retaining only nodes with contributions at or above this threshold in the candidate list to filter out nodes with weak anomaly interpretation capabilities. In another implementation, the top few nodes in the sorted list can be selected as a candidate node set, and their contribution values, corresponding key intervention scenarios, probability differences, and other information can be recorded for subsequent automatic or manual decision-making.

[0077] In this example embodiment, locking the root cause node may include the following steps S2421 to S2424. Wherein: Step S2421: Assign a unique dynamic taint label to the node with the highest causal contribution.

[0078] In this example implementation, after obtaining the root cause candidate list, the node with the highest causal contribution is prioritized as the primary candidate for root cause node. For the node with the highest contribution... A unique dynamic taint label is assigned to each node, which can embed anomaly event identifiers, timestamps, and node identifiers to ensure global uniqueness throughout the analysis process; in other words, a dynamic taint label is essentially a marker used to track the propagation path of influence in a graph structure.

[0079] Step S2422: In the dynamic heterogeneous topology graph, traverse downstream along the directed dependency connections and pass the dynamic taint label to all downstream nodes that directly or indirectly depend on the node.

[0080] In this example implementation, during the propagation process, starting from the root cause node, all downstream nodes reachable along directed edges from that node can be scanned using breadth-first or depth-first traversal. Whenever a downstream node is visited, if it is not yet marked as a visited node in this analysis task, it is given the same dynamic taint label as the root cause node and added to the traversal queue. By continuously repeating this process until the queue is empty, all sets of nodes directly or indirectly dependent on the root cause node can be found in the graph, and these nodes are uniformly marked as nodes affected by the root cause. In some exemplary embodiments, to avoid excessive propagation, constraints can be added during the traversal process, such as propagating only along edges representing data flow and not along edges representing management relationships, or limiting the maximum propagation layer to control the propagation range within a certain topological depth.

[0081] Step S2423: Identify all sets of nodes marked with the dynamic taint label, and calculate the maximum propagation boundary of the set of nodes in the graph.

[0082] In this example implementation, the maximum propagation boundary is the set of boundary nodes among all marked nodes that have direct directed edges connected to unmarked nodes, or a subset of other forms used in business to represent the edge of the influence range. In one implementation, all outgoing edges of each marked node can be checked; if an outgoing edge points to an unmarked node, the marked node is included in the boundary node set. In another implementation, boundary nodes can be further filtered by combining information such as the node's hierarchy and service domain.

[0083] Step S2424: Mark all nodes within the maximum propagation boundary as the affected range and output them as part of the locking result.

[0084] In this example implementation, the triggered playback repair may include the following steps S2431 to S2434. Wherein: Step S2431: Read the configuration parameters, dependency library versions, and input data pointers of the locked root cause node at the time of the anomaly, and generate on-site environment metadata.

[0085] In this example implementation, the on-site environment metadata includes at least three aspects of information: configuration parameter snapshots, dependency library version information, and input data pointers. For configuration parameter snapshots, the configuration version effective during the instance's runtime can be queried from the configuration center, task orchestration metadata, or version control repository based on the data logical node identifier and task execution instance identifier corresponding to the root cause node. For dependency library version information, the list of dependency libraries included in the build artifact can be queried from the build system or artifact repository based on the build record of the service or task corresponding to the root cause node, such as third-party open-source libraries, internal common components, and runtime versions. In containerized scenarios, the metadata file or software bill of materials of the container image can also be directly parsed to obtain the name and precise version number of each dependency component. For input data pointers, the range of input data processed by the root cause node at the time of the anomaly can be determined by relying on logs, metadata management services, or data catalog systems. For example, in batch processing jobs, the source table names, partition keys, and partition values, or the set of file paths involved in the execution can be identified through job parameters, task logs, or metadata tables. In streaming processing tasks, input data pointers can be represented as combinations of several intervals based on consumer group offsets, topic names, and partition numbers. The above configuration information, dependency library version information, and input data pointers are integrated into a structured object, constituting the site environment metadata.

[0086] Step S2432: Based on the site environment metadata, launch a twin sandbox environment consistent with the production environment in an isolated container cluster.

[0087] In this example implementation, the required container image version is first determined based on the dependency library version information in the site environment metadata. If the image used in the production environment is still available, the same image tag can be directly referenced in the sandbox environment; if the production image has been updated or deleted, it can be rebuilt or restored by the build system according to the recorded version information. Subsequently, the system generates a deployment description on the sandbox side based on the configuration parameter snapshot, and sets environment variables, mount configuration files, or command-line parameters in the container orchestration configuration so that the services or tasks in the sandbox load the same configuration set as at the time of the exception upon startup.

[0088] To maintain consistency in the data view at the time of the anomaly, the persistent storage volumes mounted on the root cause node also need to be addressed. One implementation involves using the storage volume identifier recorded in the environment metadata to call the underlying storage system's snapshot interface and create a logical snapshot of the storage volume at the time of the anomaly using a copy-on-write mechanism. After the snapshot is created, a new container instance is created in the isolated cluster, and the snapshot is mounted in read-only mode to a predetermined directory. This ensures that the twin sandbox sees completely consistent content when reading relevant data compared to the time of the anomaly. For services involving multiple persistent volumes, the above steps can be repeated, creating a separate snapshot for each volume and mounting it to the corresponding directory.

[0089] After completing image selection, configuration injection, and snapshot mounting, the system starts container instances, forming the basic operating unit of the twin sandbox environment. For scenarios involving multiple collaborative services or multi-instance tasks, corresponding instances can be created in the sandbox for each relevant service based on the service topology relationships recorded in the on-site environment metadata, and connected through service discovery or local configuration, thereby reproducing the service call structure when an anomaly occurs in the isolated environment.

[0090] Step S2433: Inject the repair patch to be verified into the twin sandbox environment, and load the historical data stream according to the input data pointer.

[0091] In this exemplary implementation, the patch can take the form of modifying the code corresponding to the root cause node, adjusting configuration parameters, or reverting or upgrading the version of dependent components. There are several ways to inject patches: for example, rebuilding a new container image based on the modified source code and replacing the original image with this image in the twin sandbox; or, by mounting an overwrite directory, overwriting the corresponding module in the original image with the patch code as a file; in interpreted language scenarios, a hot-loading mechanism can also be used to dynamically replace modules during sandbox operation. This exemplary embodiment does not impose any special limitations on this.

[0092] When loading historical data streams, this example implementation retrieves data shards from the data warehouse, message queue, or object storage at the time of the anomaly, based on the input data pointer. In batch processing scenarios, a simulated task execution command can be constructed, specifying the partition or file path where the historical data is located as the input parameter, causing the sandbox task to reconsume this data. In streaming processing scenarios, historical messages can be replayed in their original order using a replay engine and sent to the consumer in the sandbox, with the replay rate adjusted as needed to balance verification time and resource consumption.

[0093] Step S2434: Perform deterministic replay in the twin sandbox environment and output the repair verification results after replay.

[0094] To ensure good repeatability of the replay process, this example implementation aims to minimize non-deterministic factors within the twin sandbox. For example, a fixed random number seed is used at task startup, calls to uncontrollable external services are replaced with stubs or mock services within the sandbox, and the concurrency and scheduling strategy of task execution are constrained to avoid unstable output results due to random factors such as thread preemption or network jitter. In some exemplary embodiments, the system time of task execution can also be fixed at a specific simulated time point to avoid time-related branching logic.

[0095] In this example embodiment, the triggered playback repair may further include the following steps S2441 to S2444. Wherein: Step S2441: Extract the source code script corresponding to the locked root cause node and parse it into an abstract syntax tree.

[0096] For example, this example implementation can extract the corresponding source code script from the code repository based on the root cause node's identifier, and then hand the script over to the language parser to construct an abstract syntax tree. In the abstract syntax tree, condition judgment nodes and data transformation nodes are identified, such as if branches, conditional expressions, filtering predicates, aggregate function calls, etc.

[0097] Step S2442: Apply preset mutation operators to the condition judgment nodes and data transformation nodes in the abstract syntax tree to generate multiple candidate repair mutation trees.

[0098] After identifying the aforementioned key nodes, multiple candidate repair variants can be generated by applying preset mutation operators to these nodes. In this example implementation, mutation operators may include condition inversion, boundary value offsetting, adding null value checks, adjusting data type conversion logic, and changing default branch behavior. For example, when processing thresholds, constants in the original comparison expression can be adjusted to slightly larger or smaller values; when processing filtering conditions, defensive checks against abnormal inputs can be added. Furthermore, when applying mutation operators, each mutation can be limited to modifying only one or a limited number of syntax nodes to ensure that the difference between a single mutated version and the original version is controllable, facilitating the identification of the source of behavior changes.

[0099] Step S2443: Compile the multiple candidate repair mutation trees into executable code and deploy them in parallel to multiple shadow processes.

[0100] In this example implementation, each mutated abstract syntax tree generated can be reverse-engineered into source code or an intermediate representation, which is then used by a compiler or interpreter to generate the corresponding executable artifact. The executable artifact does not directly replace the code in the production environment; instead, it is deployed as a shadow version in a shadow process. This shadow process runs in an isolated environment, sharing the same dependency versions and runtime as the production service, but all write operations to external systems are redirected or blocked, ensuring no impact on real data and business operations.

[0101] Step S2444: Simultaneously inject abnormal sample data into the multiple shadow processes, and select the shadow processes whose output results meet expectations as the target repair solution.

[0102] During the verification phase, this example implementation extracts representative anomaly sample data from the on-site environment metadata. These samples are then injected into each shadow process using the same input interface, and the output results of each candidate repair version are recorded. Specifically, the output of each version can be automatically compared based on predefined correctness rules, data quality rules, or business invariants. For example, it checks whether error codes have disappeared, whether the output data meets constraints, and whether key indicators have returned to normal ranges. If a candidate repair version performs normally under all verification rules, it can be marked as a valid candidate repair solution. If multiple valid candidates exist simultaneously, the version with the best overall performance can be selected as the target repair solution based on performance overhead, code change magnitude, or compatibility evaluation results.

[0103] In this example implementation, this mechanism based on abstract syntax tree mutation and shadow execution can automatically explore multiple remediation paths without interfering with the production environment, thereby providing more targeted remediation patches in deterministic replays in the twin sandbox or formal deployments.

[0104] This example implementation also provides an automated data anomaly root cause localization and replayable repair system. (See reference) Figure 4 As shown, the automated data anomaly root cause localization and replayable repair system 400 may include a topology map construction module 410, a counterfactual intervention module 420, an anomaly simulation module 430, and a replay repair module 440. Wherein: The topology graph construction module 410 can be used to construct a dynamic heterogeneous topology graph containing data logical nodes and physical resource nodes in real time, and establish real-time directed dependency connections between the nodes. The counterfactual intervention module 420 can be used to generate a counterfactual intervention set assuming upstream nodes are in a baseline state based on the dynamic heterogeneous topology graph and using the Do operator in response to detected abnormal signals. The anomaly simulation module 430 can be used to substitute the counterfactual intervention set into a causal inference model for simulation, and obtain the anomaly maintenance probability that the abnormal signal still exists under the condition that the counterfactual intervention set is effective. The replay repair module 440 can be used to calculate the causal contribution based on the anomaly maintenance probability to lock the root cause node, and transmit the state data of the root cause node to the repair module to trigger replay repair.

[0105] The specific details of each module of the aforementioned automated data anomaly root cause localization and replayable repair system have been described in detail in the corresponding automated data anomaly root cause localization and replayable repair methods, so they will not be repeated here.

[0106] refer to Figure 5 As shown, an electronic device capable of implementing the above-described method is also provided. The electronic device 500 includes a processor 501 and a memory 502. The memory 502 stores computer-readable instructions, which, when executed by the processor 501, implement the method of this disclosure.

[0107] In an exemplary embodiment of this disclosure, a computer-readable storage medium is also provided, having stored thereon computer program code instructions that, when invoked by a processor, execute the method described in the embodiments.

[0108] refer to Figure 6 As shown, a program product 600 for implementing the above-described method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0109] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0110] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this disclosure and are not restrictive. Although this disclosure has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this disclosure. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. An automated data anomaly root cause localization and replayable remediation method, characterized in that, The method comprises the following steps: Real-time construction of a dynamic heterogeneous topology graph containing data logic nodes and physical resource nodes, and establishment of real-time directed dependency connections between the nodes; In response to a monitored abnormal signal, generation of a counterfactual intervention set using a Do operator based on the dynamic heterogeneous topology graph, with the assumption that an upstream node is in a baseline state; Simulation of the counterfactual intervention set in a causal inference model to obtain an abnormal maintenance probability of the abnormal signal under the condition that the counterfactual intervention set takes effect; Calculation of a causal contribution degree according to the abnormal maintenance probability to lock a root cause node, and transmission of state data of the root cause node to a repair module to trigger a playback repair.

2. The method of claim 1, wherein, The real-time construction of a dynamic heterogeneous topology graph containing data logic nodes and physical resource nodes comprises the following steps: Parsing of task scheduling metadata of a data processing pipeline, extraction of input / output modes and code version information of each computing task, and generation of a data logic node set; Calling of an infrastructure monitoring interface, collection of CPU utilization, memory level and disk I / O indicators of a server container carrying the computing task, and generation of a physical resource node set; Mapping and binding of nodes in the physical resource node set to corresponding nodes in the data logic node set based on a unique execution identifier of the computing task at runtime; Taking the bound node set as graph vertices, and establishing edge connections according to data flow direction and resource calling relationship to generate the dynamic heterogeneous topology graph.

3. The method of claim 2, wherein, The mapping and binding of nodes in the physical resource node set to corresponding nodes in the data logic node set comprises the following steps: Extraction of task start time stamp and task end time stamp of each node in the data logic node set; Time window slicing of time series monitoring data of the physical resource node set according to the task start time stamp and the task end time stamp; Writing of the physical resource state vector after time window slicing processing into the corresponding logic node object as attribute characteristics to complete the binding of node heterogeneous attributes.

4. The method of claim 1, wherein, The calculation of the causal contribution degree according to the abnormal maintenance probability comprises the following steps: Obtaining of a baseline abnormal probability of a structured causal model under the condition that it is not subjected to any counterfactual intervention; Calculation of a probability difference value between the abnormal maintenance probability and the baseline abnormal probability; Inputting of the probability difference value into a preset normalization function for weighted processing to obtain a causal contribution degree value of each upstream node for the current abnormal signal; Generation of a root cause candidate list in order from high to low according to the causal contribution degree value for locking use.

5. The method of claim 1, wherein, The triggering of the playback repair comprises the following steps: Reading of configuration parameters, dependent library versions and input data pointers of the locked root cause node at the time of abnormal occurrence to generate on-site environment metadata; Based on the on-site environment metadata, a twin sandbox environment consistent with the production environment is pulled up in an isolated container cluster; Injection of a repair patch to be verified into the twin sandbox environment, and loading of historical data streams according to the input data pointers; Execution of a deterministic playback in the twin sandbox environment, and output of a repair verification result after the playback.

6. The method of claim 5, wherein, The pulling up of a twin sandbox environment consistent with the production environment in the isolated container cluster based on the live environment metadata comprises: identifying the identifier of the persistent storage volume mounted by the root cause node at the abnormal time; creating a storage snapshot of the persistent storage volume at the abnormal time using copy-on-write; creating a new container instance and mounting the storage snapshot to the corresponding directory of the new container instance in read-only mode; starting the new container instance as a basic running unit of the twin sandbox environment.

7. The method of claim 1, wherein, The triggering of the playback repair comprises: extracting the source code script corresponding to the locked root cause node and parsing it into an abstract syntax tree; applying a preset mutation operator to the conditional judgment nodes and data conversion nodes in the abstract syntax tree to generate multiple candidate repair mutation trees; compiling the multiple candidate repair mutation trees into executable code respectively and deploying them into multiple shadow processes in parallel; injecting abnormal sample data into the multiple shadow processes simultaneously and screening out the shadow process whose output result meets the expectation as the target repair scheme.

8. The method of claim 1, wherein, The locking of the root cause node comprises: allocating a unique dynamic taint label to the node with the highest causal contribution degree; traversing downstream along the directed dependency connection in the dynamic heterogeneous topology graph and passing the dynamic taint label to all downstream nodes that directly or indirectly depend on the node; identifying the node set marked with the dynamic taint label, calculating the maximum propagation boundary of the node set in the graph, and marking all nodes within the maximum propagation boundary as the affected range and outputting them as part of the locking result. Comprise:

9. An automated data anomaly root cause localization and replayable remediation system, comprising: a topology graph construction module for constructing a dynamic heterogeneous topology graph containing data logic nodes and physical resource nodes in real time and establishing real-time directed dependency connections between the nodes; a counterfactual intervention module for generating a counterfactual intervention set assuming that the upstream nodes are in a baseline state based on the dynamic heterogeneous topology graph using Do operators in response to the monitored abnormal signal; an abnormal simulation module for simulating the counterfactual intervention set in the causal inference model to obtain the abnormal maintenance probability of the abnormal signal under the effectiveness condition of the counterfactual intervention set; a playback repair module for locking the root cause node according to the causal contribution degree calculated based on the abnormal maintenance probability and transmitting the state data of the root cause node to the repair module to trigger the playback repair. The computer program is executed by a processor to implement the method of any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 1-8.

Images