Fault root cause localization and isolation system and method

Abstract

The application discloses a fault root cause positioning and isolation system and method, the system comprises: a monitoring index collector, used for collecting performance indexes, business indexes and resource indexes of a production environment; a shadow pipeline, used for receiving a real-time duplicate copy of the production environment flow as experimental flow, and dynamically configuring an experimental component version based on experimental instructions issued by the causal reasoning engine to process the experimental flow; a causal reasoning engine, in response to detecting a fault, constructs and maintains a dynamic causal graph based on historical fault data; candidate hypotheses are generated based on the dynamic causal graph, each candidate hypothesis corresponding to a suspected fault root cause component; finding the root cause component causing the fault is modeled as a black box function optimization problem, and a Bayesian optimization method is adopted to obtain the next best candidate hypothesis to be verified according to historical experimental results through expected improvement. The application improves the fault processing efficiency and is more suitable for operation and maintenance scenarios of large-scale complex reasoning service systems.

Description

Technical Field

[0001] This application relates to artificial intelligence operations and maintenance (AIOps) technology, and in particular to an automated fault root cause localization and isolation system and method based on causal reasoning and shadow patterns. Background Technology

[0002] With the widespread application of artificial intelligence (AI) technology, inference services for deep learning models have become a core technology for many critical businesses. Ensuring high availability, high reliability, and low latency of model inference services is crucial. During testing or production runs before model deployment, failures are inevitable and varied in type, including hardware resource failures, software service failures, model performance degradation, and abnormal input data.

[0003] Currently, the industry generally adopts a resource threshold-based monitoring solution. This solution involves deploying a monitoring agent on the server to periodically collect relevant metrics of basic resources such as CPU, memory, and GPU, while also monitoring performance metrics of the application layer such as throughput and latency. By setting fixed thresholds for each metric, an alarm is triggered when the metric exceeds the threshold, allowing operations and maintenance personnel to intervene and investigate.

[0004] However, existing technologies have the following technical shortcomings: First, the false alarm and false negative rates are high because static thresholds cannot adapt to the dynamic changes in model inference load; second, the fault location capability is weak, and alarm information can only indicate abnormal phenomena such as high CPU utilization, but cannot automatically reveal the root cause such as model defects, data anomalies, or framework vulnerabilities; third, there is a lack of predictive and self-healing capabilities, as current technologies are passive response mechanisms and rely on manual operation, resulting in a long mean time to recovery (MTTR); finally, they cannot effectively isolate faults and prevent recurrence, and the impact of faults continues during manual troubleshooting, and due to unclear root cause location, similar faults are prone to recurrence.

[0005] Therefore, how to automate and accurately locate the root causes of complex system failures, and on this basis, achieve rapid and safe fault isolation and self-healing, has become a key challenge in improving the reliability of AI operation and maintenance and software systems. Summary of the Invention

[0006] This application provides a fault root cause localization and isolation system and method, electronic device and storage medium to at least solve the above-mentioned technical problems existing in the prior art.

[0007] According to a first aspect of this application, a fault root cause localization and isolation system is provided, comprising: a monitoring indicator collector, a shadow pipeline, a causal inference engine, and a policy executor, wherein:

[0008] A monitoring metrics collector is used to collect performance metrics, business metrics, and resource metrics of the production environment; the production environment includes the production environment of the model inference service.

[0009] The shadow pipeline, deployed in isolation from the production environment, is used to receive real-time copies of the production environment traffic as experimental traffic, and dynamically configures the experimental component version based on the experimental instructions issued by the causal inference engine to process the experimental traffic; the experimental component version includes at least one of the following: different versions of the model, different versions of the inference service code, and different versions of the dependency library.

[0010] The causal inference engine responds to the monitoring indicator collector detecting a fault by constructing and maintaining a dynamic causal graph based on historical fault data. Nodes in the dynamic causal graph represent system states, including component versions and performance indicators, while edges represent causal relationships and weights between nodes. Candidate hypotheses are generated based on the dynamic causal graph, with each candidate hypothesis corresponding to a suspected root cause component. The search for the root cause component is modeled as a black-box function optimization problem, employing a Bayesian optimization method. Based on historical experimental results, a Gaussian process is used as a surrogate model to predict the validation benefit and uncertainty of each candidate hypothesis, and the next best candidate hypothesis to be validated is selected through an expected improvement function. Experimental instructions are sent to the shadow pipeline, containing the component version configuration corresponding to the best candidate hypothesis. The experimental results returned by the shadow pipeline are received and analyzed: the output results of the shadow pipeline under a specific component version configuration are statistically compared with the output results of the production pipeline under a fault state. Hypothesis testing is performed based on a preset significance level. If the statistical difference narrows, the component replaced in the specific component version configuration is determined to be the root cause component of the fault.

[0011] The strategy executor is communicatively connected to the causal inference engine and is used to receive the root cause of the fault determined by the causal inference engine, and to call the application programming interface of the external system to perform fault isolation and recovery according to the predefined automatic repair strategies corresponding to different root cause types.

[0012] In some alternative implementations, the shadow pipe includes:

[0013] The traffic replication module is used to perform lossless replication of requests flowing into the production environment through the traffic mirroring function of the service mesh, and send the replicated requests as the experimental traffic to the shadow pipeline.

[0014] The environment isolation module is used to deploy the shadow pipeline in an independent namespace and ensure that the computing, storage and network resources used by the shadow pipeline are isolated from the production environment through resource quota restrictions and network policies.

[0015] The component management module is used to encapsulate each component into a container image with version tags. In response to the experimental instructions of the causal inference engine, it dynamically replaces the version tags of the container images in the shadow pipeline by calling the application programming interface of the container orchestration platform, so as to achieve the second-level switching of component versions.

[0016] The data collection module is used to generate a unique experiment identifier for each batch of experimental traffic, record the output results and experimental metadata during the shadow pipeline processing, and send the output results and experimental metadata to the message queue; the experimental metadata includes the experiment identifier, the component image version tag used, the processing timestamp, and resource usage.

[0017] In some alternative implementations, the causal reasoning engine is also used for:

[0018] Based on historical fault data, Granger causality analysis is used to establish causal edges and initial weights between nodes. Granger causality analysis uses a vector autoregression model to test whether past values ​​of the time series to be processed can improve the predictive ability of the current value of another time series.

[0019] After the experiment, the topology and edge weights of the dynamic causal graph were iteratively optimized using the Bayesian network structure learning algorithm and the new experimental results.

[0020] In some alternative implementations, the causal reasoning engine is further used for:

[0021] Define the benefit or cost of validating candidate hypotheses as the objective function;

[0022] Using a Gaussian process as a surrogate model for the objective function, the posterior probability distribution of the objective function is obtained by fitting the completed experimental data; the posterior probability distribution is determined by the mean function and the covariance function, and is used to predict the function value and uncertainty of the new candidate hypothesis;

[0023] Expected improvement is used as the acquisition function to balance the utilization of regions with high predictive returns with the exploration of regions with high uncertainty;

[0024] The candidate hypothesis that yields the largest function value is selected as the next best candidate hypothesis to be tested.

[0025] In some alternative implementations, the causal reasoning engine is further used for:

[0026] Numerical indices are tested using a two-sample t-test or the Mann-Whitney U-test. For probability distribution outputs, the Kohlbek-Leibler divergence is calculated to measure the differences in distribution.

[0027] The calculated statistic is compared with a preset significance threshold. If the statistic indicates a difference greater than the threshold, the null hypothesis is rejected and the verified component is taken as the root cause of the failure.

[0028] In some alternative implementations, the policy executor maintains a versioned policy library that stores remediation policies defined in declarative files, which are mapped to corresponding automated operations based on different root cause types of failures.

[0029] When the root cause is the model version, the application interface of the model management service is called to perform a model version rollback.

[0030] When the root cause is the service code version, the rollback pipeline of the continuous integration / continuous deployment system is triggered;

[0031] When the root dependency library version is incorrect, the package management tool is invoked to perform a dependency library downgrade installation.

[0032] The policy executor atomically executes the automated operation and records an audit log.

[0033] According to a second aspect of this application, a method for fault root cause localization and isolation applied to the aforementioned fault root cause localization and isolation system is provided, the method comprising:

[0034] Collect various indicators of the production environment of the running model inference service. If a fault is detected based on the indicators, construct and maintain a dynamic causal graph representing the causal relationship between system states based on historical data.

[0035] Candidate hypotheses are generated based on dynamic cause-effect graphs, and each candidate hypothesis corresponds to a suspected root cause component of the failure.

[0036] The Bayesian optimization method is used to select the next best candidate hypothesis to be tested.

[0037] In response to receiving experimental instructions from the shadow pipeline, the shadow pipeline configuration is indicated to match the component version combination corresponding to the best candidate hypothesis;

[0038] The real-time copy of the production environment traffic is used as the experimental traffic, and the experimental traffic is processed after the component version is dynamically configured according to the experimental instructions.

[0039] Return the processing results of the shadow pipeline to the causal reasoning engine;

[0040] The causal reasoning engine is used to statistically compare the processing results of the shadow pipeline with the output of the production pipeline and perform hypothesis testing to determine the root cause component of the failure.

[0041] Based on the identified root cause of the fault, perform predefined automated repair operations.

[0042] In some alternative implementations, the shadow pipeline replicates production traffic losslessly via the service mesh's traffic mirroring functionality;

[0043] According to the experimental instructions, the container image versions of one or more components are dynamically switched by calling the application programming interface of the container orchestration platform;

[0044] Generate a unique identifier for the experimental traffic and record the processing results and experimental metadata to the message queue.

[0045] In some alternative implementations, the construction and maintenance of a dynamic causal graph representing the causal relationships between system states based on historical data includes:

[0046] Based on historical fault data, causal edges are established using Granger causality analysis.

[0047] After each experiment, the topology and edge weights of the causal graph are iteratively updated using a Bayesian network structure learning algorithm.

[0048] In some alternative implementations, the selection of the next best candidate hypothesis to be verified includes:

[0049] A Gaussian process was used as a surrogate model to fit the posterior distribution of historical experimental results;

[0050] By employing the expected improved acquisition function, a trade-off is made between regions with high predicted returns and high uncertainty, and the next experimental hypothesis is selected.

[0051] Correspondingly, the step of statistically comparing and hypothesis testing the processing results of the shadow pipeline with the output of the production pipeline includes:

[0052] Select the appropriate statistical test method or difference measurement algorithm based on the type of output data;

[0053] Calculate the test statistic or difference value and compare it with the preset significance threshold;

[0054] If the value exceeds the significance threshold, the component verified in the current experiment will be taken as the root cause of the failure.

[0055] According to a third aspect of this application, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing the computer to perform the steps of the fault root cause localization and isolation method.

[0056] According to a fourth aspect of this application, an electronic device is provided, comprising:

[0057] At least one processor;

[0058] and a memory communicatively connected to the at least one processor; wherein,

[0059] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the fault root cause localization and isolation method.

[0060] The technical solution of this application replicates production traffic to an independent, isolated environment for experimentation via shadow pipes, avoiding impact on production operations and resolving the business risks inherent in existing invasive testing methods. Simultaneously, shadow pipes support dynamic component plugging / unplugging and traffic ratio configuration, adapting to fault location needs of varying scales and types. This application integrates causal inference and Bayesian optimization techniques, establishing causal relationships between components and metrics through dynamic causal graphs, avoiding the susceptibility to noise interference inherent in traditional correlation analysis. Furthermore, by combining a Gaussian process surrogate model and an expected improvement acquisition function, it balances experimental efficiency and location accuracy, significantly improving the accuracy and efficiency of root cause localization. Therefore, this application achieves end-to-end closed-loop processing from metric collection, anomaly detection, root cause localization, experimental verification to automated repair, requiring no manual intervention, significantly reducing operation and maintenance costs, improving fault handling efficiency, and making it more suitable for the operation and maintenance scenarios of large-scale, complex inference service systems.

[0061] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0062] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0063] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0064] Figure 1 A schematic diagram of the composition structure of the fault root cause localization and isolation system according to an embodiment of this application is shown;

[0065] Figure 2 This illustration shows an application diagram of the fault root cause localization and isolation system according to an embodiment of this application;

[0066] Figure 3 A schematic diagram illustrating the implementation flow of the fault root cause localization and isolation method according to an embodiment of this application is shown;

[0067] Figure 4 This application provides a schematic diagram of the composition structure of an electronic device. Detailed Implementation

[0068] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] The technical problems to be solved by the embodiments of this application include: how to achieve automated root cause localization of model inference test failures, quickly and accurately determine whether the problem is due to code, model, data, or environment dependency, reduce false positives and false negatives, and replace inefficient manual troubleshooting; and how to automatically perform isolation and recovery operations after locating the root cause to achieve fault self-healing and significantly shorten the mean time to recovery (MTTR). The following specific examples illustrate in detail the essence of the technical solutions of the embodiments of this application.

[0070] Figure 1 A schematic diagram of the composition structure of the fault root cause localization and isolation system according to an embodiment of this application is shown, such as... Figure 1 As shown, the innovation of the fault root cause localization and isolation system in this application embodiment lies in the introduction of a causal inference engine 12 and a shadow pipeline 11. The system not only passively monitors but also actively performs experimental inference, automatically deriving the root cause of the fault by comparing and analyzing results from different versions of components. Specifically, the fault root cause localization and isolation system in this application embodiment includes: a monitoring indicator collector 10, a shadow pipeline 11, a causal inference engine 12, and a strategy executor 13, wherein:

[0071] The monitoring metrics collector 10 is used to collect performance metrics, business metrics, and resource metrics of the production environment; the production environment includes the production environment of the model inference service.

[0072] Shadow Pipeline 11 is deployed in isolation from the production environment to receive real-time replicated copies of production environment traffic as experimental traffic. Based on experimental instructions issued by the causal inference engine 12, it dynamically configures the experimental component version to handle experimental traffic. The experimental component version includes at least one of the following: different versions of the model, different versions of the inference service code, and different versions of the dependency library.

[0073] The causal inference engine 12 responds to the monitoring index collector 10 detecting a fault by constructing and maintaining a dynamic causal graph based on historical fault data. Nodes in the dynamic causal graph represent system states, including component versions and performance indicators, while edges represent causal relationships and weights between nodes. Candidate hypotheses are generated based on the dynamic causal graph, with each candidate hypothesis corresponding to a suspected root cause component of the fault. The search for the root cause component is modeled as a black-box function optimization problem, employing a Bayesian optimization method. Based on historical experimental results, a Gaussian process is used as a surrogate model to predict the validation benefit and uncertainty of each candidate hypothesis. The next optimal candidate hypothesis to be validated is selected through an expectation improvement acquisition function. In the Gaussian process surrogate model, a zero-mean function is used for the mean function, and a squared exponential covariance function is used for the covariance function. The model hyperparameters are optimized using a maximum likelihood estimation method. In the expectation improvement acquisition function, a dynamic balance between experimental efficiency and positioning accuracy is achieved by adjusting the exploration-utilization tradeoff parameter (range 0.1-2.0). In emergency fault scenarios, increasing the parameter prioritizes exploring high-probability regions. Send an experimental instruction to the shadow pipeline 11; the experimental instruction contains the component version configuration corresponding to the best candidate hypothesis; receive the experimental results returned by the shadow pipeline 11 and analyze them: statistically compare the output results of the shadow pipeline 11 under the specific component version configuration with the output results of the production pipeline under the fault state, perform hypothesis testing according to the preset significance level, and if the statistical difference narrows, determine that the component replaced in the specific component version configuration is the root cause component of the fault.

[0074] The strategy executor 13 is connected to the causal inference engine 12 and is used to receive the root cause of the fault determined by the causal inference engine 12. Based on the predefined automatic repair strategies corresponding to different root cause types, it calls the application programming interface of the external system to perform fault isolation and recovery.

[0075] As one implementation, the shadow pipe 11 implemented in this application may include:

[0076] The traffic replication module is used to perform lossless replication of requests flowing into the production environment through the traffic mirroring function of the service mesh, and send the replicated requests as experimental traffic to the shadow pipe 11.

[0077] The environment isolation module is used to deploy the shadow pipe 11 in an independent namespace and ensure that the computing, storage and network resources used by the shadow pipe 11 are isolated from the production environment through resource quota restrictions and network policies.

[0078] The component management module is used to encapsulate each component into a container image with version tags. In response to the experimental instructions of the causal inference engine 12, it dynamically replaces the version tags of the container images in the shadow pipeline 11 by calling the application programming interface of the container orchestration platform, so as to achieve the second-level switching of component versions.

[0079] The data collection module is used to generate a unique experimental identifier for each batch of experimental traffic, record the output results and experimental metadata during the processing of Shadow Pipeline 11, and send the output results and experimental metadata to the message queue. The experimental metadata includes the experimental identifier, the version tag of the component image used, the processing timestamp, and the resource usage.

[0080] The causal inference engine 12 implemented in this application is also used for: establishing causal edges and initial weights between nodes based on historical fault data using Granger causality analysis, wherein Granger causality analysis uses a vector autoregression model to test whether past values ​​of the time series to be processed can improve the predictive ability of the current value of another time series; after the experiment, using a Bayesian network structure learning algorithm, combined with the new experimental results, iteratively optimizing the topology and edge weights of the dynamic causal graph. Furthermore, the benefit or cost of verifying candidate hypotheses is defined as the objective function; a Gaussian process is used as a surrogate model for the objective function, and the posterior probability distribution of the objective function is obtained by fitting it to the completed experimental data; the posterior probability distribution is determined by the mean function and the covariance function, and is used to predict the function value and uncertainty of new candidate hypotheses; expected improvement is used as the acquisition function to balance the utilization of regions with high predictive benefit and the exploration of regions with high uncertainty; the candidate hypothesis with the largest acquisition function value is taken as the next best candidate hypothesis to be verified. In addition, the causal inference engine 12 of this application embodiment can use a two-sample T test or a Mann-Whitney U test to test numerical indicators. For probability distribution outputs, the Kohlbek-Leibler divergence is calculated to measure the difference in distribution. The calculated statistic is compared with a preset significance threshold. If the statistic indicates that the difference is greater than the threshold, the null hypothesis is rejected and the verified component is taken as the root cause of the failure.

[0081] As one implementation, the policy executor 13 implemented in this application maintains a versioned policy library, which stores remediation policies defined in declarative files. These remediation policies map to corresponding automated operations based on different root cause types. When the root cause is a model version issue, the application programming interface of the model management service is invoked to perform a model version rollback. When the root cause is a service code version issue, the rollback pipeline of the continuous integration / continuous deployment system is triggered. When the root cause is a dependency library version issue, the package management tool is invoked to perform a dependency library downgrade installation. The policy executor 13 atomically executes the automated operations and records audit logs. The policy library supports adding custom policies, defining new root cause types, matching conditions, and remediation action sequences through standardized YAML templates, extending self-healing capabilities without modifying the executor's core code.

[0082] Figure 2 This paper illustrates an application diagram of the fault root cause localization and isolation system according to an embodiment of this application, such as... Figure 2As shown, the application methods of each processing component in the fault root cause localization and isolation system of this application embodiment are as follows:

[0083] The monitoring metrics collector 10 continuously collects performance, business, and resource metrics from the main production pipeline, providing a data foundation for fault detection. The monitoring metrics collector 10 employs a multi-dimensional data collection architecture, implemented by a lightweight agent deployed on the inference server, and utilizes a scalable collection framework such as OpenTelemetry for standardized metric collection. For hardware resource metrics, it collects data such as CPU utilization, memory usage, GPU memory utilization, and temperature once per second by reading Linux system files (e.g., / proc / stat, / proc / meminfo) and calling GPU vendor interfaces. For business metrics, the collector is integrated into an inference framework (e.g., TensorFlow Serving, Triton Inference Server), intercepting gRPC / REST API calls to collect latency, throughput, and error codes for each request in real time. The data is serialized using ProtoBuf and streamed via gRPC to a time-series database (e.g., Prometheus or TimescaleDB) for storage, providing a high-performance query interface for subsequent analysis.

[0084] The shadow pipeline 11 in this embodiment serves as a parallel and isolated inference environment from the main production pipeline and is a key component of the invention. Its key features are non-intrusiveness; it can receive and process copies of the same input data as the main pipeline but does not return results to the user, remaining completely transparent and unobtrusive. Furthermore, its pluggable component design supports flexible configuration of different versions of models, code, or dependency libraries. For example, it can simultaneously run various combinations such as new model (A) + old code (Old), old model (B) + new code (New). Its technical implementation is as follows:

[0085] Intelligent Traffic Replication and Lossless Mirroring: The input to Shadow Pipeline 11 originates from real-time replication of production traffic. This embodiment employs the traffic mirroring functionality of a service mesh (such as Istio) as the implementation method. By configuring mirroring rules on the service mesh's data plane (such as the Envoy proxy), requests flowing into the main production pipeline can be copied to Shadow Pipeline 11 in a lossless manner. This process is completely transparent to the application and maintains all characteristics of the original requests, including HTTP headers, body content, and session state, ensuring consistency between experimental and real traffic. Compared to simple load balancer replication, the service mesh solution better handles complex call relationships between microservices, ensuring that traffic throughout the entire call chain is correctly replicated.

[0086] Leveraging cloud-native strong isolation and environment construction: Shadow Pipeline 11 is deployed in a separate Kubernetes namespace, which forms the basis for resource and environment isolation. Through Kubernetes' resource quotas and limit ranges, the CPU, memory, and GPU resources used by Shadow Pipeline 11 can be strictly limited, ensuring that any experiments conducted will not crowd out or interfere with the performance of the main production environment. Furthermore, network policies can be used to restrict communication between Shadow Pipeline 11 and other sensitive components in the production system, further ensuring secure isolation.

[0087] Component version traceability and dynamic pluggability: Each component in the pipeline (such as machine learning models, inference service code, third-party dependency libraries, etc.) is encapsulated as an immutable Docker container image and versioned using tags (e.g., model-service:v1.2.3). All images are stored in a unified container repository, ensuring environment consistency. Dynamic pluggability is achieved through the Kubernetes API. The causal inference engine 12 does not require a complex continuous deployment system; it can directly call the Kubernetes API to deploy shadow pipeline 11 to the tag field of the container image, achieving second-level component version switching. For example, the model-service image can be replaced from v1.2.3 (problem version) to v1.2.2 (stable version). This declarative API-based approach is highly reliable and efficient, key to achieving automated experiments.

[0088] The collection and association of experimental metadata and results: Every request processed by Shadow Pipeline 11 has complete traceability. The system generates a unique Experiment ID for each batch of experimental requests. This ID is injected into the request context when processing replication traffic. After the pipeline is completed, it not only outputs the results (such as inference results and latency), but also structurally records and sends the complete metadata of this experiment, such as the Experiment ID, the component image version tag used, the processing timestamp, and resource usage, to a high-performance message queue (such as Apache Kafka). This design enables massive amounts of experimental results to be processed in an orderly and asynchronous manner. Causal inference engine 12, as a consumer, can read data from Kafka in real time and easily correlate and compare the results of Shadow Pipeline 11 with the results of the main production pipeline based on the Experiment ID.

[0089] In summary, the shadow pipeline 11 of this application embodiment achieves full-process automation from traffic replication to environment isolation to component switching and data collection through the aforementioned technical means. As the cornerstone of closed-loop automation and self-healing, it provides a safe, reliable, and efficient sandbox-style experimental field for the causal inference engine 12. The causal inference engine 12 can boldly and repeatedly verify various hypotheses in this sandbox, ultimately accurately locating the root cause of the failure and providing irrefutable data evidence for the final self-healing operation (such as version rollback) of the policy executor 13.

[0090] The causal reasoning engine 12 in this embodiment serves as the intelligent decision-making center of the system and is implemented using a microservice architecture. Internally, the engine maintains a dynamically updated causal graph. ,in, It is a set of nodes that represents the system state, including component versions. (such as models, code, dependencies, etc.) and performance metrics (like , ). The edge set represents a causal relationship, and the edge... Represents a node The change is the node The cause of change. For example, ( , This indicates that deploying model version A may cause increased latency in P99.

[0091] Initial weights of edges Based on historical failure data, Granger causality is initially established using Granger causality analysis. For example, Granger causality is tested using the following vector autoregressive (VAR) model:

[0092]

[0093] in, It is an autoregressive coefficient matrix, used to measure... The impact of its historical values ​​on the current value. This is the cross-regression coefficient matrix, used to measure the influence of historical values ​​of X on the current value of Y. If it includes... Past values ​​can significantly improve the understanding of Predictive power of the current value (i.e., the null hypothesis) If rejected, then it is considered yes Grangerin, and construct an edge in the graph.

[0094] The causal graph in this embodiment is dynamically updated. After each experiment, regardless of success or failure, the results are recorded, and the topology and edge weights of the graph are iteratively optimized using a Bayesian network structure learning algorithm to make it increasingly accurate in reflecting the true causal structure of the system.

[0095] When a failure occurs, the causal inference engine 12 generates a set of candidate hypotheses. Each of the hypotheses This means that a component is the root cause (e.g.) =="Model_vA is faulty"). This models the search for the root cause component leading to the failure as a black-box function optimization problem, defining the objective function. Input a hypothesis Output the benefits of validating this hypothesis (e.g.) Or costs such as experimental time. The objective of this application's embodiments is to find the hypothesis that maximizes the benefits. ,Right now Due to the call (i.e., conducting a shadow experiment) is costly, so this application uses Bayesian optimization to approximate this function.

[0096] The optimizer's surrogate model predicts the probability of each hypothesis being true and its uncertainty based on historical experimental data, and uses a Gaussian process (GP) to fit the objective function. GP provides a probability distribution for the function values, which is entirely determined by the mean function. Sum of covariance functions Decision. Given points that have already been tested. A new hypothesis The function values ​​follow a posterior distribution:

[0097]

[0098] Among them, the mean and variance Solutions with closed-form: , . It is the covariance matrix between the experimental points. yes The covariance vector between the experimental points and the known experimental points. Variance The uncertainty of the forecast has been quantified.

[0099] Get function Based on the posterior distribution of the surrogate model, the selection of regions with high uncertainty and high predicted returns is balanced to guide the hypothesis for the next optimal experimental point. Expected Improvement (EI) is used as the acquisition function.

[0100]

[0101] in, This is the best observation among the currently tested points. For a Gaussian process, EI has an analytical expression, as follows:

[0102]

[0103] in, , and These are the cumulative distribution function and probability density function of the standard normal distribution, respectively. This is a parameter used to adjust the exploration-exploitation tradeoff. The next experimental hypothesis is chosen to maximize the yield function. .

[0104] The causal inference engine 12 sends experimental instructions to the controller of the shadow pipeline 11 via an asynchronous message queue. During each experiment, the shadow pipeline 11 configures specific component combinations (such as a combination of the old model and new code) according to the engine's instructions and uses replicated real-time traffic for inference computation. After the shadow experiment is completed, the engine receives the results, and its statistical analysis module calculates the statistical difference between the output of the shadow pipeline 11 and the main pipeline output. The null hypothesis is defined. The component currently being tested This is not the root cause of the failure (i.e., the output distribution of shadow pipe 11 is not significantly different from that of the main pipe). The testing method should be selected based on different output information. For numerical indicators (such as delay), a two-sample T-test or Mann-Whitney U test should be used. For embedding vectors, cosine similarity should be calculated. Taking a classification model as an example, KL divergence can be calculated to measure the difference between the two probability distributions.

[0105]

[0106] This formula measures the distribution when using the main production pipeline. To approximate the actual distribution of shadow pipe 11 This results in information loss.

[0107] Based on whether the output difference under a certain component combination is significantly reduced (e.g., by setting a configurable threshold), the replaced component in that combination is determined to be the root cause component with high confidence. A statistic is then calculated. (For example: KL divergence value, t-value of T-test, average delay difference between two pipelines, etc.) and a pre-set significance level. (e.g., 0.05) for comparison. If Then reject the null hypothesis. Accept alternative assumptions (i.e., component) (This is the root cause), and the experiment was successful. Otherwise, it would be impossible to refuse. The experiment continues. Through this architecture that deeply integrates causal theory, Bayesian optimization, and statistical inference, the causal inference engine 12 can automatically locate the root cause of failures in complex systems with the fewest experiments and the highest confidence level, and push the diagnostic conclusions to the policy executor 13. The entire testing process, thanks to the use of machine learning-optimized priority ranking, can quickly locate the root cause with the fewest experiments, improving fault diagnosis efficiency.

[0108] The policy executor 13 in this embodiment is a key component for realizing the system's self-healing capability; essentially, it is an automated operation and maintenance robot. It maintains a versioned policy library, with policies stored in YAML files, defining the remediation actions corresponding to different root causes. For example, when the root cause is a model version, the policy calls the API of a model management service (such as MLflow) to roll back the production environment model version to the previous stable version; when the root cause is a code version, the policy triggers the rollback pipeline of the CI / CD system to redeploy the service container image to the old version; when the root cause is a dependency library, the policy calls package management tools (such as pip, conda) to execute a downgrade installation command. The executor integrates with the entire technology stack (Kubernetes, model repository, CI / CD system) through APIs. Upon receiving a diagnostic conclusion, it atomically executes the predefined policies and confirms the success of the policy execution by querying the APIs of each system. All operations are recorded in detail in the audit log for subsequent traceability and review. The entire process from detection and localization to recovery forms a complete automated closed loop, requiring no manual intervention, significantly improving the system's reliability and availability. The entire process requires no human intervention, achieving fault self-healing within seconds or minutes.

[0109] The fault root cause localization and isolation system of this application constitutes a complete closed-loop system from fault detection, intelligent root cause analysis to automatic self-healing, which greatly improves the efficiency and reliability of model inference testing.

[0110] Figure 3 This diagram illustrates the implementation flow of the fault root cause localization and isolation method according to an embodiment of this application. The fault root cause localization and isolation method according to an embodiment of this application is applied to... Figure 1 and Figure 2 In the fault root cause localization and isolation system shown, such as Figure 3As shown, the fault root cause localization and isolation method of this application embodiment includes the following processing steps:

[0111] Step 301: Collect various indicators of the production environment of the running model inference service. If a fault is detected based on the indicators, construct and maintain a dynamic causal graph representing the causal relationship between system states based on historical data.

[0112] The various metrics include performance, business, and resource metrics for the main production pipeline. For example, for hardware resource metrics, data such as CPU utilization, memory usage, GPU memory utilization, and temperature are collected once per second by reading Linux system files (e.g., / proc / stat, / proc / meminfo) and calling GPU vendor interfaces. For business metrics, the collector is integrated into an inference framework (e.g., TensorFlow Serving, Triton Inference Server), intercepting gRPC / REST API calls to collect latency, throughput, and error codes for each request in real time. Data is serialized using ProtoBuf and streamed via gRPC to a time-series database (e.g., Prometheus or TimescaleDB) for storage, providing a high-performance query interface for subsequent analysis.

[0113] The process involves constructing and maintaining a dynamic causal graph representing the causal relationships between system states based on historical data. This includes: establishing causal edges using Granger causality analysis based on historical fault data; and iteratively updating the topology and edge weights of the causal graph using a Bayesian network structure learning algorithm after each experiment.

[0114] Step 302: Generate candidate hypotheses based on the dynamic cause-effect graph, with each candidate hypothesis corresponding to a suspected root cause component.

[0115] In this embodiment, the causal inference engine responds to the monitoring index collector detecting a fault by constructing and maintaining a dynamic causal graph based on historical fault data. Nodes in the dynamic causal graph represent system states, which include component versions and performance indicators. Edges represent causal relationships and weights between nodes. Candidate hypotheses are generated based on the dynamic causal graph, with each candidate hypothesis corresponding to a suspected root cause component of the fault. The search for the root cause component of the fault is modeled as a black-box function optimization problem.

[0116] Step 303: Using the Bayesian optimization method, select the next best candidate hypothesis to be verified.

[0117] The embodiments of this application employ a Bayesian optimization method. Based on historical experimental results, a Gaussian process is used as a surrogate model to predict the verification benefits and uncertainties of each candidate hypothesis, and the next best candidate hypothesis to be verified is selected through the expectation improvement acquisition function.

[0118] Step 304, in response to receiving experimental instructions from the shadow pipeline, instructs the shadow pipeline configuration to combine component versions corresponding to the best candidate hypothesis.

[0119] In this embodiment, the shadow pipeline replicates production traffic losslessly through the traffic mirroring function of the service mesh; according to the experimental instructions, it dynamically switches the container image versions of one or more components by calling the application programming interface of the container orchestration platform; it generates a unique identifier for the experimental traffic and records the processing results and experimental metadata to the message queue.

[0120] Each component in the pipeline, such as the machine learning model, inference service code, and third-party dependencies, is encapsulated as an immutable Docker container image and version-managed using tags (e.g., model-service:v1.2.3). All images are stored in a unified container repository to ensure environment consistency. The causal inference engine does not require a complex continuous deployment system; it can directly deploy shadow pipelines by calling the Kubernetes API and tagging the container image. Experimental component versions include at least one of the following: different versions of the model, different versions of the inference service code, or different versions of the dependencies.

[0121] Step 305: The real-time copy of the production environment traffic is used as the experimental traffic, and the component version is dynamically configured and the experimental traffic is processed according to the experimental instructions.

[0122] By using the traffic mirroring function of the service mesh, requests flowing into the main production environment are copied without loss, and the copied requests are sent to the shadow pipeline as the experimental traffic.

[0123] Step 306: Return the processing results of the shadow pipeline to the causal reasoning engine.

[0124] Step 307: Using the causal reasoning engine, the processing results of the shadow pipeline are statistically compared with the output of the production pipeline and hypothesis testing is performed to determine the root cause component of the failure.

[0125] In this embodiment of the application, the selection of the next best candidate hypothesis to be verified includes: using a Gaussian process as a surrogate model to fit the posterior distribution of historical experimental results; and using the expected improvement acquisition function to weigh the regions with high prediction returns and high uncertainty, and select the next experimental hypothesis.

[0126] The processing results of the shadow pipeline are statistically compared and hypothesis tested with the output of the production pipeline, including: selecting the appropriate statistical test method or difference measurement algorithm according to the type of output data; calculating the test statistic or difference value and comparing it with the preset significance threshold; if it is greater than the significance threshold, the component verified in the current experiment is taken as the root cause of the failure.

[0127] Step 308: Execute predefined automated repair operations based on the determined root cause of the fault.

[0128] The implementation of the fault root cause localization and isolation method in this application embodiment can be understood by referring to the relevant description of the aforementioned fault root cause localization and isolation system.

[0129] According to embodiments of this application, this application also describes an electronic device and a readable storage medium.

[0130] Figure 4 A schematic block diagram of an example electronic device that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0131] like Figure 4 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0132] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the root cause localization and isolation method. For example, in some embodiments, the root cause localization and isolation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the root cause localization and isolation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the steps of the fault root cause localization and isolation method by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0140] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0141] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0142] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A fault root cause localization and isolation system, characterized in that, The system includes: a monitoring metric collector, a shadow pipeline, a causal inference engine, and a policy executor, wherein: A monitoring metrics collector is used to collect performance metrics, business metrics, and resource metrics of the production environment; the production environment includes the production environment of the model inference service. The shadow pipeline, deployed in isolation from the production environment, is used to receive real-time copies of the production environment traffic as experimental traffic, and dynamically configures the experimental component version based on the experimental instructions issued by the causal inference engine to process the experimental traffic; the experimental component version includes at least one of the following: different versions of the model, different versions of the inference service code, and different versions of the dependency library. The causal inference engine responds to the monitoring indicator collector detecting a fault by constructing and maintaining a dynamic causal graph based on historical fault data. Nodes in the dynamic causal graph represent system states, including component versions and performance indicators, while edges represent causal relationships and weights between nodes. Candidate hypotheses are generated based on the dynamic causal graph, with each candidate hypothesis corresponding to a suspected root cause component. The search for the root cause component is modeled as a black-box function optimization problem, employing a Bayesian optimization method. Based on historical experimental results, a Gaussian process is used as a surrogate model to predict the validation benefit and uncertainty of each candidate hypothesis, and the next best candidate hypothesis to be validated is selected through an expected improvement function. Experimental instructions are sent to the shadow pipeline, containing the component version configuration corresponding to the best candidate hypothesis. The experimental results returned by the shadow pipeline are received and analyzed: the output results of the shadow pipeline under a specific component version configuration are statistically compared with the output results of the production pipeline under a fault state. Hypothesis testing is performed based on a preset significance level. If the statistical difference narrows, the component replaced in the specific component version configuration is determined to be the root cause component of the fault. The strategy executor is communicatively connected to the causal inference engine and is used to receive the root cause of the fault determined by the causal inference engine, and to call the application programming interface of the external system to perform fault isolation and recovery according to the predefined automatic repair strategies corresponding to different root cause types.

2. The system according to claim 1, characterized in that, The shadow conduit includes: The traffic replication module is used to perform lossless replication of requests flowing into the production environment through the traffic mirroring function of the service mesh, and send the replicated requests as the experimental traffic to the shadow pipeline. The environment isolation module is used to deploy the shadow pipeline in an independent namespace and ensure that the computing, storage and network resources used by the shadow pipeline are isolated from the production environment through resource quota restrictions and network policies. The component management module is used to encapsulate each component into a container image with version tags. In response to the experimental instructions of the causal inference engine, it dynamically replaces the version tags of the container images in the shadow pipeline by calling the application programming interface of the container orchestration platform, so as to achieve the second-level switching of component versions. The data collection module is used to generate a unique experiment identifier for each batch of experimental traffic, record the output results and experimental metadata during the shadow pipeline processing, and send the output results and experimental metadata to the message queue; the experimental metadata includes the experiment identifier, the component image version tag used, the processing timestamp, and resource usage.

3. The system according to claim 1 or 2, characterized in that, The causal reasoning engine is also used for: Based on historical fault data, Granger causality analysis is used to establish causal edges and initial weights between nodes. Granger causality analysis uses a vector autoregression model to test whether past values ​​of the time series to be processed can improve the predictive ability of the current value of another time series. After the experiment, the topology and edge weights of the dynamic causal graph were iteratively optimized using the Bayesian network structure learning algorithm and the new experimental results.

4. The system according to claim 3, characterized in that, The causal reasoning engine is also used for: Define the benefit or cost of validating candidate hypotheses as the objective function; Using a Gaussian process as a surrogate model for the objective function, the posterior probability distribution of the objective function is obtained by fitting the completed experimental data; the posterior probability distribution is determined by the mean function and the covariance function, and is used to predict the function value and uncertainty of the new candidate hypothesis; Expected improvement is used as the acquisition function to balance the utilization of regions with high predictive returns with the exploration of regions with high uncertainty; The candidate hypothesis that yields the largest function value is selected as the next best candidate hypothesis to be tested.

5. The system according to claim 1, characterized in that, The causal reasoning engine is also used for: Numerical indices are tested using a two-sample t-test or the Mann-Whitney U-test. For probability distribution outputs, the Kohlbek-Leibler divergence is calculated to measure the differences in distribution. The calculated statistic is compared with a preset significance threshold. If the statistic indicates a difference greater than the threshold, the null hypothesis is rejected and the verified component is taken as the root cause of the failure.

6. The system according to claim 1, characterized in that, The policy executor maintains a versioned policy library, which stores repair policies defined in declarative files. These repair policies are mapped to corresponding automated operations based on different root cause types of failures. When the root cause is the model version, the application interface of the model management service is called to perform a model version rollback. When the root cause is the service code version, the rollback pipeline of the continuous integration / continuous deployment system is triggered; When the root dependency library version is incorrect, the package management tool is invoked to perform a dependency library downgrade installation. The policy executor atomically executes the automated operation and records an audit log.

7. A method for fault root cause localization and isolation applied to the system described in any one of claims 1 to 6, characterized in that, The method includes: Collect various indicators of the production environment of the running model inference service. If a fault is detected based on the indicators, construct and maintain a dynamic causal graph representing the causal relationship between system states based on historical data. Candidate hypotheses are generated based on dynamic cause-effect graphs, and each candidate hypothesis corresponds to a suspected root cause component of the failure. The Bayesian optimization method is used to select the next best candidate hypothesis to be tested. In response to receiving experimental instructions from the shadow pipeline, the shadow pipeline configuration is indicated to match the component version combination corresponding to the best candidate hypothesis; The real-time copy of the production environment traffic is used as the experimental traffic, and the experimental traffic is processed after the component version is dynamically configured according to the experimental instructions. Return the processing results of the shadow pipeline to the causal reasoning engine; The causal reasoning engine statistically compares the processing results of the shadow pipeline with the output of the production pipeline and performs hypothesis testing to determine the root cause component of the failure. Based on the identified root cause of the fault, perform predefined automated repair operations.

8. The method according to claim 7, characterized in that, The shadow pipeline replicates production traffic losslessly through the service mesh's traffic mirroring function; According to the experimental instructions, the container image versions of one or more components are dynamically switched by calling the application programming interface of the container orchestration platform; Generate a unique identifier for the experimental traffic and record the processing results and experimental metadata to the message queue.

9. The method according to claim 7, characterized in that, The dynamic causal graph, which is constructed and maintained based on historical data to represent the causal relationships between system states, includes: Based on historical fault data, causal edges are established using Granger causality analysis. After each experiment, the topology and edge weights of the causal graph are iteratively updated using a Bayesian network structure learning algorithm.

10. The method according to claim 7, characterized in that, The selection of the next best candidate hypothesis to be verified includes: A Gaussian process was used as a surrogate model to fit the posterior distribution of historical experimental results; By employing the expected improved acquisition function, a trade-off is made between regions with high predicted returns and high uncertainty, and the next experimental hypothesis is selected. Correspondingly, the step of statistically comparing and hypothesis testing the processing results of the shadow pipeline with the output of the production pipeline includes: Select the appropriate statistical test method or difference measurement algorithm based on the type of output data; Calculate the test statistic or difference value and compare it with the preset significance threshold; If the value exceeds the significance threshold, the component verified in the current experiment will be taken as the root cause of the failure.

Images