Fault root cause positioning method, device, equipment, medium and product

Abstract

The invention provides a fault root cause positioning method and device, equipment, a medium and a product, and relates to the technical field of IT, the method comprises the steps that cluster multi-modal data collected by a data collection agent is acquired, and the multi-modal data comprises structured time sequence monitoring data and unstructured operation logs; performing multi-modal feature fusion on the time sequence monitoring data and the running log to obtain a fusion feature vector representing the current state of the cluster; according to the fusion feature vector, performing anomaly detection on the cluster to obtain a detection result, the detection result including an anomaly score and an anomaly category; and under the condition that the anomaly score is greater than a first threshold value, performing fault root cause analysis according to the anomaly category, the time sequence monitoring data related to the anomaly category and the fusion feature vector to obtain an analysis result. Therefore, the problem that abnormity diagnosis based on single-mode data is likely to fail is solved.

Description

Technical Field

[0001] This application relates to the field of information technology (IT), and in particular to a method, apparatus, equipment, medium and product for locating the root cause of a fault. Background Technology

[0002] With the adoption of advanced architectures such as distributed systems, microservices, and cloud-native architectures in IT enterprises, these architectures support agile development, rapid iteration, and elastic scaling. They break down monolithic applications into multiple independently deployed, interconnected applications, giving IT business systems high scalability and flexibility. However, with the exponential growth in the number of applications, the dependencies between business modules become increasingly complex, and it is difficult to establish real-time and effective mapping relationships between different business layers and dimensions. Simultaneously, with frequent container starts and stops, changes in monitoring objects and their metrics become commonplace, making it difficult to preserve fault scenes and effectively locate faults. This inherent complexity not only increases the difficulty of fault diagnosis but may also trigger cascading failures, causing profound negative impacts on system operational efficiency and user experience.

[0003] Existing fault diagnosis solutions are primarily based on single-modal data, employing anomaly detection algorithms and combining them with the experience of operations and maintenance experts to construct rule-based channel-based fault diagnosis systems. The operations and maintenance field possesses abundant multi-source and multi-type data, such as metric data, log data, and call trace data, among other multi-modal operations and maintenance data. Existing fault diagnosis systems based on single-modal data may fail to capture or completely omit certain anomaly information caused by faults, potentially leading to diagnostic failures. Summary of the Invention

[0004] This application provides a method, apparatus, device, medium, and product for locating the root cause of a fault, which solves the problem that fault diagnosis based on single-modal data may lead to diagnosis failure.

[0005] Firstly, to achieve the above objectives, embodiments of this application provide a method for locating the root cause of a fault, the method comprising:

[0006] Acquire multimodal data of the cluster collected by the data acquisition agent, wherein the multimodal data includes structured time-series monitoring data and unstructured operation logs;

[0007] Multimodal feature fusion is performed on the time series monitoring data and the operation log to obtain a fused feature vector representing the current state of the cluster;

[0008] Based on the fused feature vector, anomaly detection is performed on the cluster to obtain detection results, which include anomaly score and anomaly category;

[0009] If the anomaly score is greater than the first threshold, a root cause analysis is performed based on the anomaly category, the time-series monitoring data related to the anomaly category, and the fused feature vector to obtain the analysis results.

[0010] Specifically, based on the fused feature vector, anomaly detection is performed on the cluster to obtain detection results, including:

[0011] The fused feature vector is input into a multimodal anomaly discrimination network to obtain the anomaly score and anomaly category of the current state of the cluster by the multimodal anomaly discrimination network.

[0012] Specifically, the fused feature vector is input into a multimodal anomaly detection network to obtain the anomaly score and anomaly category of the current state of the cluster by the multimodal anomaly detection network, including:

[0013] The feature vector corresponding to the real-time data in the fused feature vector is input into the adjustable encoder of the multimodal anomaly discrimination network to obtain the first feature vector output by the adjustable encoder;

[0014] The feature vector corresponding to the positive label data in the fused feature vector is input into the freeze encoder of the multimodal anomaly detection network to obtain the second feature vector output by the freeze encoder; the positive label data is pre-labeled data;

[0015] Using a loss function, calculate the loss values ​​corresponding to the first feature vector and the second feature vector;

[0016] If the loss value is greater than the second threshold, the feature vector corresponding to the real-time data is input into the discrimination network to obtain the anomaly score and anomaly category.

[0017] Specifically, the root cause analysis of the fault is performed based on the anomaly category, the time-series monitoring data related to the anomaly category, and the fused feature vector to obtain the analysis results, including:

[0018] A causal strength score is obtained by comparing and testing the regression models constructed from the values ​​of any two time series indicators at L time points; L is the maximum lag order, and the two time series indicators belong to the time series monitoring data related to the anomaly category.

[0019] Based on the time series indicators and the causal strength score, a causal dependency graph between different nodes is constructed; the nodes are the nodes in the cluster.

[0020] The causal dependency graph and the fused feature vector are input into a pre-designed ranking algorithm to obtain the influence score of each node;

[0021] The analysis results are obtained based on the influence score and the causal dependency graph. The analysis results include candidate root cause nodes and explanatory chain information.

[0022] Specifically, a regression model constructed from the values ​​of any two time series indicators in the time series monitoring data at L time points is compared and tested to obtain a causal strength score, including:

[0023] If the values ​​of any two time series indicators at L time points satisfy the pre-set stationarity condition, the maximum lag order L is selected using the Akaike Information Content Criterion (AIC) minimization principle.

[0024] For a target time series indicator, the F-statistic of each regression model group is calculated based on the regression model group containing the target time series indicator and the regression model group not containing the target time series indicator; wherein, the target time series indicator is one of the two time series indicators.

[0025] The causal strength score is obtained based on the difference between the two F-statistics.

[0026] Specifically, based on the time series indicators and the causal strength score, a causal dependency graph among different indicators is constructed, including:

[0027] Based on the derivative relationships of the statistical time series indicators, the initial causal edge weights of the static service dependency graph are calculated.

[0028] The initial causal edge weights are adjusted based on the causal strength score.

[0029] The corrected initial causal edge weights are fused with the predefined rule weights in the knowledge base to obtain the target causal edge weights;

[0030] The causal dependency graph is obtained based on the target causal edge weights.

[0031] The method further includes:

[0032] Based on at least one of the following: cluster operation and maintenance logs and monitoring data, fault cases, operation and maintenance manuals and emergency manuals, official documents and maintenance manuals of each software product, application and device example data, topology relationship data, operation and maintenance documents, and work order records, construct operation and maintenance knowledge base data;

[0033] The operation and maintenance knowledge base data is injected into the locally deployed multimodal anomaly detection network; wherein, the multimodal anomaly detection network includes a temporary low-rank adaptation module;

[0034] The multimodal anomaly discrimination network is iteratively optimized.

[0035] The method further includes:

[0036] The data acquisition agent uses the Dynamic Weighted Network Time Protocol (NTP) to periodically synchronize its time with multiple reference time sources within the cluster.

[0037] Secondly, to achieve the above objectives, embodiments of this application provide a fault root cause localization device, the device comprising:

[0038] The acquisition module is used to acquire multimodal data of the cluster collected by the data acquisition agent. The multimodal data includes structured time-series monitoring data and unstructured operation logs.

[0039] The feature fusion module is used to perform multimodal feature fusion on the time series monitoring data and the operation log to obtain a fused feature vector characterizing the current state of the cluster.

[0040] An anomaly detection module is used to perform anomaly detection on the cluster based on the fused feature vector and obtain detection results, the detection results including anomaly score and anomaly category;

[0041] The analysis module is used to perform root cause analysis of the fault based on the anomaly category, time series monitoring data related to the anomaly category, and the fused feature vector when the anomaly score is greater than a first threshold, and to obtain the analysis results.

[0042] Thirdly, to achieve the above objectives, embodiments of this application provide a fault root cause localization system, including a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; the transceiver transmits and receives data under the control of the processor, and the processor executes the program to implement the fault root cause localization method as described in the first aspect.

[0043] Fourthly, to achieve the above objectives, embodiments of this application provide a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the fault root cause localization method as described in the first aspect.

[0044] Fifthly, to achieve the above objectives, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the fault root cause localization method as described in the first aspect.

[0045] The beneficial effects of the above technical solution in this application are as follows:

[0046] In the fault root cause localization method of this application embodiment, firstly, multimodal data of the cluster collected by the data acquisition agent is acquired, the multimodal data including structured time-series monitoring data and unstructured operation logs; secondly, multimodal feature fusion is performed on the time-series monitoring data and the operation logs to obtain a fused feature vector representing the current state of the cluster; thirdly, anomaly detection is performed on the cluster based on the fused feature vector to obtain detection results, the detection results including anomaly score and anomaly category; finally, if the anomaly score is greater than a first threshold, fault root cause analysis is performed based on the anomaly category, the time-series monitoring data related to the anomaly category, and the fused feature vector to obtain analysis results. Thus, anomaly detection is achieved using multimodal data that comprehensively reflects the system's operating state, enabling adaptation and diagnosis of a wider range of fault types and the discovery of finer-grained fault root causes. This solves the problem that fault diagnosis based on single-modal data may fail due to the inability to capture or complete omission of abnormal information caused by certain faults. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the fault root cause localization method according to an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the anomaly detection framework according to an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of the root cause analysis framework of an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the large model training process according to an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of the Temp-Lora method according to an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of the structure of the fault root cause location device according to an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of a fault root cause localization device according to an embodiment of this application. Detailed Implementation

[0054] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0055] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0056] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0057] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0058] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A, but can also be determined based on A and / or other information.

[0059] Embodiments of this application provide a method for locating the root cause of a fault, such as... Figure 1 As shown, the method includes:

[0060] Step 101: Obtain multimodal data of the cluster collected by the data acquisition agent. The multimodal data includes structured time-series monitoring data and unstructured operation logs.

[0061] For example, the above steps can involve collecting monitoring data from a distributed monitoring environment, and then deriving multimodal data from this monitoring data. Structured time-series monitoring data includes metrics such as Central Processing Unit (CPU) utilization, memory usage, disk input / output, and network latency; unstructured runtime logs include service call records and error text. In this way, the multimodal data can cover cluster operating status, performance metrics, and historical behavior.

[0062] For example, in a distributed containerized environment of an operations and maintenance system, a data acquisition agent is deployed inside each container (Pod) of the cluster (such as Kubernetes) and is responsible for collecting container logs, performance metrics and network traffic data in real time.

[0063] Step 102: Perform multimodal feature fusion on the time series monitoring data and the operation log to obtain a fused feature vector representing the current state of the cluster.

[0064] In the above steps, by fusing time series monitoring data and operation logs with multimodal features, the system's operating status is comprehensively reflected from information from different sources, providing more accurate data for subsequent anomaly detection. Compared with relying solely on single-modal time series data, this can improve the accuracy of anomaly detection and avoid diagnostic failures.

[0065] Step 103: Based on the fused feature vector, perform anomaly detection on the cluster to obtain detection results, which include anomaly score and anomaly category.

[0066] In the above steps, anomaly detection is performed based on the fused feature vector obtained from multimodal data fusion. Compared with anomaly detection based solely on single-modal time-series data, this method can improve the accuracy of anomaly detection and avoid diagnostic failures.

[0067] Step 104: If the anomaly score is greater than the first threshold, perform root cause analysis based on the anomaly category, the time series monitoring data related to the anomaly category, and the fused feature vector to obtain the analysis results.

[0068] In the fault root cause localization method of this application embodiment, firstly, multimodal data of the cluster collected by the data acquisition agent is acquired, including structured time-series monitoring data and unstructured operation logs; secondly, multimodal feature fusion is performed on the time-series monitoring data and the operation logs to obtain a fused feature vector representing the current state of the cluster; thirdly, anomaly detection is performed on the cluster based on the fused feature vector to obtain detection results, including anomaly scores and anomaly categories; finally, if the anomaly score is greater than a first threshold, fault root cause analysis is performed based on the anomaly category, the time-series monitoring data related to the anomaly category, and the fused feature vector to obtain analysis results. Thus, anomaly detection is achieved using multimodal data that comprehensively reflects the system's operating state, enabling adaptation to and diagnosis of a wider range of fault types and the discovery of finer-grained fault root causes. This solves the problem that fault diagnosis based on single-modal data may fail due to the inability to capture or complete omission of abnormal information caused by certain faults.

[0069] As an example, after step 101 above, the method may further include:

[0070] The multimodal data is preprocessed; the preprocessing may include at least one of the following, but is not limited to: format normalization, time alignment, and noise removal. This provides a clean data foundation for subsequent multimodal feature fusion. For example, this step can be implemented by a data preprocessing subsystem.

[0071] As an optional implementation, step 102 involves performing multimodal feature fusion on the time-series monitoring data and the operation logs to obtain a fused feature vector representing the current state of the cluster, including:

[0072] Sub-step 1: Extract features from the time-series monitoring data to obtain a multi-dimensional time-series representation vector. This vector includes feature vectors with periodic patterns, trend components, and anomalous fluctuations. For example, this sub-step can use a 12-layer Temporal Convolutional Network (TCN) for feature extraction. The TCN is a neural network model utilizing causal convolution and dilated / attenuated convolution. For instance, the time-series monitoring data includes 253 monitoring indicators, with each layer's dilation factor d increasing in powers of 2 (d=1, 2, 4, ..., 2048). The final output is a 512-dimensional time-series representation vector containing periodic patterns, trend components, and anomalous fluctuations. The calculation process satisfies the following:

[0073]

[0074] in, This represents the output feature vector of the l-th layer network at time step t. Let be the weight matrix of the k-th convolutional kernel in the l-th layer; The input features of the previous layer at time step t-dk; d is the inflation factor, d=2 for each layer. l ; is the bias term of the l-th layer; K is the size of the convolution kernel, usually K=3.

[0075] Sub-step two: Extract contextual semantic features from the runtime log and perform deep processing on these features to obtain a multi-dimensional log feature embedding vector. For example, this sub-step can be implemented using the DeepSeek R1 log encoder for multi-granularity analysis. This sub-step can be: log template extraction based on a Bidirectional Long Short-Term Memory (BiLSTM) network, such as abstracting "Failed to connect to 10.2.3.4:3306" as " <ip> : <port>"Connection failed"; 768-dimensional semantic embeddings were generated using a pre-trained model:

[0076]

[0077] Among them, E log The deep-processed log features are embedded into vectors; LayerNorm(·) is layer normalization, which stabilizes the training process and accelerates convergence; R1_Transformer(·) is an encoder based on the Transformer architecture to extract contextual semantic features; Tokenize(log_text) converts the original log text into a sequence of word vectors.

[0078] Sub-step 3: Perform cross-modal alignment on the multi-dimensional time-series representation vector and the multi-dimensional log feature embedding vector to obtain the fused feature vector. For example, this sub-step can use a designed cross-modal attention mechanism to calculate the association weight matrix between the multi-dimensional time-series identifier vector and the multi-dimensional log feature embedding vector:

[0079]

[0080] in, The association weight between the i-th time-series feature vector and the j-th log feature embedding vector; The i-th query vector of the time series feature vectors has a dimension of 512; Let j be the j-th key vector of the log feature embedding vector, with dimension 512; This is a scaling factor used to prevent the dot product value from becoming too large. Finally, a 1024-dimensional fused feature vector is generated through weighted concatenation. This will serve as a unified input feature in subsequent anomaly detection and root cause analysis, supporting key tasks such as multimodal anomaly identification, log tracing, and causal reasoning.

[0081] For example, the above optional implementation can be implemented by a feature extraction server cluster, which consists of 32 Graphics Processing Unit (GPU) nodes deployed on a cloud computing platform, including a dedicated TCN time series processor and a DeepSeek R1 text encoder. After cleaning the multimodal data, a parallel feature extraction process can be initiated. Specifically, the above optional implementation uses a dual-channel structure to process monitoring data and runtime logs separately. For time series metrics, a 12-layer dilated causal convolutional network (TCN) is constructed to extract periodic, trend, and short-term fluctuation features, generating a 512-dimensional time series representation vector. For log text, a Transformer-based DeepSeek R1 model is used for contextual semantic modeling, generating a 768-dimensional semantic embedding, which is then compressed and aligned to 512 dimensions using an attention mechanism. Finally, the two are concatenated into a fused feature vector. It is used to uniformly represent the current state of the system and is used for subsequent anomaly detection.

[0082] Among the aforementioned optional implementation methods, a cross-modal attention mechanism is designed to jointly analyze temporal fluctuations and textual semantics by utilizing multimodal data collected by the deployed data collection agent, such as performance metrics, container logs, and network traffic. This addresses the current reliance on multidimensional temporal data for root cause localization, which fails to integrate heterogeneous data sources such as log text and network traffic, resulting in a lack of contextual semantic information for abnormal events. For example, when service response latency suddenly increases, existing methods cannot quickly narrow down the investigation scope by combining "Remote Procedure Call (RPC) timeout" error messages in the logs, potentially misjudging resource contention as hardware failure.

[0083] The above-mentioned optional implementation methods not only rely on multi-dimensional time-series data, but also integrate various types of data from different sources (such as metric data, log data, call chain data, etc.), thereby further enhancing the comprehensiveness and accuracy of fault diagnosis.

[0084] As an optional implementation, step 103 involves performing anomaly detection on the cluster based on the fused feature vector to obtain detection results, including:

[0085] The fused feature vector is input into a multimodal anomaly detection network to obtain the anomaly score and anomaly category of the current state of the cluster. For example, the multimodal anomaly detection network can be a Transformer or a Multilayer Perceptron (MLP). In short, this step allows a pre-trained multimodal anomaly detection network to score the current state of the cluster based on the fused feature vector and further identify the anomaly category. For example, anomaly categories include resource bottlenecks, service timeouts, and connection anomalies.

[0086] For example, the multimodal anomaly discrimination network is a large model DeepSeek R1, which is deployed in a Kubernetes service mesh for real-time inference. The model size is compressed to 4.7GB using FP16 quantization technology, and the latency of a single inference is controlled within 8ms.

[0087] The two optional implementation methods mentioned above, by introducing a 12-layer dilated causal convolutional network (TCN) to process time series data and combining FP16 quantization technology to compress the model size (4.7GB), achieve a single inference latency of <8ms, thus achieving a balance between high accuracy and low latency. This solves the problem that the current fixed prediction model (such as LSTM) used to generate expected values ​​is difficult to adapt to nonlinear relationships in complex scenarios (such as resource competition caused by sudden traffic), and the model inference latency is high (>50ms), which cannot meet the needs of real-time operation and maintenance.

[0088] As a specific implementation, the step of "inputting the fused feature vector into a multimodal anomaly discrimination network to obtain the anomaly score and anomaly category of the current state of the cluster by the multimodal anomaly discrimination network" in the above optional implementation includes:

[0089] Sub-step 1: Input the feature vector corresponding to the real-time data in the fused feature vector into the adjustable encoder of the multimodal anomaly discrimination network to obtain the first feature vector output by the adjustable encoder; here, the real-time data is the data collected in real time by the data acquisition agent.

[0090] Sub-step 2: Input the feature vector corresponding to the positive label data in the fused feature vector into the frozen encoder of the multimodal anomaly discrimination network to obtain the second feature vector output by the frozen encoder; the positive label data is pre-labeled data; wherein, the feature vector corresponding to the positive label data, i.e. the second feature vector, is the multimodal data and features of the cluster in normal state; for example, the above-mentioned frozen encoder can also be called a frozen baseline encoder.

[0091] Here, it should be noted that, as Figure 2 As shown, the adjustable encoder and the frozen encoder constitute a dual-tower network structure. Based on this dual-tower network structure, the above sub-step one and sub-step two are comparative feature learning processes: positive label data generates a reference vector through the frozen encoder, and real-time data generates a comparison vector through the adjustable encoder.

[0092] Sub-step 3: Calculate the loss values ​​corresponding to the first feature vector and the second feature vector using the loss function; wherein the loss function is designed as follows:

[0093]

[0094] Among them, E live E is a feature vector generated from real-time data using an adjustable encoder. ref These are feature vectors generated from normal data using a frozen encoder. It is the square of the L2 norm, used to measure the distance in the feature space; Jensen-Shannon divergence, used to measure the difference in probability distributions; weights of the loss term. =0.6, =0.4; P live P ref These are represented as the current strategy parameters and the reference strategy parameters, respectively. A balance between anomaly sensitivity and false alarm rate is achieved through dynamic adjustment.

[0095] Sub-step four: If the loss value is greater than the second threshold, input the feature vector corresponding to the real-time data into the discrimination network to obtain the anomaly score and anomaly category.

[0096] In other words, a significant increase in the loss value is an early signal of an abnormal system state, reflecting a larger deviation in the model's fitting or comparison of data features, implicitly indicating a potential problem with the system; abnormal loss values, such as those exceeding the second threshold, indicate a problem. Figure 2 As shown, this will further trigger the model to calculate an "anomaly score" (a precise quantification of the degree of anomaly). The subsequent root cause localization process is then triggered based on the anomaly score. For example, if the loss value is greater than a second threshold, the discriminant network determines the anomaly score and anomaly type based on real-time data. When the anomaly type is determined and the "anomaly score" exceeds the threshold, the root cause localization process is triggered.

[0097] Here, it should be noted that, firstly, the real data refers to the actual collected, non-falsified raw / preprocessed data. Specifically, the real data is multimodal data, meaning it corresponds to the aforementioned multimodal data. Figure 2 The feature vector corresponding to the real data in the above-mentioned implementation is the same feature vector after multimodal data fusion. The real-time data and positively labeled data in the above-mentioned implementation both fall under the category of real data. Positively labeled data refers to data with positive sample labels, that is, data after labeling the collected real data. Real-time data is the data to be compared. Positively labeled data refers to data collected and labeled during periods when no anomalies occurred in the system. Second, the dual-tower network structure and the discriminant network constitute a multimodal anomaly discriminant network. As mentioned earlier, the dual-tower network structure is... Figure 2 The framework corresponding to the contrastive feature learning in [the context].

[0098] As an optional implementation, step 104 involves performing root cause analysis based on the anomaly category, time-series monitoring data related to the anomaly category, and the fused feature vector to obtain analysis results, including:

[0099] Sub-step 1: Compare and test the regression model constructed from the values ​​of any two time series indicators at L time points to obtain a causal strength score; L is the maximum lag order, and the two time series indicators belong to the time series monitoring data related to the anomaly category. For example, let x be the value of any two time series indicators. i and x j For example, the causal strength score is obtained by comparing and testing a regression model constructed from values ​​over the past L time points. Here, L time points refer to the L time points within the abnormal interval.

[0100] Sub-step two: Construct a causal dependency graph between different nodes based on the time series indicators and the causal strength score; the nodes are nodes in the cluster; exemplarily, this causal dependency graph is constructed based on the static service dependency graph, using time series indicators and causal strength scores. In other words, the causal strength score is used not only to determine the influence between variables but also to construct the causal dependency graph between nodes.

[0101] Sub-step 3: Input the causal dependency graph and the fused feature vector into a pre-designed ranking algorithm to obtain the influence score of each node.

[0102] For example, the pre-designed ranking algorithm described above is an improved PageRank algorithm. Based on this, sub-step three can be expressed by the following formula:

[0103]

[0104] Among them, PR(v i ) is node v i The PageRank value reflects its global influence; N is the total number of nodes in the cause-effect graph; W ji For node v j to v i The edge weights reflect the strength of causality; In(v i ) for all pointers to v i The set of nodes; Out(v j ) for all indicated v j The set of nodes; the damping factor d is set to 0.85. The number of iterations does not exceed 20 to ensure real-time performance. In a certain end-to-end failure, the algorithm accurately identified the Redis cluster master-slave switchover failure as the root cause event from 387 abnormal indicators within 8 seconds.

[0105] Sub-step four: Based on the influence score and the causal dependency graph, obtain the analysis results, which include candidate root cause nodes and explanatory chain information.

[0106] The aforementioned optional implementations, based on graph neural networks to dynamically construct service topologies, can generate sub-topology graphs within 300ms. Combined with an improved PageRank algorithm, this ensures that root cause analysis adapts to the dynamic changes in the microservice architecture. This addresses the problem of current methods relying on static service dependency graphs, which fail to update dynamic relationships such as TCP connections and RPC calls between containers in real time. This can lead to the failure of root cause analysis paths when the Kubernetes cluster undergoes elastic scaling. For example, after a Pod restarts abnormally, changes in upstream and downstream service dependencies can cause traditional root cause analysis methods to become inaccurate.

[0107] As a specific implementation, sub-step one above involves performing root cause analysis based on the anomaly category, time-series monitoring data related to the anomaly category, and the fused feature vector to obtain analysis results, including:

[0108] If the values ​​of any two time series indicators at L time points satisfy a pre-set stationarity condition, the maximum lag order L is selected using the Akaike Information Criterion (AIC) minimization principle. For a target time series indicator, the F-statistic of each regression model group is calculated based on the regression model group containing the target time series indicator and the regression model group not containing the target time series indicator. The target time series indicator is one of the two time series indicators. The causal strength score is obtained based on the difference between the two F-statistics.

[0109] Following the previous example, let x be one of the target time series indicators. i For example, the specific implementation process described above is as follows: While ensuring the stationarity of each sequence, the maximum lag order L is automatically selected using the AIC minimization principle. Then, based on whether the sequence x is included or not... i The F-statistics of the two regression models are calculated, and the maximum difference between them is taken as G. i→j When G i→j If the value is greater than the significance threshold, for example, p < 0.05, then x is considered to be... i For x j There is a significant causal relationship, and this G i→j The value serves as a system score to reflect the causal strength between indicators. The causal strength can be expressed as:

[0110]

[0111] Where, x i and x j These are two time series indicators, where L is the maximum lag order and F(•) is the F-statistic of the regression model.

[0112] Additionally, in the example above, "each sequence" refers to the sequence formed by x. i and x j A sequence is constructed from the values ​​of two time series indicators at L time points, wherein the value of one time series indicator at one time point constitutes a sequence; in addition, the causal relationship in the embodiments of this application may specifically be a Granger causal relationship.

[0113] Here, to verify the applicability of the above specific implementation methods under various failure scenarios, two typical examples are given: a failure scenario of database connection pool exhaustion and an application programming interface (API) latency anomaly caused by upstream service load. These examples more intuitively illustrate the flexibility and physical significance of score calculation.

[0114] In the failure scenario of database connection pool exhaustion, the metric x, which represents the number of real-time active connections, was selected. C And an indicator x representing the current queue length of pending business requests. Q Through the Granger causality strength calculation process described above, i.e., constructing models containing and excluding x respectively. C The regression model of historical values ​​is used to calculate the corresponding F-statistic, and the maximum value of the difference within the lag order range of 1 to L is obtained. ,Right now:

[0115]

[0116] Among them, when the database connection pool resources are close to the limit, causing queuing and backlog, G C→Q This will significantly improve performance, which not only statistically demonstrates the causal impact of the number of connections on queue length, but also provides a basis for system alarms and root cause localization.

[0117] In scenarios where API latency anomalies are caused by upstream service load, the API response latency metric x is used as an example. L (Unit: milliseconds) and upstream service CPU utilization metrics x U (Unit: percentage) is the object of analysis. To better highlight the impact of recent load fluctuations, an exponentially decaying weighting function is introduced. Here, α is an empirically set value, and the weighted causality score is obtained by taking the maximum value of the F-differences for each lag order, based on the calculation of the standard Granger causality strength. In real-world scenarios, when a sudden surge in upstream CPU utilization causes service response latency, This will result in a significant increase, helping operations and maintenance personnel quickly pinpoint that the upstream load issue is causing the downstream API performance fluctuations.

[0118] As another specific implementation, sub-step two above: constructing a causal dependency graph between different indicators based on the time series indicators and the causal strength score, including:

[0119] 1) Calculate the initial causal edge weights of the static service dependency graph based on the derivative relationships of the statistical time series indicators; for example, this step can be expressed by the following formula:

[0120]

[0121] in, This represents the causal strength score calculated using statistical methods; T is the total length of the time series; τ is the time delay parameter, describing the lag effect of the causal relationship between variables; For the variable x in the time series j The value at time t, For x j The value at time t-τ.

[0122] 2) Adjust the initial causal edge weights based on the causal strength score.

[0123] 3) The corrected initial causal edge weights and the predefined rule weights in the knowledge base are fused together to obtain the target causal edge weights.

[0124] For example, steps 2) and 3) above can be based on the semantic knowledge scoring mechanism of the large model DeepSeek R1 to fuse and correct the initial causal edge weights.

[0125] 4) Obtain the causal dependency graph based on the target causal edge weights. For example, the causal dependency graph is represented as... The graph structure (V, E) inherits from the static service dependency graph; edge weights The final causal strength is obtained by fusing statistical reasoning and knowledge rules; the fusion coefficient β is dynamically determined by the DeepSeek R1 model based on the log context confidence.

[0126] Therefore, the dynamic causal graph is not an independently constructed graph, but a causal augmentation graph based on the service dependency graph, combined with data and semantic context for edge weight annotation and real-time adjustment. This graph preserves the static dependency paths between services while possessing the dynamic ability to depict the true causal impact under the current abnormal context. Next, knowledge correction is performed, dynamically generating fusion coefficients by utilizing the inference capabilities of the large model DeepSeek R1, fusing data-driven weights with preset rules in the knowledge base, where:

[0127]

[0128] in, For the final causal weight, data-driven approach and knowledge integration are employed. These are dynamic fusion coefficients, calculated by DeepSeek R1 based on data confidence levels; The causal strength is calculated based on statistical methods; This assigns weights to predefined rules in the knowledge base. For example, when a correlation is detected between Kafka message backlog and order payment failure, the system automatically loads the expert rule "insufficient consumer threads" from the relevant specification, increasing the weight of the relevant edge by 35%. The log semantic information processed by the DeepSeek R1 module is precisely what was mentioned above. Part of it.

[0129] In other words, the target causal weight is a dynamic edge weight that is weighted and updated on the static service dependency graph based on the aforementioned causal strength score. Thus, the obtained causal dependency graph is a dynamic causal dependency graph.

[0130] The construction of this dynamic causal dependency graph is based on enhanced reasoning using a service dependency graph. Specifically, a static service dependency graph between microservices or components is pre-maintained. Its structure is collected by call chain analysis and topology tracing tools, containing call relationships between nodes but without quantifying causal strength.

[0131] The root cause analysis in the above-mentioned implementation methods can be performed by... Figure 3 The root cause analysis engine is implemented in [the system / platform], and it runs on a dedicated server, for example, a dedicated server equipped with an NVIDIA A100 80GB GPU. Below, [the system / platform] will be combined with [other components / processes]. Figure 3 A brief explanation of the root cause analysis process:

[0132] First, the root cause analysis engine constructs a dynamic cause-effect graph: this engine takes multimodal data collected by the service monitoring system as input, and simultaneously uses a pre-built service / component dependency graph (i.e., Figure 3 The "knowledge graph" in the model serves as a prior constraint (limiting the node range and initial associations), generating an initial dynamic causal graph based on the causal graph modeling algorithm. Secondly, based on the initial causal graph, the knowledge correction module quantifies the causal association confidence of the edges in the graph using the root cause strength scoring model, completing the initial correction of the edge weights. Thirdly, the DeepSeek R1 model, based on the semantic knowledge scoring mechanism, combines Fusion context information and its own output semantic embedding vector (i.e., the aforementioned fusion feature vector) to perform fusion correction on the corrected causal graph, finally obtaining a dynamic causal graph with edge weights (i.e., the runtime service dependency graph or the aforementioned causal dependency graph).

[0133] In short, after triggering root cause analysis based on anomaly scoring, the first step is to perform causal modeling on each monitoring indicator within the anomaly interval, and then construct a weighted dynamic causal graph G=(V, E) by combining it with the original service dependency graph. The edge weights in the graph reflect the causal strength between variables. Based on this, and combined with F... fusion Using contextual information, semantic embedding vectors output by DeepSeek R1, and known dependencies in the knowledge graph, causal path scoring (i.e., the aforementioned influence score) is performed through graph reasoning algorithms such as PageRank, and finally, candidate root cause nodes and explanatory chains are output.

[0134] The combination of these implementation methods can control cross-node time errors within ±20ms. Spatiotemporal topology mapping is implemented by a graph neural network component extending the R1 model. This component dynamically constructs a service dependency graph by analyzing the number of TCP connections between containers, RPC call relationships, and resource contention. When constructing the service dependency graph using the graph neural network, each microservice is first treated as a node in the graph. Node features include service CPU / memory utilization, QPS, and response time. Edges represent TCP connections or RPC calls between services, and edge features can include call frequency, latency, and error rate. By collecting system monitoring data and call logs within a certain time window, a feature-rich graph structure is constructed as input to the GNN. Real service dependencies (such as call frequencies exceeding a threshold) are used as supervision labels to train the model to learn service dependency patterns. RPC call relationships are used both to determine the existence and strength of edges and as a source of edge features or labels, helping the model accurately identify and predict service dependencies, thereby dynamically generating the service dependency graph. For example, when a sudden increase in API response latency is detected in a microservice Pod, the system will generate a sub-topology graph containing upstream and downstream service nodes within 300ms and mark the critical path weights.

[0135] It should be noted that the embodiments of this application can also visualize the analysis results. In order to support the visualization of the analysis results, the embodiments of this application deploy a model training sandbox in an independent VPC network, connect to the production environment data bus through a dedicated line, and include an incremental learning mechanism that includes memory playback and elastic training.

[0136] The memory playback process employs the K-Center greedy algorithm to select representative samples from historical data, ensuring that at least 50 samples are retained for each type of fault. The storage format is as follows: Where M is the set of key samples in the memory bank; KL(p,q) is the Kullback-Leibler divergence, which measures the difference between the outputs of the old and new models; γ is the sample selection threshold, with a default value of γ=0.3. This is the predicted output under the current model parameters θ; For historical model parameters θ - The predicted output is as follows.

[0137] Among them, the elastic training is designed with a two-stage optimization objective. The first stage freezes the backbone network and only fine-tunes the classifier head:

[0138]

[0139] in, These are the optimized parameters for the model, used to minimize the objective function. The new dataset contains the input sample x and its corresponding label y; The cross-entropy loss function is used; in the second stage, the model is fine-tuned overall with a learning rate of 0.01 to prevent catastrophic forgetting. After training, the model update effect is verified through A / B testing. The new model is only pushed to the production environment when the F1-score improves by more than 2% and the false positive rate decreases by more than 5%.

[0140] The above approach, based on the comparative feature learning of your dual-tower network (JS divergence loss function), combined with the K-Centere algorithm to select key samples for incremental training, reduces computational resource consumption by 70% while maintaining a localization accuracy of over 92%. This solves the problem that current binary classification selection of the Squeeze / Cave algorithm does not quantify the computational resource consumption (such as GPU memory usage) under different scenarios, making it difficult to balance localization accuracy and response speed.

[0141] It should be noted here that, to achieve continuous learning and optimization, the operations and maintenance console is built on the React framework and uses a WebGL-accelerated rendering engine to visualize large-scale graph data. Specifically, in terms of dynamic layout, a heterogeneous force-guided algorithm for anomaly scoring is introduced, and the force calculation between nodes is as follows:

[0142]

[0143] Among them, F ij The resultant force between sections i and j; the repulsive force coefficient k r =150; elastic coefficient k s =0.3; initial spacing l0=120 pixels; p i p j Let i and j be the coordinates of nodes i and j on the canvas. The score indicates the degree to which the node is evaluated as "abnormal" by the model, with a higher risk indicating a closer score to 1. Anomaly scores primarily originate from statistical anomaly scores based on historical indicators, unsupervised anomaly detection models, prediction error-driven scores, and cumulative importance scores in the Shapley Additive Explanations (SHAP) model. For example, when the autoencoder detects anomalies in the reconstruction error of the Kafka Broker, or when the SHAP model evaluates "connection pool configuration" as a high-risk feature, the system can assign anomaly scores of 0.81 and 0.62 to the corresponding node, respectively, for use in subsequent graph layout and root cause inference.

[0144] In terms of interactive diagnostics, it supports clicking on nodes to view SHAP interpretation results. Feature importance calculation uses:

[0145]

[0146] in, is the SHAP value of the i-th feature, reflecting its contribution to the prediction result; S is a feature subset that does not include the i-th feature; This means removing element i from set N, which is the set of all natural numbers, excluding 1. M represents the total number of features; f(S) is the model prediction output when using a subset of features S. For example, in a database failure, the visualization interface shows that connection pool size (SHAP=0.62), query complexity (SHAP=0.28), and network latency (SHAP=0.15) are the key influencing factors.

[0147] In terms of decision support, an integrated contingency plan recommendation engine is used. When Kafka consumer lag is detected, a decision tree automatically pops up with operation suggestions such as "expand consumer group", "adjust fetch.max.bytes parameter", and "optimize deserialization logic". Each option is accompanied by an R1-estimated success probability (e.g., 89%, 76%, 63%). Dynamic contingency plan generation uses a reinforcement learning framework, and the reward function is designed as follows:

[0148]

[0149] in, ΔSLA represents the degree of improvement in the Service Level Agreement, typically expressed as a percentage performance improvement or an increase in the achievement rate; Cost Reduction represents the amount of cost reduction, reflecting a t The direct economic benefits of the action; RollbackRisk is the rollback risk, which quantifies the probability or potential loss of an action failure requiring a rollback operation; weighting coefficients w1=0.7, w2=0.2, w3=0.1, to ensure that the decision takes into account both effectiveness and safety.

[0150] Furthermore, as an optional implementation, the method also includes:

[0151] Based on at least one of the following: cluster operation and maintenance logs and monitoring data, fault cases, operation and maintenance manuals and emergency manuals, official documents and maintenance manuals of each software product, application and device example data, topology relationship data, operation and maintenance documents, and work order records, construct operation and maintenance knowledge base data;

[0152] The operation and maintenance knowledge base data is injected into the locally deployed multimodal anomaly detection network; wherein, the multimodal anomaly detection network includes a Temp-Lora module;

[0153] The multimodal anomaly discrimination network is iteratively optimized.

[0154] As mentioned above, the large model in this application embodiment, such as the aforementioned multimodal anomaly discrimination network, adopts the open-source DeepSeek model. Based on this, in the optional implementation methods described above, the application system (i.e., the aforementioned cluster), operation and maintenance logs and monitoring data, fault cases, operation and maintenance manuals and emergency manuals, official documents and maintenance manuals of various software products (such as system maintenance manuals), application and device instance configuration management database (CMDB) data, and topology relationship data are integrated to form a proprietary operation and maintenance knowledge base. Training is based on telecommunications operator industry data, and through fine-tuning, a knowledge graph based on the operator's business, products, R&D design, processes, ecosystem, and other vertical fields is constructed, providing accurate and targeted information. Private industry knowledge content is combined with the DeepSeek model to provide personalized, real-time knowledge services for IT support (i.e., in addition to providing content supported by the DeepSeek large model, a private domain database is also provided). This application uses an open-source large model algorithm. Specifically, as follows... Figure 4 As shown, it includes:

[0155] Step 1: Data Preparation and Knowledge Base Construction

[0156] Integrate application system operation and maintenance logs and monitoring data, fault cases, operation and maintenance manuals and emergency manuals, official documents and maintenance manuals of various software products (such as system maintenance manuals), application and device instance CMDB data, topology relationship data, operation and maintenance documents, work order records, fault cases and other data. It is recommended to use Markdown or structured table format.

[0157] Clean the data, remove noise (such as log redundancy), and label key entities (such as server IP, error codes, business codes, etc.).

[0158] Knowledge is vectorized by using DeepSeek-R1's Embedding interface to convert text into vectors, employing dynamic chunking strategies (such as segmentation by paragraph or semantics). The vectors are then stored in a vector database, and indexing parameters (such as HNSW levels) are optimized to improve recall.

[0159] Step Two: Model Deployment and Optimization

[0160] The large model is deployed locally. To ensure domain compatibility, operation and maintenance knowledge base data is injected, and the RAG dynamic retrieval and prompting (Prompts) project is used, such as adding the system command "You are a senior operation and maintenance expert" to improve the professionalism of the answers. To improve performance optimization, INT4 quantization is used to reduce inference latency.

[0161] Step 3: System Integration and Function Development

[0162] Configure dialogue chains using FlowiseAI, integrating model services, knowledge retrieval, and context management modules. Enables multi-turn dialogue memory and tracing, supporting answer-related knowledge fragment references. Alarm linkage: Connects to alarm information, automatically parses alarm messages, and triggers knowledge retrieval. Proactive diagnosis: Based on dynamic thinking chain technology, guides the model to autonomously break down problems, such as "high CPU load → check processes → analyze logs".

[0163] Introducing technical manuals or reference documents into the context is a common method to improve question-answering capabilities and is widely used in root cause localization. However, manuals or documents are usually lengthy, and the model's generation speed is proportional to the square of the context length. Therefore, multi-turn question answering based on document knowledge is slow and often affects the efficiency of root cause localization. To address the problem of slow model question answering in long document scenarios, this application's embodiments introduce Temp-Lora, an efficient alternative method for generating long texts, into the DeepSeek R1 model generation process. Figure 5 As shown:

[0164] The Temp-Lora method divides the text into multiple chunks and trains a LoRa model to remember previous inputs after each generation of a fixed-length chunk. Subsequently, when generating the next chunk, the chunk generated in the previous round is used as the new input, while earlier input information is stored in the parameters of the LoRa model.

[0165] Step 4: Verification and Iteration

[0166] A test set covering high-frequency scenarios (such as slow SQL optimization and disaster recovery failover) is constructed, and accuracy is quantified through a combination of manual scoring and automated metrics (BLEU, ROUGE). For bad cases, optimizations are implemented by adjusting the chunking strategy, expanding the knowledge base, or adding a rejection mechanism. Automated metrics include, for example, Bilingual Evaluation Understudy (BLER) and Recall-Oriented Understudy for Gisting Evaluation (ROUGE).

[0167] To ensure continuous iteration, a feedback loop is established, automatically marking incorrect answers based on user ratings and periodically fine-tuning the model. The knowledge base is dynamically updated, with scheduled tasks synchronizing the latest maintenance documentation and triggering incremental updates to the vector library.

[0168] Furthermore, as an optional implementation, the method also includes:

[0169] The data acquisition agent uses the Network Time Protocol (NTP) to periodically synchronize its time with multiple reference time sources within the cluster.

[0170] As an example, one possible implementation is that the data acquisition agent has a built-in clock synchronization module. This module uses dynamically weighted NTP and actively synchronizes its time with multiple (e.g., three) reference time sources within the cluster every 5 seconds. Time deviation compensation is optimized using a calibration coefficient α, for example, α can range from 0.2 to 0.8. The reference time source is an NTP server, and the specific time synchronization formula is as follows:

[0171]

[0172] in, The calibrated timestamp, in milliseconds; The original, uncalibrated time is the local time. Let be the dynamic weight of the i-th NTP server; The base time returned by the i-th NTP server; The current value of the local system clock; N is the number of NTP servers participating in the calibration.

[0173] Among the above-mentioned optional implementation methods, dynamic weighted NTP is used to achieve moderate synchronization at cross-boundary points, controlling the time error within ±20ms, thereby ensuring the spatiotemporal consistency of multi-source data; it solves the problem that the time drift problem of distributed systems is not fully considered at present, and the cross-node time error may exceed 100ms, resulting in the distortion of the timing alignment of abnormal events. For example, the concurrent failure of multiple Pods is incorrectly associated due to timestamp deviation.

[0174] In the above-described root cause localization method of this application embodiment, firstly, by deploying intelligent agents to collect container log text, network traffic characteristics, and performance indicators in real time, a cross-modal attention mechanism is designed, combined with a dynamic weighted NTP clock synchronization protocol (time error ±20ms), to achieve spatiotemporal alignment and joint analysis of multi-source data. This solves the limitation of existing technologies relying on single time-series data and significantly improves the contextual relevance of root cause localization. Secondly, based on graph neural networks (GNN), a service topology is dynamically constructed (generation latency <300ms), an improved PageRank algorithm is introduced, and a Granger causal matrix is ​​combined to quantify the dynamic causal relationships between nodes. This solves the problem of root cause localization failure in elastic scaling scenarios caused by existing technologies relying on static service dependency graphs, and improves the reliability of root cause localization under microservice architecture. Thirdly, a dynamic weight adjustment mechanism is designed, integrating knowledge base rules (such as database connection pool exhaustion failure modes), and a SHAP value visualization module is used to explain feature contribution, overcoming the poor interpretability problem of pure data-driven methods, and integrating historical experience and real-time data to improve the credibility of root cause localization. Fourthly, the K-Center algorithm is used to screen key samples (KL divergence threshold). =0.3 control), combined with dual-tower network comparative feature learning, and FP16 quantization to compress the model size to 4.7GB. This achieves rapid adaptation to sudden anomaly modes under lightweight model conditions, reducing computational resource consumption by 70% compared to existing technologies. Fifthly, a dynamically weighted NTP protocol (calibration coefficient) is adopted. By optimizing the compensation algorithm and combining it with the clock drift prediction model (LSTM network prediction error compensation amount), high-precision clock synchronization across nodes is achieved, reducing the time error of the distributed system from >100ms in the existing technology to ±20ms, and avoiding time alignment distortion of multi-source data.

[0175] The method described in this application embodiment can be implemented by a fault root cause localization device / system, which may include: a data acquisition agent deployed in a distributed environment for collecting multimodal data; a feature extraction server cluster for extracting feature data from the multimodal data and performing feature fusion; an anomaly detection network, including a large-scale DeepSeek R1 real-time inference instance and a causal inference module in the k8s service mesh, wherein the large-scale DeepSeek R1 real-time inference instance performs comparative feature learning, and the causal inference module is used to determine causal strength; and a causal analysis engine that integrates a domain knowledge graph and real-time monitoring data for constructing a dynamic causal graph and locating root causes.

[0176] Embodiments of this application also provide a fault root cause location device, such as... Figure 6 As shown, the device includes:

[0177] The acquisition module 601 is used to acquire multimodal data of the cluster collected by the data acquisition agent, wherein the multimodal data includes structured time-series monitoring data and unstructured operation logs;

[0178] The feature fusion module 602 is used to perform multimodal feature fusion on the time series monitoring data and the operation log to obtain a fused feature vector characterizing the current state of the cluster.

[0179] Anomaly detection module 603 is used to perform anomaly detection on the cluster based on the fused feature vector and obtain detection results, the detection results including anomaly score and anomaly category;

[0180] The analysis module 604 is used to perform root cause analysis of the fault based on the anomaly category, time series monitoring data related to the anomaly category, and the fused feature vector when the anomaly score is greater than a first threshold, and to obtain the analysis results.

[0181] Specifically, the anomaly detection module 603 is used to: input the fused feature vector into a multimodal anomaly discrimination network to obtain the anomaly score and anomaly category of the current state of the cluster by the multimodal anomaly discrimination network.

[0182] Specifically, when the anomaly detection module 603 is used to: input the fused feature vector into the multimodal anomaly discrimination network to obtain the anomaly score and anomaly category of the current state of the cluster by the multimodal anomaly discrimination network, it is used to:

[0183] The feature vector corresponding to the real-time data in the fused feature vector is input into the adjustable encoder of the multimodal anomaly discrimination network to obtain the first feature vector output by the adjustable encoder;

[0184] The feature vector corresponding to the positive label data in the fused feature vector is input into the freeze encoder of the multimodal anomaly detection network to obtain the second feature vector output by the freeze encoder; the positive label data is pre-labeled data;

[0185] Using a loss function, calculate the loss values ​​corresponding to the first feature vector and the second feature vector;

[0186] If the loss value is greater than the second threshold, the feature vector corresponding to the real-time data is input into the discrimination network to obtain the anomaly score and anomaly category.

[0187] The analysis module 604 includes:

[0188] The verification submodule is used to compare and verify the regression model constructed from the values ​​of any two time series indicators at L time points to obtain a causal strength score; L is the maximum lag order, and the two time series indicators belong to the time series monitoring data related to the anomaly category.

[0189] A submodule is constructed to build a causal dependency graph between different nodes based on the time series indicators and the causal strength score; the nodes are nodes in the cluster.

[0190] The ranking submodule is used to input the causal dependency graph and the fused feature vector into a pre-designed ranking algorithm to obtain the influence score of each node;

[0191] The acquisition submodule is used to acquire the analysis results based on the influence score and the causal dependency graph. The analysis results include candidate root cause nodes and explanatory chain information.

[0192] The inspection submodule includes:

[0193] The selection unit is used to select the maximum lag order L by using the Akaike Information Content Criterion (AIC) minimization principle, provided that the values ​​of any two time series indicators at L times satisfy a pre-set stationarity condition.

[0194] The first calculation unit is used to calculate the F-statistic of each regression model group based on a regression model group containing the target time series indicator and a regression model group not containing the target time series indicator for a target time series indicator; wherein, the target time series indicator is one of the two time series indicators.

[0195] The first acquisition unit is used to obtain the causal strength score based on the difference between the two F statistics.

[0196] The construction sub-module includes:

[0197] The second calculation unit is used to calculate the initial causal edge weights of the static service dependency graph based on the derivative relationship of the statistical time series indicators.

[0198] The correction unit is used to correct the initial causal edge weights based on the causal strength score.

[0199] The fusion unit is used to fuse the corrected initial causal edge weights with the predefined rule weights in the knowledge base to obtain the target causal edge weights.

[0200] The second acquisition unit is used to obtain the causal dependency graph based on the target causal edge weights.

[0201] The device further includes:

[0202] The module is used to build an operation and maintenance knowledge base based on at least one of the following: operation and maintenance logs and monitoring data of the cluster, fault cases, operation and maintenance manuals and emergency manuals, official documents and maintenance manuals of each software product, application and device example data, topology relationship data, operation and maintenance documents, and work order records.

[0203] An input module is used to inject the operation and maintenance knowledge base data into the locally deployed multimodal anomaly discrimination network; wherein, the multimodal anomaly discrimination network includes a temporary low-rank adaptation module;

[0204] An optimization module is used to iteratively optimize the multimodal anomaly discrimination network.

[0205] The device further includes:

[0206] The time synchronization module is used by the data acquisition agent to periodically synchronize with multiple reference time sources within the cluster using the Dynamic Weighted Network Time Protocol (NTP).

[0207] It should be noted that the fault root cause localization device provided in this application embodiment can implement all the method steps implemented in the above fault root cause localization method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0208] Another embodiment of the electronic device in this application, such as Figure 7 As shown, it includes a transceiver 710, a processor 700, a memory 720, and a program or instructions stored in the memory 720 and executable on the processor 700; when the processor 700 executes the program or instructions, it implements the above-mentioned method for reviewing the consistency of the interaction protocol.

[0209] The transceiver 710 is used to receive and send data under the control of the processor 700.

[0210] Among them, Figure 7 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 700) and memory (memory 720). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 710 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 700 is responsible for managing the bus architecture and general processing, and the memory 720 can store data used by the processor 700 during operation.

[0211] An embodiment of this application provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the fault root cause localization method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0212] The processor mentioned above is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0213] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0214] Through the above description of the implementation methods, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, disk, optical disk) and includes several instructions for executing the methods described in the various embodiments of this application.

[0215] Therefore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed by a processor, implement the fault root cause localization method as described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0216] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0217] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0218] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0219] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms "a," "an," and "the" are intended to include all such forms. It will be further understood that the terms "comprising" and / or "including," when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0220] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / port> < / ip>

Claims

1. A method for locating the root cause of a fault, characterized in that, The method includes: Acquire multimodal data of the cluster collected by the data acquisition agent, wherein the multimodal data includes structured time-series monitoring data and unstructured operation logs; Multimodal feature fusion is performed on the time series monitoring data and the operation log to obtain a fused feature vector representing the current state of the cluster; Based on the fused feature vector, anomaly detection is performed on the cluster to obtain detection results, which include anomaly score and anomaly category; If the anomaly score is greater than the first threshold, a root cause analysis is performed based on the anomaly category, the time-series monitoring data related to the anomaly category, and the fused feature vector to obtain the analysis results.

2. The method according to claim 1, characterized in that, Based on the fused feature vector, anomaly detection is performed on the cluster to obtain detection results, including: The fused feature vector is input into a multimodal anomaly discrimination network to obtain the anomaly score and anomaly category of the current state of the cluster by the multimodal anomaly discrimination network.

3. The method according to claim 2, characterized in that, The fused feature vector is input into a multimodal anomaly detection network to obtain the anomaly score and anomaly category of the current state of the cluster by the multimodal anomaly detection network, including: The feature vector corresponding to the real-time data in the fused feature vector is input into the adjustable encoder of the multimodal anomaly discrimination network to obtain the first feature vector output by the adjustable encoder; The feature vector corresponding to the positive label data in the fused feature vector is input into the freeze encoder of the multimodal anomaly detection network to obtain the second feature vector output by the freeze encoder; the positive label data is pre-labeled data; Using a loss function, calculate the loss values ​​corresponding to the first feature vector and the second feature vector; If the loss value is greater than the second threshold, the feature vector corresponding to the real-time data is input into the discrimination network to obtain the anomaly score and anomaly category.

4. The method according to claim 1 or 3, characterized in that, Based on the anomaly category, time-series monitoring data related to the anomaly category, and the fused feature vector, root cause analysis is performed to obtain analysis results, including: A causal strength score is obtained by comparing and testing the regression models constructed from the values ​​of any two time series indicators at L time points; L is the maximum lag order, and the two time series indicators belong to the time series monitoring data related to the anomaly category. Based on the time series indicators and the causal strength score, a causal dependency graph between different nodes is constructed; the nodes are the nodes in the cluster. The causal dependency graph and the fused feature vector are input into a pre-designed ranking algorithm to obtain the influence score of each node; The analysis results are obtained based on the influence score and the causal dependency graph. The analysis results include candidate root cause nodes and explanatory chain information.

5. The method according to claim 4, characterized in that, A comparative test is performed on the regression model constructed from the values ​​of any two time series indicators in the time series monitoring data at L time points to obtain a causal strength score, including: If the values ​​of any two time series indicators at L time points satisfy the pre-set stationarity condition, the maximum lag order L is selected using the Akaike Information Content Criterion (AIC) minimization principle. For a target time series indicator, the F-statistic of each regression model group is calculated based on the regression model group containing the target time series indicator and the regression model group not containing the target time series indicator; wherein, the target time series indicator is one of the two time series indicators. The causal strength score is obtained based on the difference between the two F-statistics.

6. The method according to claim 4, characterized in that, Based on the time series indicators and the causal strength score, a causal dependency graph among different indicators is constructed, including: Based on the derivative relationships of the statistical time series indicators, the initial causal edge weights of the static service dependency graph are calculated. The initial causal edge weights are adjusted based on the causal strength score. The corrected initial causal edge weights are fused with the predefined rule weights in the knowledge base to obtain the target causal edge weights; The causal dependency graph is obtained based on the target causal edge weights.

7. The method according to claim 3, characterized in that, The method further includes: Based on at least one of the following: cluster operation and maintenance logs and monitoring data, fault cases, operation and maintenance manuals and emergency manuals, official documents and maintenance manuals of each software product, application and device example data, topology relationship data, operation and maintenance documents, and work order records, construct operation and maintenance knowledge base data; The operation and maintenance knowledge base data is injected into the locally deployed multimodal anomaly detection network; wherein, the multimodal anomaly detection network includes a temporary low-rank adaptation module; The multimodal anomaly discrimination network is iteratively optimized.

8. The method according to claim 1, characterized in that, The method further includes: The data acquisition agent uses the Dynamic Weighted Network Time Protocol (NTP) to periodically synchronize its time with multiple reference time sources within the cluster.

9. A fault root cause location device, characterized in that, The device includes: The acquisition module is used to acquire multimodal data of the cluster collected by the data acquisition agent. The multimodal data includes structured time-series monitoring data and unstructured operation logs. The feature fusion module is used to perform multimodal feature fusion on the time series monitoring data and the operation log to obtain a fused feature vector characterizing the current state of the cluster. An anomaly detection module is used to perform anomaly detection on the cluster based on the fused feature vector and obtain detection results, the detection results including anomaly score and anomaly category; The analysis module is used to perform root cause analysis of the fault based on the anomaly category, time series monitoring data related to the anomaly category, and the fused feature vector when the anomaly score is greater than a first threshold, and to obtain the analysis results.

10. A fault root cause localization device, comprising a transceiver, a processor, a memory, and a program stored in the memory and executable on the processor; characterized in that, The transceiver transmits and receives data under the control of the processor, and when the processor executes the program, it implements the fault root cause localization method as described in any one of claims 1 to 8.

11. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the fault root cause localization method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the fault root cause localization method as described in any one of claims 1 to 8.

Images