Fault root cause determination method and device, electronic equipment, storage medium and product

Abstract

The invention provides a fault root cause determination method and device, electronic equipment, a storage medium and a product, and belongs to the technical field of cloud computing. The method comprises the steps that after a target service requested by a target service request fails, tracking data, log data and index data corresponding to each node are acquired based on node information of at least one node on a service path corresponding to the target service request, and the nodes provide a service interface called in the process for the target service; performing feature extraction on the tracking data, the log data and the index data corresponding to each node to obtain a tracking feature vector, a log feature vector and an index feature vector corresponding to each node; fusing the tracking feature vector, the log feature vector and the index feature vector corresponding to at least one node to obtain a total fusion feature vector; and determining a fault root cause of the target service based on the total fusion feature vector. The fault root cause determination efficiency can be improved, and the determination cost can be reduced.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a method, apparatus, electronic device, storage medium and product for determining the root cause of a fault. Background Technology

[0002] Microservice architecture breaks down large, complex systems into multiple independently developable, deployable, and manageable microservices. These microservices collaborate to achieve the overall system's functionality. However, as the number of microservices increases, maintaining system reliability becomes challenging. A failure in one microservice can cause the entire system to crash. To maintain system availability and stability, it is necessary to identify the root cause of the failure and implement effective fault recovery strategies accordingly.

[0003] When determining the root cause of a fault, the relevant technologies mainly adopt the following methods: collecting multiple modal data such as log data, tracking data, and indicator data; based on the threshold corresponding to each modal data, identifying abnormal data in each modal data, and providing the abnormal data of each modality to the operation and maintenance personnel, who then determine the root cause of the fault through further manual analysis or with the help of predefined rules.

[0004] However, the above methods require manual intervention from maintenance personnel to determine the root cause of the fault, which is inefficient and costly. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, storage medium, and product for determining the root cause of a fault. This method requires no manual intervention from maintenance personnel and can directly output the root cause of the fault, improving determination efficiency and reducing determination costs. The technical solution is as follows:

[0006] Firstly, a method for determining the root cause of a fault is provided, the method comprising:

[0007] After the target service requested by the target service request fails, based on the node information of at least one node on the service path corresponding to the target service request, the tracking data, log data and indicator data corresponding to each node are obtained, and the node is the service interface called during the process of the target service.

[0008] Feature extraction is performed on the tracking data, log data, and indicator data corresponding to each node to obtain the tracking feature vector, log feature vector, and indicator feature vector corresponding to each node.

[0009] The tracking feature vector, log feature vector, and indicator feature vector corresponding to at least one node are fused to obtain the total fused feature vector.

[0010] Based on the total fused feature vector, the root cause of the failure of the target service is determined.

[0011] Secondly, a device for determining the root cause of a fault is provided, the device comprising:

[0012] The acquisition module is used to acquire, based on the node information of at least one node on the service path corresponding to the target service request, the tracking data, log data and indicator data of each node after the target service requested by the target service request fails. The node is the service interface called during the process of the target service.

[0013] The extraction module is used to extract features from the tracking data, log data and indicator data corresponding to each node, and obtain the tracking feature vector, log feature vector and indicator feature vector corresponding to each node.

[0014] The fusion module is used to fuse the tracking feature vector, log feature vector and indicator feature vector corresponding to at least one node to obtain the total fused feature vector;

[0015] The determination module is used to determine the root cause of the failure of the target service based on the total fused feature vector.

[0016] Thirdly, an electronic device is provided, including a processor and a memory; the memory stores at least one piece of program code; the at least one piece of program code is called and executed by the processor to implement the method for determining the root cause of a fault as described in the first aspect.

[0017] Fourthly, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, and when executed by a processor, the at least one computer program is capable of implementing the method for determining the root cause of a fault as described in the first aspect.

[0018] Fifthly, a computer program product is provided, the computer program product comprising a computer program, which, when executed by a processor, is capable of implementing the method for determining the root cause of a fault as described in the first aspect.

[0019] The beneficial effects of the technical solutions provided in this application are:

[0020] In this embodiment, after a failure occurs in the target service requested by the target service request, in order to promptly determine the root cause of the failure, a total fused feature vector corresponding to the target service request is obtained through three stages: data alignment, feature extraction, and data fusion. Then, based on the total fused feature vector, the root cause of the target service failure is determined. In the data alignment stage, based on the node information of at least one node on the service path corresponding to the target service request, the tracking data, log data, and indicator data are aligned on a node-by-node basis. This node represents the service interface called during the target service provision process. Therefore, node-by-node alignment ensures that multiple modalities of data correspond to the same service interface, thus reflecting the system state at the same point in time or within the same event context. In the feature extraction stage, features are extracted from the tracking data, log data, and indicator data corresponding to the same node, obtaining the tracking feature vector, log feature vector, and indicator feature vector corresponding to the same node. By extracting features from the tracking data, log data, and indicator data that call the same service interface, data from different modalities can be mapped to the same feature space, preserving the integrity of the inherent features of different modalities and ensuring the accuracy of the root cause determination. During the data fusion phase, feature vectors from multiple modalities corresponding to each node are fused to obtain a total fused feature vector. Based on this total fused feature vector, the root cause of the target service failure can be directly determined without manual intervention, improving the efficiency of root cause determination and reducing the cost. Furthermore, since this total fused feature vector can characterize the system state during the provision of the target service requested by the target service request, the determined root cause is more accurate. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a method for determining the root cause of a fault, provided in an embodiment of this application.

[0023] Figure 2 This is a flowchart of another process for determining the root cause of a fault, provided in an embodiment of this application.

[0024] Figure 3 This is a schematic diagram of the structure of a fault root cause determination device provided in an embodiment of this application;

[0025] Figure 4 This is a structural block diagram of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0027] It is understood that the terms "each," "multiple," and "any" used in the embodiments of this application, etc., mean that "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the corresponding multiples. For example, multiple words include 10 words, and "each word" refers to each of the 10 words, while "any word" refers to any one of the 10 words.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0029] Before implementing the embodiments of this application, the terms involved in the embodiments of this application will be explained.

[0030] Root cause diagnosis refers to the process of finding the root cause of a system or equipment failure. It aims to identify the root cause of the problem so that appropriate solutions can be found and maintenance and preventative measures can be optimized.

[0031] Cross-Attention: In deep learning, especially in Transformer networks, cross-attention refers to a mechanism that calculates attention between different sequences or data. Cross-attention allows the model to focus on information in different input sequences, thereby effectively capturing contextual relationships.

[0032] Gaussian distribution: An important probability distribution, often called the normal distribution. Its value is maximum near the mean and gradually decreases towards both sides, forming a bell-shaped curve. The Gaussian distribution is widely used in statistics and natural sciences, and many random phenomena are believed to conform to this distribution.

[0033] Multimodal data fusion refers to the process of integrating data from different sources (such as images, text, and sound) to achieve a more comprehensive understanding or enhance model performance. This fusion helps leverage the strengths of each modality to improve the expressive power of information.

[0034] SREs (Site Reliability Engineers) are a role combining software engineering and operations expertise, responsible for ensuring the availability, reliability, and performance of online services. SREs optimize system operation by automating critical tasks such as system monitoring, capacity planning, and fault response.

[0035] Trace: In distributed systems, tracing refers to capturing the flow of a series of operations or requests between multiple services, which helps to identify performance bottlenecks and monitor system health.

[0036] Span: In a tracking system, a span is a representation of a specific operation, including information such as its start and end times. It is usually a measure of the lifecycle of a request or operation.

[0037] Metrics are quantitative indicators used to measure and evaluate system performance, health status, or other characteristics. Examples include response time and request volume.

[0038] GAT (Graph Attention Network) uses an attention mechanism to assign weights to nodes in a graph, thereby enhancing the connectivity between nodes. GAT can effectively process graph data and capture more complex relationships.

[0039] GRU (Gated Recurrent Unit): Helps solve the gradient vanishing problem faced by traditional RNNs (Recurrent Neural Networks) in long-sequence training. GRU uses a gating mechanism to control the forgetting and updating of information, making the model perform better in long-short-term dependencies.

[0040] LSTM (Long Short-Term Memory): Another type of RNN with long-term memory capabilities. By introducing memory units and gating mechanisms, it can better capture long-term dependency information in sequence data.

[0041] BERT (Bidirectional Encoder Representations from Transformers): A pre-trained language representation model that learns the contextual relationships of text through a bidirectional Transformer architecture, significantly improving the performance of natural language processing tasks and is widely used in various NLP (Natural Language Processing) tasks.

[0042] Convolution: In signal processing and computer vision, convolution is a mathematical operation used to extract features from signals or images. Convolutional Neural Networks (CNNs) have achieved excellent performance in tasks such as image classification and object detection by utilizing convolution operations.

[0043] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0044] With the development of the internet, real-time data streams, and the diversification of connected devices, driven by demands for search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly emerged. Unlike previous parallel distributed computing, cloud computing is fundamentally revolutionizing the entire internet model and enterprise management model. With the development of cloud computing technology, microservice architecture has been widely adopted due to its loose coupling, flexibility, and scalability. However, as the number of microservices in a microservice architecture increases, maintaining the reliability of the microservice system becomes challenging. When a microservice fails, this failure can propagate throughout the system, causing the entire system to crash. For Site Reliability Engineers (SREs), automating and quickly identifying the root cause of a failure and developing effective recovery strategies directly impacts the availability and stability of the system.

[0045] To determine the root cause of a failure, relevant technologies typically utilize service brokers, APIs (Application Programming Interfaces), or built-in code. This involves acquiring data in one or two modalities, such as trace data, log data, or metric data, through application performance monitoring (APM), container monitoring, cloud infrastructure monitoring, log management, and user experience monitoring. Causal inference or deep learning techniques are then used to automatically extract the causal relationship between the failure and system behavior from the collected data. While this method can identify the root cause without much human intervention or reliance on expert experience, the same type of failure manifests differently in different modalities of data. Therefore, determining the root cause based on only one or two modalities of data is not accurate.

[0046] To improve the accuracy of root cause analysis, it is necessary to fuse features from multiple modalities, such as tracking data, log data, and indicator data. However, fusing features from multiple modalities presents two significant challenges: first, how to minimize information loss when extracting features from each modality; and second, how to effectively capture and utilize the relationships between different modalities.

[0047] Currently, some fusion methods lack consideration for the differences in data structure among the three modalities, using the same feature extraction method to extract data features from the three modalities, resulting in the loss of important information. Although some fusion methods consider the structural differences of different modalities and adopt different feature extraction methods for different modalities, they fail to fully consider the relationships between modalities during the feature extraction process, resulting in a lack of correlation among the features extracted from the three modalities. The fused features cannot reflect the state of the system, and the root causes of faults determined based on the fused features are not accurate enough.

[0048] To improve the accuracy and effectiveness of root cause analysis and ensure the stability and reliability of microservice systems, this application provides a root cause analysis method, which is improved in the following aspects:

[0049] Firstly, addressing the issue of independently processing multiple modalities of data in related technologies, this application aligns the tracking data, log data, and metric data corresponding to each node on the service path. This ensures effective correlation between data from different modalities, enabling analysis within the same context. This approach not only accurately matches the tracking data, log data, and metric data corresponding to service requests but also enhances the correlation between different modalities. This correlation not only improves the accuracy of data processing but also significantly reduces information loss caused by independent analysis. Related technologies often struggle to capture the complex relationships between different modalities. However, the solution in this application correlates data from different modalities and constructs a graph structure for each modality, intuitively representing the service path of the request and the corresponding modal data, maintaining the consistency and integrity of the data across all modalities.

[0050] Secondly, addressing the lack of end-to-end automation, this application's solution introduces different processing mechanisms in the feature extraction and data fusion stages. This reduces information loss during feature extraction, uncovers relationships between different modalities, and improves the accuracy of the fused features. Specifically, in the feature extraction stage, different models are used to extract corresponding features based on the data structure of different modalities. For example, graph attention networks are used to extract features from tracking data, long short-term memory networks to extract features from log data, and models with Gaussian attention layers to extract features from indicator data. By using different models to extract features from different modalities, information loss between different modalities is reduced. In the data fusion stage, a cross-attention mechanism is employed. By dynamically adjusting the information weights between different modalities, the complex relationships between modalities are fully explored, thereby achieving full automation of fault root cause determination. This reduces manual intervention by operations personnel, especially site reliability engineers, and improves the efficiency of fault determination.

[0051] Furthermore, during feature extraction of the indicator data, a Gaussian attention mechanism is employed to effectively identify indicator components relevant to the current service request, thereby assigning these components higher weights. Similarly, the cross-attention mechanism used in the data fusion stage ensures a smoother and more focused information integration process when handling data from different modalities. Through these two mechanisms, the deep relationships between modalities can be better utilized during the data analysis stage, avoiding information loss related to relevant technologies and improving the effectiveness of the analysis.

[0052] In summary, this application aligns multi-modal data based on nodes along the service path and dynamically extracts features from different modalities. Gaussian attention and cross-attention mechanisms are employed during feature extraction and fusion to effectively uncover complex relationships between different modalities, avoiding information loss and improving the efficiency and accuracy of root cause analysis. Furthermore, end-to-end automation reduces manual intervention and operational complexity, lowering the cost of root cause analysis. In short, this application's solution not only optimizes the fault diagnosis process but also proposes a novel approach to multi-modal data analysis, enabling rapid response and resolution of system faults, reducing operational costs, enhancing fault location accuracy, and improving user satisfaction.

[0053] This application provides a method for determining the root cause of a fault. Taking an electronic device executing this application embodiment as an example, the electronic device has strong computing power and can be a terminal, such as a laptop or desktop computer, or a server, such as a standalone physical server, a cluster of multiple physical servers, or a distributed system. See also Figure 1 The method flow provided in this application embodiment includes:

[0054] 101. After the target service requested by the target service request fails, based on the node information of at least one node on the service path corresponding to the target service request, obtain the tracking data, log data and indicator data corresponding to each node.

[0055] The target service can be a service provided by a microservice system, implemented collaboratively by multiple microservices. Each microservice can correspond to a service interface, and the corresponding function of the microservice can be implemented by calling the service interface. For ease of subsequent description, the service interfaces called during the provision of the target service in this embodiment can be referred to as nodes. To facilitate the differentiation of different nodes, different node identifiers can be set for different nodes, which are also called span IDs. Multiple nodes called during the provision of services in response to service requests can form a service path. To differentiate the service paths corresponding to different service requests, different service path identifiers can be set for different service paths, which can also be called tracking IDs.

[0056] To obtain a target service, a user can send a target service request to the microservice system. In response to this request, if the target service fails during the process of providing the target service to the user, in order to promptly determine the root cause of the failure and take effective measures to recover from it, this embodiment of the application needs to obtain the tracking data, log data, and indicator data corresponding to each node based on the node information of at least one node on the service path corresponding to the target service request. The node information includes the service path identifier, node identifier, and execution time range of the service path where the node is located. Specifically, when obtaining the tracking data, log data, and indicator data corresponding to each node based on the node information of at least one node on the service path corresponding to the target service request after the target service requested has failed, the following steps can be taken:

[0057] 1011. For any node, based on the node's service path identifier and node identifier, retrieve the tracking data containing the node's service path identifier and node identifier from the tracking data related to the target service request, and use it as the tracking data corresponding to the node.

[0058] In this embodiment, each node on the service path has a service path identifier and a node identifier. The trace data generated in response to a target service request calling different nodes includes the service path identifier and node identifier of the service path corresponding to the target service request. Therefore, for any node on the service path, trace data containing the service path identifier and node identifier of that node can be obtained from the trace data related to the target service request, based on the node's service path identifier and node identifier. This obtained trace data can then be used as the trace data corresponding to that node. For example, if the service path identifier of the service path corresponding to the target service request is 1, and the service path includes nodes A, B, C, and D, then the node identifier of node A is a, the node identifier of node B is b, the node identifier of node C is c, and the node identifier of node D is c. For node A, the trace data with service path identifier 1 and node identifier a can be obtained as the trace data corresponding to node A.

[0059] 1012. Based on the service path identifier and node identifier of the node, obtain the log data containing the service path identifier and node identifier of the node from the log data related to the target service request, and use it as the log data corresponding to the node.

[0060] In this embodiment, during the process of responding to a target service request by calling any service interface to provide the target service, whenever a log is generated, the service path identifier and the node identifier corresponding to the called interface can be injected into the generated log. For example, the service path identifier corresponding to the target service request is Trace ID 0ad0d4, the service path includes nodes S1, S2, S3, and S4, and the node identifier of node S2 is Span ID 53452c. During the process of responding to a target service request by calling node S2 to provide the target service, each log generated for the call to node S2 will be injected with Trace ID: 0ad0d4 and Span ID: 53452c.

[0061] Since each log generated for each service interface call is injected with a service path identifier and a node identifier during the process of providing the target service in response to the target service request, when determining the log data corresponding to any node, the log data containing the service path identifier and node identifier of the node can be obtained from the log data related to the target service request based on the service path identifier and node identifier of that node, and then the obtained log data can be used as the log data corresponding to that node.

[0062] 1013. Based on the execution time range of the node, obtain the indicator data whose execution time falls within the execution time range from the indicator data related to the target service request, and use it as the indicator data corresponding to the node.

[0063] In the application embodiments, during the process of responding to a target service request by calling any service interface to provide the target service, each service interface, i.e., service node, corresponds to an execution time range, which can be obtained from the host of the microservice system. Furthermore, each microservice in the microservice system corresponds to at least one host and at least one container. Each host corresponds to a host IP, and each container corresponds to a container IP. Based on the host IP and container IP corresponding to the microservice, during the process of calling the service interface corresponding to the microservice, various indicator data reflecting the actual execution status of the target service request can be obtained from the host and container corresponding to the microservice. Each indicator data has an indicator curve, with the horizontal axis representing time and the vertical axis representing the indicator value, used to represent the change of the indicator value over time. For any node, when determining the indicator data corresponding to that node, indicator data whose execution time falls within the execution time range of that node can be obtained from the indicator data related to the target service request, and the obtained indicator data can be used as the indicator data corresponding to that node.

[0064] Optionally, after aligning the tracking data, log data, and metric data for each node, the aligned data can be represented as a graph structure. For example, tracking data can be represented as at least one node, log data as a table, and metric data as a metric curve. By representing the aligned data as a graph structure, the service path of the request and its corresponding modal data can be clearly displayed.

[0065] This application employs a node-based alignment strategy to ensure that data from different modalities can reflect the system state at the same point in time or within the same event context. Since the service path corresponding to a node represents a user request path, this strategy more accurately reflects the specific behavior of the system compared to time-window-based alignment strategies.

[0066] 102. Extract features from the tracking data, log data, and indicator data corresponding to each node to obtain the tracking feature vector, log feature vector, and indicator feature vector corresponding to each node.

[0067] After determining the tracking data, log data, and metric data corresponding to each node, features of the tracking data, log data, and metric data corresponding to each node can be extracted separately to obtain the tracking feature vector, log feature vector, and metric feature vector corresponding to each node.

[0068] In the feature extraction stage, the goal of this application is to map aligned multimodal data with different structures to the same feature space to achieve similar representations while preserving the integrity of the intrinsic features of each modality. To achieve this goal, taking any node as an example, the embodiments of this application can use the following methods when extracting features from the tracking data, log data, and indicator data corresponding to that node:

[0069] 1021. The first processing method is used to extract features from the tracking data corresponding to the node to obtain the tracking feature vector corresponding to the node.

[0070] Specifically, the first processing method is used to extract features from the tracking data corresponding to the node to obtain the tracking feature vector corresponding to the node, including the following steps:

[0071] 10211. Merge the tracking data of the nodes from different dimensions to obtain the fused tracking data of the nodes.

[0072] Typically, tracking data has multiple dimensions. To better reflect the system's behavior, tracking data from different dimensions corresponding to a node can be fused to obtain the fused tracking data for that node. When fusing tracking data from different dimensions, the data can be input into a fully connected layer, and the fused tracking data is obtained through the processing of the fully connected layer.

[0073] 10212. Input the fused tracking data of each node into the first feature extraction model so that the first feature extraction model adjusts the weight values ​​of each neighbor node of the node, encodes the fused tracking data of the node and the tracking data of each neighbor node and their weight values, and outputs the tracking feature vector corresponding to the node.

[0074] The first feature extraction model can be a graph attention network, which can extract spatial information and effectively capture global topological relationships while preserving node features by dynamically adjusting the weights of neighboring nodes. When the fused tracking data of each node is input into the first feature extraction model, the model dynamically adjusts the weights of each neighboring node, calculates the product of the tracking data of each neighboring node and its weight, fuses the product with the fused tracking data of the node, and then encodes the fused result to obtain the tracking feature vector corresponding to the node, which is then output. This tracking feature vector incorporates the features of the tracking data of the node's neighboring nodes, comprehensively extracting the intrinsic features of the tracking data and avoiding information loss.

[0075] 1022. The second processing method is used to extract features from the log data corresponding to the node to obtain the log feature vector corresponding to the node.

[0076] Specifically, the second processing method is used to extract features from the log data corresponding to the node to obtain the log feature vector corresponding to the node, including the following steps:

[0077] 10221. Extract the semantic features of each log data corresponding to the node.

[0078] For a node with multiple log data entries, each log data entry can be input into the BERT model, and the semantic features of each log data entry can be extracted through the BERT model.

[0079] 10222. Input the semantic features of each log data corresponding to the node into the second feature extraction model, so that the second feature extraction model encodes the semantic features of each log data based on the context order of each log data and outputs the log feature vector corresponding to the node.

[0080] The second feature extraction model can be a Long Short-Term Memory (LSTM) network, etc. After extracting the semantic features of multiple log data corresponding to a node, the semantic features of each log data can be input into the second feature extraction model. The second feature extraction model encodes the semantic features of each log data based on the contextual order of the log data to capture the contextual relationships between the log data, thereby obtaining the log feature vector corresponding to that node, and then outputting this log feature vector. This log feature vector can reflect the contextual relationships between log data, avoiding information loss in the log data.

[0081] 1023 uses a third processing method to extract features from the indicator data corresponding to the node, and obtains the indicator feature vector corresponding to the node.

[0082] In this process, the features of the indicator data related to the target service request in the indicator feature vector are enhanced. Specifically, a third processing method is used to extract features from the indicator data corresponding to the node to obtain the indicator feature vector corresponding to the node, including the following steps:

[0083] 10231. Merge the various indicator data corresponding to the nodes to obtain the merged indicator data of the nodes.

[0084] For multiple indicator data corresponding to a node, these indicator data can be fused to obtain fused indicator data for the node. This fused indicator data accurately models the relationship between multi-dimensional time series indicators and can accurately reflect the state of the system. When fusing the various indicator data corresponding to a node, the various indicator data corresponding to the node can be input into a one-dimensional convolutional network, and the fused indicator data is output through the convolution operation of the one-dimensional convolutional network.

[0085] 10232. Based on the correlation between the fusion index data and the target service request, determine the Gaussian weight value corresponding to the fusion index data of the node.

[0086] This embodiment associates data from different modalities through step 101, facilitating the modeling of relationships between these data. Since the degree of association between data from different modalities varies—for example, a tracking event is related not only to its collected metrics during execution but also to the context before and after execution—extracting only the metrics related to the node's tracking data would result in a loss of contextual information if the metrics are too short, while extracting all metrics corresponding to the node's tracking data would introduce redundancy if the metrics are too long. To avoid data loss and conserve resources in the feature extraction process, this embodiment combines Gaussian distribution and attention mechanisms. By evaluating the relationship between different fused metric segments and the tracking data, it assigns larger Gaussian weights to fused metric segments more relevant to the target service request (e.g., those closer to the execution time) and smaller Gaussian weights to fused metric segments less relevant to the target service request (e.g., those before and after the execution time). The following formula can be used to determine the Gaussian weights of different fused metric segments:

[0087]

[0088] Where, m i It is the current sampling point, that is, the currently judged fusion index segment, G(m) i ) represents the Gaussian weight value of the current sampling point, k is the sampling point most relevant to the running trace, i.e., the fusion index segment closer to the execution time, and σ is the variance of the Gaussian distribution that controls the decay of its correlation.

[0089] By incorporating Gaussian weights, this application can control the degree of correlation to decrease towards both sides, thereby ensuring the integrity of contextual information while avoiding the introduction of redundancy.

[0090] 10233. Input the fusion index data of the nodes and their corresponding Gaussian weights into the third feature extraction model, so that the third feature extraction model encodes the fusion index data based on the Gaussian weight values ​​corresponding to the fusion index data and outputs the index feature vector corresponding to the nodes.

[0091] The third feature extraction model can be a model with a Gaussian attention layer. After obtaining the fusion index data of a node and its corresponding Gaussian weights, different fusion index segments of the fusion index data and their corresponding Gaussian weight values ​​can be input into the third feature extraction model. The third feature extraction model encodes different fusion index segments based on the Gaussian weight values ​​corresponding to each fusion index segment, obtains the index feature vector corresponding to the fusion index data, and then outputs the index feature vector corresponding to the node.

[0092] 103. Fuse the tracking feature vector, log feature vector and indicator feature vector corresponding to at least one node to obtain the total fused feature vector.

[0093] After obtaining the tracking feature vector, log feature vector, and indicator feature vector corresponding to at least one node, the multimodal feature vectors corresponding to each node can be fused to obtain a total fused feature vector reflecting the system state. Specifically, fusing the tracking feature vector, log feature vector, and indicator feature vector corresponding to at least one node to obtain the total fused feature vector includes the following steps:

[0094] 1031. For any node, a cross-attention mechanism is used to process the tracking feature vector, log feature vector, and indicator feature vector corresponding to the node to obtain multiple attention feature vectors corresponding to the node.

[0095] For any node, a cross-attention mechanism can be used to perform cross-attention between any two feature vectors from the node's tracking feature vector, log feature vector, and metric feature vector, resulting in six attention feature vectors. Specifically, for the tracking and log feature vectors, the tracking feature vector can be used as Q (query), and the log feature vector as K (key) and V (value) for a cross-attention calculation to obtain the first attention feature vector; then, the log feature vector can be used as Q (query), and the tracking feature vector as K (key) and V (value) for another cross-attention calculation to obtain the second attention feature vector. For the tracking and metric feature vectors, the tracking feature vector can be used as Q (query), and the metric feature vector as K (key) and V (value) for a cross-attention calculation to obtain the third attention feature vector; then, the metric feature vector can be used as Q (query), and the tracking feature vector as K (key) and V (value) for another cross-attention calculation to obtain the fourth attention feature vector. For log feature vectors and indicator feature vectors, the log feature vector can be used as Q (query), and the indicator feature vector can be used as K (key) and V (value) to perform a cross-attention calculation to obtain the fifth attention feature vector; then the log feature vector can be used as Q (query), and the tracking feature vector can be used as K (key) and V (value) to perform a cross-attention calculation to obtain the sixth attention feature vector.

[0096] Taking log data feature vectors and tracking feature vectors as examples, the cross-attention formula can be:

[0097]

[0098] Among them, L i For log feature vectors, T i To track feature vectors.

[0099] 1032. Concatenate the multiple attention feature vectors corresponding to a node to obtain the concatenated feature vector corresponding to the node.

[0100] For multiple attention feature vectors corresponding to a node, the concatenated feature vector corresponding to the node can be obtained by concatenating these attention feature vectors. When concatenating multiple attention feature vectors, they can be concatenated randomly or in a certain order; this application does not impose specific limitations on this.

[0101] 1033. Combine the concatenated feature vectors corresponding to at least one node to form a feature matrix.

[0102] By processing at least one node using the method described above, we can obtain the concatenated feature vector corresponding to at least one node. For each concatenated feature vector corresponding to at least one node, we can obtain a feature matrix by treating the concatenated feature vector of each node as a row or a column.

[0103] 1034. Input the feature matrix into the feature fusion model and output the total fused feature vector.

[0104] The feature fusion model can be a model with fully connected layers, trained from training samples. These training samples can be the sample fusion feature vectors and labeled root causes of failures. When the feature matrix is ​​input into the feature fusion model, the total fusion feature vector can be obtained through the processing of the feature fusion model.

[0105] 104. Based on the total fusion feature vector, determine the root cause of the target service failure.

[0106] This application embodiment can train a fault root cause determination model. This model represents a mapping relationship between a total fused feature vector and fault root causes. When the total fused feature vector is input into the fault root cause determination model, the fault root cause corresponding to the total fused feature vector can be determined based on the mapping relationship between the total fused feature vector and the fault root cause. Then, the fault root cause of the target service is output. Furthermore, effective recovery measures can be taken based on the determined fault root cause to improve service availability.

[0107] Figure 2 A flowchart of the method for determining the root cause of a fault provided in an embodiment of this application is shown. See also: Figure 2The root cause determination process includes three stages: data alignment, feature extraction, and data fusion and root cause determination. For the data alignment stage, for a service path including nodes S1, S2, S3, and S4, taking node S2 as an example, node S2's Trace ID is 0ad0d4 and Span ID is 53452c. From the trace data corresponding to the service request, trace data with Trace ID 0ad0d4 and Span ID 53452c is obtained and used as the trace data corresponding to node S2. From the log data corresponding to the service request, log data including Trace ID 0ad0d4 and Span ID 53452c is obtained and used as the log data corresponding to node S2. Based on the execution time range of node S2, indicator data whose execution time falls within the execution time range of node S2 is obtained from the indicator data corresponding to the service request and used as the indicator data corresponding to node S2. Through the data alignment stage, the trace data, log data, and indicator data of the nodes can be aligned, thereby enabling the modeling of the relationships between data from different modalities.

[0108] In the feature extraction stage, the tracking data corresponding to node S2 can be input into a fully connected layer to fuse tracking data from different dimensions, resulting in fused tracking data. Then, the fused feature data for each node is input into the GAT network. The GAT network dynamically adjusts the weights of each neighboring node of node S2, thereby encoding the fused tracking data for node S2 and the fused tracking data and weights of each neighboring node, resulting in the tracking feature vector for node S2. For the log data corresponding to node S2, each log entry can be input into BERT to obtain the semantic features of each log entry. Then, the semantic features of each log entry are input into an LSTM. The LSTM encodes the semantic features of each log entry based on its contextual order, resulting in the log feature vector for node S2. For the indicator data corresponding to node S2, various indicator data can be input into a one-dimensional convolutional layer to obtain fused indicator data. Then, based on the correlation between different fused indicator segments in the fused indicator data and service requests, the Gaussian weight values ​​of different fused indicator segments in the fused indicator data are determined. Subsequently, the different fused indicator segments in the fused indicator data and their Gaussian weight values ​​are input into a Gaussian attention layer to output the indicator feature vector corresponding to node S2.

[0109] In the data fusion and root cause determination phase, a cross-attention mechanism can be employed. This involves calculating cross-attention between the tracking feature vector, log feature vector, and metric feature vector corresponding to each node, resulting in six attention feature vectors for each node. These attention feature vectors are then concatenated to obtain a concatenated feature vector for each node. Next, these concatenated feature vectors are combined to form a feature matrix, which is then input into the fusion feature model to output a total fusion feature vector. Finally, the total fusion feature vector is input into the fault root cause determination model to output the fault root cause, such as CPU overload.

[0110] Compared with related technologies, the solution in this application has the following improvements:

[0111] The first is tracking-based alignment: This method ensures that data from different modalities are effectively correlated at the same point in time or within the same event context, overcoming the data isolation problem caused by time windows in related technologies.

[0112] The second improvement is the dynamic feature extraction mechanism: different feature extractors are used to address the structural characteristics of different modal data, which reduces information loss and improves the expressiveness of features.

[0113] The third improvement is the Gaussian attention mechanism: a Gaussian attention mechanism is introduced to dynamically evaluate the importance of metric data in a specific request scenario, thereby ensuring that metrics related to the current request receive higher weights and enhancing sensitivity to the request context.

[0114] The fourth improvement is the cross-attention mechanism: a cross-attention mechanism is adopted to dynamically adjust the information weights between different modalities during the fusion stage, fully explore the complex relationships between modalities, and improve the limitations of related technologies.

[0115] The fifth improvement is a fully automated end-to-end process: achieving comprehensive automated data processing, from data collection, alignment, feature extraction to final fault diagnosis, reducing the need for manual intervention and improving the efficiency and accuracy of fault identification.

[0116] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0117] Please refer to Figure 3 The diagram illustrates a structural schematic of a fault root cause determination device provided in an embodiment of this application. This device can be implemented through software, hardware, or a combination of both, and can be all or part of an electronic device. The device includes:

[0118] The acquisition module 301 is used to acquire, based on the node information of at least one node on the service path corresponding to the target service request, the tracking data, log data and indicator data of each node after the target service requested by the target service request fails, and the node provides the service interface called during the process of the target service.

[0119] The extraction module 302 is used to extract features from the tracking data, log data and indicator data corresponding to each node, and obtain the tracking feature vector, log feature vector and indicator feature vector corresponding to each node.

[0120] The fusion module 303 is used to fuse the tracking feature vector, log feature vector and indicator feature vector corresponding to at least one node to obtain a total fused feature vector;

[0121] The determination module 304 is used to determine the root cause of the failure of the target service based on the total fused feature vector.

[0122] In another embodiment of this application, the node information includes at least one of a service path identifier, a node identifier, and an execution time range. The acquisition module is configured to, for any node, acquire, based on the node's service path identifier and node identifier, from the target service request-related tracking data, and use this as the node's corresponding tracking data; acquire, based on the node's service path identifier and node identifier, from the target service request-related log data, and use this as the node's corresponding log data; and acquire, based on the node's execution time range, from the target service request-related indicator data, indicator data whose execution time falls within the execution time range, and use this as the node's corresponding indicator data.

[0123] In another embodiment of this application, the device further includes:

[0124] The injection module is used to inject the service path identifier and the node identifier corresponding to the called interface into the generated log whenever a log is generated in response to a request from the target service to call any service interface to provide the target service.

[0125] In another embodiment of this application, the extraction module is configured to, for any node, perform feature extraction on the tracking data corresponding to the node using a first processing method to obtain a tracking feature vector corresponding to the node, wherein the tracking feature vector incorporates features of the tracking data of the node's neighboring nodes; perform feature extraction on the log data corresponding to the node using a second processing method to obtain a log feature vector corresponding to the node, wherein the log feature vector can reflect the contextual relationship between log data; and perform feature extraction on the indicator data corresponding to the node using a third processing method to obtain an indicator feature vector corresponding to the node, wherein the features of the indicator data related to the target service request in the indicator feature vector are enhanced.

[0126] In another embodiment of this application, the extraction module is used to fuse tracking data of different dimensions of a node to obtain fused tracking data of the node; input the fused tracking data of each node into a first feature extraction model so that the first feature extraction model adjusts the weight values ​​of each neighboring node of the node, encodes the tracking data of the node and the tracking data of each neighboring node and their weight values, and outputs the tracking feature vector corresponding to the node.

[0127] In another embodiment of this application, the extraction module is used to extract the semantic features of each log data corresponding to the node; input the semantic features of each log data corresponding to the node into the second feature extraction model, so that the second feature extraction model encodes the semantic features of each log data based on the context order of each log data, and outputs the log feature vector corresponding to the node.

[0128] In another embodiment of this application, the extraction module is used to fuse various indicator data corresponding to a node to obtain fused indicator data of the node; determine the Gaussian weight value corresponding to the fused indicator data based on the correlation between the fused indicator data and the target service request; and input the fused indicator data of the node and its corresponding Gaussian weight value into a third feature extraction model, so that the third feature extraction model encodes the fused indicator data based on the Gaussian weight value corresponding to the fused indicator data and outputs the indicator feature vector corresponding to the node.

[0129] In another embodiment of this application, the fusion module is used to process the tracking feature vector, log feature vector, and indicator feature vector corresponding to any node using a cross-attention mechanism to obtain multiple attention feature vectors corresponding to the node; concatenate the multiple attention feature vectors corresponding to the node to obtain a concatenated feature vector corresponding to the node; form a feature matrix from the concatenated feature vectors corresponding to at least one node; and input the feature matrix into a feature fusion model so that the feature fusion model encodes the concatenated feature vectors corresponding to at least one node and outputs a total fused feature vector.

[0130] In another embodiment of this application, the determination module is used to input the total fusion feature vector into the fault root cause determination model, so that the fault root cause determination model determines the fault root cause of the target service based on the mapping relationship between the total fusion feature vector and the fault root cause, and outputs the fault root cause of the target service.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] Figure 4 This diagram illustrates a structural block diagram of an electronic device 400 provided in an exemplary embodiment of this application. Typically, the electronic device 400 includes a processor 401 and a memory 402.

[0133] Processor 401 can be implemented in at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor; the main processor is a processor for processing data in the wake-up state, and the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an artificial intelligence processor for handling computational operations related to machine learning.

[0134] The memory 402 may include one or more computer-readable storage media, which may be non-transitory computer-readable storage media, such as CD-ROM (Compact Disc Read-Only Memory), ROM, RAM (Random Access Memory), magnetic tape, floppy disk, and optical data storage devices. The computer-readable storage medium stores at least one computer program, which, when executed, enables a method for determining the root cause of a fault.

[0135] Of course, the aforementioned electronic device may also include other components, such as input / output interfaces and communication components. Input / output interfaces provide an interface between the processor and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between the electronic device and other devices.

[0136] Those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the electronic device 400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0137] This application provides a computer-readable storage medium storing at least one computer program, which, when executed by a processor, can implement the above-mentioned method for determining the root cause of a fault.

[0138] This application provides a computer program product, which includes a computer program that, when executed by a processor, can implement the above-mentioned method for determining the root cause of a fault.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for determining the root cause of a fault, characterized in that, The method includes: After the target service requested by the target service request fails, based on the node information of at least one node on the service path corresponding to the target service request, the tracking data, log data and indicator data corresponding to each node are obtained, and the node is the service interface called during the process of the target service. Feature extraction is performed on the tracking data, log data, and indicator data corresponding to each node to obtain the tracking feature vector, log feature vector, and indicator feature vector corresponding to each node. The tracking feature vector, log feature vector, and indicator feature vector corresponding to at least one node are fused to obtain the total fused feature vector. Based on the total fused feature vector, the root cause of the failure of the target service is determined.

2. The method according to claim 1, characterized in that, The node information includes at least one of a service path identifier, a node identifier, and an execution time range. The step of obtaining tracking data, log data, and metric data corresponding to each node based on the node information of at least one node on the service path corresponding to the target service request includes: For any node, based on the service path identifier and node identifier of the node, obtain the tracking data containing the service path identifier and node identifier of the node from the tracking data related to the target service request, and use it as the tracking data corresponding to the node; Based on the service path identifier and node identifier of the node, log data containing the service path identifier and node identifier of the node is obtained from the log data related to the target service request, and used as the log data corresponding to the node; Based on the execution time range of the node, obtain the indicator data whose execution time falls within the execution time range from the indicator data related to the target service request, and use it as the indicator data corresponding to the node.

3. The method according to claim 2, characterized in that, Before obtaining log data containing the service path identifier and node identifier of the node from the log data related to the target service request based on the service path identifier and node identifier of the node, and using it as the log data corresponding to the node, the method further includes: In response to the target service request, during the process of calling any service interface to provide the target service, whenever a log is generated, the service path identifier and the node identifier corresponding to the called interface are injected into the generated log.

4. The method according to claim 1, characterized in that, The step involves extracting features from the tracking data, log data, and metric data corresponding to each node to obtain the tracking feature vector, log feature vector, and metric feature vector for each node, including: For any node, the tracking data corresponding to the node is processed using the first processing method to extract features, thereby obtaining the tracking feature vector corresponding to the node. The tracking feature vector incorporates features from the tracking data of the node's neighboring nodes. The second processing method is used to extract features from the log data corresponding to the node to obtain the log feature vector corresponding to the node. The log feature vector can reflect the contextual relationship between log data. The third processing method is used to extract features from the indicator data corresponding to the node to obtain the indicator feature vector corresponding to the node. The features of the indicator data related to the target service request in the indicator feature vector are enhanced.

5. The method according to claim 4, characterized in that, The first processing method is used to extract features from the tracking data corresponding to the node to obtain the tracking feature vector corresponding to the node, including: The tracking data of the node in different dimensions are fused to obtain the fused tracking data of the node; The fused tracking data of each node is input into the first feature extraction model, so that the first feature extraction model adjusts the weight values ​​of each neighbor node of the node, encodes the fused tracking data of the node and the tracking data of each neighbor node and their weight values, and outputs the tracking feature vector corresponding to the node.

6. The method according to claim 4, characterized in that, The second processing method is used to extract features from the log data corresponding to the node to obtain the log feature vector corresponding to the node, including: Extract the semantic features of each log data corresponding to the node; The semantic features of each log data corresponding to the node are input into the second feature extraction model, so that the second feature extraction model encodes the semantic features of each log data based on the context order of each log data and outputs the log feature vector corresponding to the node.

7. The method according to claim 4, characterized in that, The third processing method is used to extract features from the indicator data corresponding to the node, resulting in an indicator feature vector corresponding to the node, including: The various indicator data corresponding to the node are fused to obtain the fused indicator data of the node; Based on the correlation between the fusion index data and the target service request, determine the Gaussian weight value corresponding to the fusion index data; The fusion index data of the node and its corresponding Gaussian weight value are input into the third feature extraction model, so that the third feature extraction model encodes the fusion index data based on the Gaussian weight value corresponding to the fusion index data and outputs the index feature vector corresponding to the node.

8. The method according to claim 1, characterized in that, The process of fusing the tracking feature vector, log feature vector, and indicator feature vector corresponding to at least one node to obtain a total fused feature vector includes: For any node, a cross-attention mechanism is used to process the tracking feature vector, log feature vector, and metric feature vector corresponding to the node to obtain multiple attention feature vectors corresponding to the node. The multiple attention feature vectors corresponding to the node are concatenated to obtain the concatenated feature vector corresponding to the node. The concatenated feature vectors corresponding to at least one node are used to form a feature matrix; The feature matrix is ​​input into the feature fusion model so that the feature fusion model encodes the concatenated feature vector corresponding to at least one node and outputs the total fused feature vector.

9. The method according to claim 1, characterized in that, The step of determining the root cause of the target service's failure based on the total fused feature vector includes: The total fusion feature vector is input into the fault root cause determination model, so that the fault root cause determination model determines the fault root cause of the target service based on the mapping relationship between the total fusion feature vector and the fault root cause, and outputs the fault root cause of the target service.

10. A device for determining the root cause of a fault, characterized in that, The device includes: The acquisition module is used to acquire, based on the node information of at least one node on the service path corresponding to the target service request, the tracking data, log data and indicator data of each node after the target service requested by the target service request fails. The node is the service interface called during the process of the target service. The extraction module is used to extract features from the tracking data, log data and indicator data corresponding to each node, and obtain the tracking feature vector, log feature vector and indicator feature vector corresponding to each node. The fusion module is used to fuse the tracking feature vector, log feature vector and indicator feature vector corresponding to at least one node to obtain the total fused feature vector; The determination module is used to determine the root cause of the failure of the target service based on the total fused feature vector.

11. An electronic device, characterized in that, It includes a processor and a memory; the memory stores at least one piece of program code; the at least one piece of program code is called and executed by the processor to implement the method for determining the root cause of a fault as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, enables the determination of the root cause of a fault as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, enables the determination of the root cause of a fault as described in any one of claims 1 to 9.

Images