Fault root cause analysis method and device of distributed software system and storage medium

Abstract

The invention discloses a fault root cause analysis method and device for a distributed software system and a storage medium, and the method comprises the steps: obtaining fault data of the distributed software system, and extracting key information from the fault data; processing the key information according to a triple model to obtain a knowledge graph corresponding to the key information; when a fault analysis task is received, performing abnormal chain reasoning based on the knowledge graph, and generating a fault reason description text of the distributed software system based on an abnormal chain reasoning result; and inputting the fault reason description text into the large model to obtain a fault analysis result which corresponds to the fault analysis task and is deduced by the large model. According to the scheme, the technical problems that a traditional fault detection and root cause analysis method is low in analysis efficiency and inaccurate in root cause positioning are solved, and rapid and accurate fault attribution is achieved through the knowledge graph and the large language model.

Description

Technical Field

[0001] The present invention relates to the technical field of software system fault analysis, and in particular to a method, device and storage medium for fault root cause analysis of a distributed software system. Background Art

[0002] With the rapid development of information technology, the scale and complexity of modern large-scale distributed software systems are also increasing. These systems are usually composed of many services, components and nodes, each of which has different functions and tasks. They are widely used in cloud computing, Internet of Things, financial transactions and e-commerce. Large-scale distributed software systems usually contain multiple interdependent services and components, relying on network communication and data exchange. This high degree of coupling makes troubleshooting extremely difficult, because the failure of one component may trigger a chain reaction, causing problems in other components or services. Failures in the system manifest in various forms, including performance degradation, service interruption, network delay, etc.

[0003] Currently, many organizations rely on traditional fault detection and root cause analysis methods, such as manual log analysis and rule-based monitoring systems. These methods face the following limitations: Due to the huge amount of log data and its dispersion, manual log analysis takes a lot of time and effort, and it is easy to ignore implicit fault clues. Traditional monitoring tools usually monitor based on preset alarm rules, but these rules are often only applicable to specific fault scenarios and it is difficult to cover all potential fault types. Moreover, the setting of rules relies too much on historical experience, and it is easy to have false positives or false negatives in the face of complex new fault modes. Failures in distributed systems often span multiple components and nodes. Traditional methods find it difficult to understand the fault propagation path and the interdependencies in the system from a global perspective, resulting in inaccurate root cause location. How to efficiently and accurately locate the root cause of faults is an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a method, device and storage medium for fault root cause analysis of a distributed software system to achieve efficiency and accuracy of fault analysis.

[0005] According to one aspect of the present invention, a method for analyzing root causes of faults in a distributed software system is provided, comprising:

[0006] Obtaining fault data of a distributed software system and extracting key information from the fault data;

[0007] Process the key information according to the triple model to obtain a knowledge graph corresponding to the key information;

[0008] When a fault analysis task is received, an exception chain reasoning is performed based on the knowledge graph, and a fault cause description text of the distributed software system is generated based on the exception chain reasoning result;

[0009] The fault cause description text is input into the large model to obtain the fault analysis result inferred by the large model and corresponding to the fault analysis task.

[0010] In a possible implementation, the acquiring fault data of the distributed software system and extracting key information from the fault data includes:

[0011] Obtaining log data, monitoring data, and fault reports of the distributed software system, and determining the log data, monitoring data, and fault reports as the fault data;

[0012] The fault data is preprocessed to obtain the key information.

[0013] In a possible implementation, the processing of the key information according to the triple model to obtain a knowledge graph corresponding to the key information includes:

[0014] Converting the key information into triple-form information according to the triple-form model;

[0015] Convert the triple-form information into a graph structure to obtain a knowledge graph corresponding to the key information;

[0016] The triple form information includes entities, attributes and relationships, and the knowledge graph includes nodes and edges;

[0017] The entity corresponds to a service, application, node, database or network device in the distributed software system; the attribute includes at least one of state, load and response time;

[0018] The relationship includes at least one of service dependency, interaction, causality, and data flow path.

[0019] In a possible implementation, when receiving a fault analysis task, performing abnormal chain reasoning based on the knowledge graph, and generating a fault cause description text of the distributed software system based on the abnormal chain reasoning result, includes:

[0020] Determine a performance change sequence corresponding to each node in the knowledge graph, and calculate the weight of an edge between two adjacent nodes in the knowledge graph based on the performance change sequence;

[0021] Determine a target node in the knowledge graph according to the fault analysis task;

[0022] Traversing the knowledge graph based on a breadth-first search algorithm to obtain at least two candidate paths associated with the target node, and determining a target fault path based on the weights of the edges in the candidate paths and the lengths of the candidate paths;

[0023] The node data corresponding to each node in the target fault path is determined based on the knowledge graph, and the fault cause description text is generated based on the node data.

[0024] In a possible implementation, the calculating the weight of the edge between two adjacent nodes in the knowledge graph based on the performance change sequence includes:

[0025] For an adjacent first node and a second node, calculating a Pearson correlation coefficient based on a performance change sequence of the first node and a performance change sequence of the second node;

[0026] The calculated Pearson correlation coefficient is determined as the weight of the edge between the first node and the second node.

[0027] In a possible implementation, inputting the fault cause description text into a large model to obtain a fault analysis result corresponding to the fault analysis task inferred by the large model includes:

[0028] Determining a user question text according to the fault analysis task, filtering the user question text, and retrieving professional knowledge from a knowledge base;

[0029] The fault cause description text, the filtered user question text and the professional knowledge are input into the large model to obtain the fault analysis result inferred by the large model.

[0030] In a possible implementation, after obtaining the fault analysis result corresponding to the fault analysis task inferred by the large model, the method further includes:

[0031] Extracting the fault analysis result through an information extraction module to obtain triplet information corresponding to the fault analysis result;

[0032] The triple information is matched and verified with the knowledge graph.

[0033] In a possible implementation, the method includes:

[0034] Based on the fault data, the knowledge graph is checked according to a preset period to determine the content to be updated in the knowledge graph;

[0035] The content to be updated is updated according to preset update rules to obtain the updated knowledge graph.

[0036] According to another aspect of the present invention, there is provided a device for analyzing root causes of faults in a distributed software system, comprising:

[0037] A key information extraction module, used to obtain fault data of a distributed software system and extract key information from the fault data;

[0038] A knowledge graph construction module, used to process the key information according to a triple model to obtain a knowledge graph corresponding to the key information;

[0039] An exception chain reasoning module is used to perform exception chain reasoning based on the knowledge graph when receiving a fault analysis task, and generate a fault cause description text of the distributed software system based on the exception chain reasoning result;

[0040] The fault result determination module is used to input the fault cause description text into the large model to obtain the fault analysis result inferred by the large model and corresponding to the fault analysis task.

[0041] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0042] at least one processor;

[0043] and a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for root cause analysis of a distributed software system described in any embodiment of the present invention.

[0045] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for root cause analysis of a distributed software system described in any embodiment of the present invention when executed.

[0046] The technical solution of the embodiment of the present invention includes: obtaining the fault data of the distributed software system and extracting key information from the fault data; processing the key information according to the triple model to obtain the knowledge graph corresponding to the key information; when receiving the fault analysis task, performing abnormal chain reasoning based on the knowledge graph, and generating a fault cause description text of the distributed software system based on the abnormal chain reasoning result; inputting the fault cause description text into the large model to obtain the fault analysis result corresponding to the fault analysis task reasoned by the large model. The solution of the embodiment of the present invention solves the technical problems of low analysis efficiency and inaccurate root cause location of traditional fault detection and root cause analysis methods, and realizes the rapid and accurate attribution of faults using knowledge graphs and large language models.

[0047] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A flowchart of a method for analyzing root causes of failures in a distributed software system provided in Embodiment 1 of the present invention;

[0050] Figure 2 The knowledge graph corresponding to the distributed software system provided in the first embodiment of the present invention;

[0051] Figure 3 A flow chart of a method for analyzing root causes of failures in a distributed software system provided in Embodiment 2 of the present invention;

[0052] Figure 4 An example diagram of a knowledge graph constructed with monitoring target KPIs provided in the second embodiment of the present invention;

[0053] Figure 5 This is an example diagram of tracking the root cause of performance anomalies provided in the second embodiment of the present invention;

[0054] Figure 6 A question-and-answer flow chart provided for Embodiment 2 of the present invention;

[0055] Figure 7 A diagram of the information filtering model training process provided in the second embodiment of the present invention;

[0056] Figure 8 A schematic diagram of the structure of a device for analyzing root causes of faults in a distributed software system provided in Embodiment 3 of the present invention;

[0057] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0059] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] Embodiment 1

[0061] Figure 1 This is a flow chart of a method for analyzing the root cause of a fault in a distributed software system provided in the first embodiment of the present invention. This embodiment is applicable to the case of analyzing the root cause of a fault in a distributed software system. The method can be executed by a device for analyzing the root cause of a fault in a distributed software system. The device can be implemented in the form of hardware and / or software, and the device can be configured in a computer device. Figure 1 As shown, the method comprises the following steps:

[0062] S110. Obtain fault data of the distributed software system, and extract key information from the fault data.

[0063] The fault data may be data related to the root cause analysis of the fault in the distributed software system, and the key information may be data obtained after processing the fault data. The root cause of the fault in the distributed software system may be determined by analyzing the key information.

[0064] In an embodiment of the present invention, obtaining fault data of a distributed software system and extracting key information from the fault data may include: obtaining log data, monitoring data and fault reports of the distributed software system, determining the log data, monitoring data and fault reports as fault data; and preprocessing the fault data to obtain key information.

[0065] Specifically, large-scale distributed software system fault data is usually presented in the form of logs, alarm information, and monitoring indicators, and the above data is collected and organized through system monitoring tools and analysis tools. Then, the fault data is preprocessed, for example, the collected fault data can be cleaned to remove noise, errors, and duplicate data, and the data format can be unified and the data units can be standardized for subsequent processing.

[0066] After obtaining the preprocessed fault data, key information can be extracted from it, including system components, the interaction relationship between services, the timestamp of the fault event, the characteristics of abnormal behavior, etc.

[0067] It should be noted that the extracted key information should cover entities (such as services, nodes, databases, etc.) and their attributes (such as status, load, response time, etc.), as well as the relationships between entities (such as service dependencies, data flow paths, etc.), so that it can be used to build a knowledge graph.

[0068] In an optional implementation, the data source of fault data mainly relies on multiple monitoring tools used by the enterprise IT team to obtain visibility into the environment, including hosts, applications, logs, and network infrastructure. At the same time, applications of distributed software systems can also use tracking tools such as Jaeger and Zipkin to track application-level indicators such as RPC latency and errors.

[0069] In this embodiment, an application-aware multi-cloud network observability platform can be used to obtain passive telemetry data from multiple data sources, including: VM management platforms and SDDC controllers (monitoring individual hosts and virtual networks in turn), physical network devices (routers, switches, firewalls, etc.), public cloud providers' APIs, and Netflow / IPFIX data sources to obtain data flow information. This telemetry includes metadata about various entity types, typically with multiple performance indicators for each entity, each of which is a time series collected at minute intervals. For most indicators, data for more than one day will be aggregated into longer time intervals and stored as rolling summary windows.

[0070] S120. Process the key information according to the triple model to obtain a knowledge graph corresponding to the key information.

[0071] Specifically, the triple model can be used to structure the extracted key information into entities and their relationships, build a knowledge graph, and store the knowledge graph in a graph database. In other words, system components, events, and dependencies are stored in the form of a graph database.

[0072] In order to facilitate the display of the relationship between services, applications, nodes, databases and network devices in distributed software systems, visualization tools can also be used to visualize the knowledge graph to intuitively display the dependency paths and potential fault propagation paths between the various components of the system.

[0073] In an embodiment of the present invention, processing the key information according to the triple model to obtain a knowledge graph corresponding to the key information may include: converting the key information into triple form information according to the triple model; converting the triple form information into a graph structure to obtain a knowledge graph corresponding to the key information;

[0074] Among them, the information in triple form includes entities, attributes and relationships, and the knowledge graph includes nodes and edges; entities correspond to services, applications, nodes, databases or network devices in distributed software systems; attributes include at least one of status, load and response time; relationships include at least one of service dependency, interaction, causality and data flow path.

[0075] Specifically, converting key information into triples is the basis for building a knowledge graph. A triple is a data structure consisting of three parts: an entity, an attribute, and another entity. Entities represent various elements in a distributed software system, such as services, applications, nodes, databases, or network devices. These entities are the basic units in the knowledge graph, and they can be any system components with unique identification and specific functions. Attributes are used to describe the characteristics or status of an entity, such as status (such as running status, health status), load (such as CPU usage, memory usage), and response time. Relationships define the connection and interaction between entities, such as service dependency (one service depends on another service), interaction (communication between services), causality (one event causes another event to occur), and data flow path (the transmission path of data in the system). Relationships reveal the dynamic connection between entities.

[0076] like Figure 2 As shown, the knowledge graph corresponding to the distributed software system provided in the first embodiment of the present invention is shown. After the key information is converted into triples, these triples can be organized into a graph structure. In the knowledge graph, nodes represent entities, which are connected to each other by edges. Each node contains a set of attributes that describe the characteristics of the node. Edges represent the relationship between entities. Edges can also be accompanied by weight or type information to further refine the nature of the relationship.

[0077] By mapping the entities, attributes, and relationships in triples to the nodes and edges of a graph, we can build a knowledge graph that comprehensively reflects the structure and dynamic behavior of distributed software systems. The graph is not only easy to visualize, but also convenient for complex queries and root cause analysis operations.

[0078] S130. When a fault analysis task is received, abnormal chain reasoning is performed based on the knowledge graph, and a fault cause description text of the distributed software system is generated based on the abnormal chain reasoning result.

[0079] Specifically, when receiving a fault analysis task, the constructed knowledge graph can be used to perform exception chain reasoning to quickly locate and solve problems in the distributed software system.

[0080] First, it is necessary to monitor the key indicators of the distributed software system in real time, such as service response time, resource utilization, error logs, etc. When an abnormal value is detected, the fault analysis task is triggered, and the abnormality is preliminarily classified and located to determine the approximate scope of the abnormality.

[0081] Furthermore, the knowledge graph is used for in-depth anomaly chain reasoning: Based on the results of the preliminary analysis, entities related to the anomaly can be located in the knowledge graph. Starting from the anomaly entity, the relationship in the knowledge graph (such as service dependency, interaction, causal relationship, etc.) is traversed to find possible anomaly propagation paths. Based on the traversal results, one or more anomaly chains are constructed, which reflect the propagation path and possible causes of the anomaly in the system.

[0082] After the exception chain is constructed, a fault cause description text may be generated. The fault cause description text may be generated based on specific data corresponding to entities, attributes, and relationships in the exception chain.

[0083] In some embodiments, the abnormal chain reasoning based on the knowledge graph can be: according to predefined heuristic rules, link walking is performed on the knowledge graph, and the association relationship between entities and system component dependencies are combined to infer the root cause of the abnormal situation through a causal search algorithm, and generate a detailed fault cause description text.

[0084] S140: Input the fault cause description text into the large model to obtain the fault analysis result inferred by the large model and corresponding to the fault analysis task.

[0085] Specifically, the large language model is used to deeply process the generated fault cause description text, simplify the complex text, and extract key fault information, so as to obtain the fault analysis results corresponding to the fault analysis task, making the fault description more concise and easy to understand.

[0086] In this embodiment, with the help of the powerful natural language processing capabilities of the big model, the fault cause description text is automatically analyzed, the fault cause description is simplified, and a concise and easy-to-understand fault analysis result is generated, which greatly reduces manual intervention and improves the efficiency and accuracy of the analysis. The automated text processing and information simplification capabilities not only reduce data processing time, but also improve the readability of the analysis results. Technicians only need to focus on the extracted core information, which greatly reduces the workload of analysis and reduces dependence on senior technicians, while greatly improving the efficiency and accuracy of fault analysis.

[0087] In some embodiments, after obtaining the fault analysis results corresponding to the fault analysis task inferred by the large model, the fault analysis results can also be extracted through the information extraction module to obtain triple information corresponding to the fault analysis results; the triple information is matched and verified with the knowledge graph.

[0088] Specifically, the fault analysis results are extracted through the information extraction module to obtain triple information corresponding to the fault analysis results. In this way, the use of knowledge extraction to extract triples and match and verify with the knowledge graph can enhance the professionalism of subsequent answers from the large model.

[0089] In some embodiments, the knowledge graph can also be checked according to a preset period based on the fault data to determine the content to be updated in the knowledge graph; the content to be updated can be updated according to preset update rules to obtain an updated knowledge graph.

[0090] It is understandable that the fault data corresponding to each unit or component in the distributed system changes in real time, and the corresponding knowledge graph also needs to be continuously updated as the fault data changes, so that when the fault analysis results are determined based on the knowledge graph and the large model, the accuracy of the analysis can be achieved. Therefore, the knowledge graph can be checked according to a pre-set cycle to determine the content that needs to be updated, that is, the content to be updated. Then, the content to be updated is updated according to the preset update rules to obtain an updated knowledge graph.

[0091] For example, in practical applications, the knowledge graph can be checked regularly to carefully check whether there is any erroneous information in the knowledge graph, such as incorrect entity relationships, inaccurate attribute values, etc. At the same time, attention should be paid to whether there is outdated information. With the continuous updating and evolution of the system, some knowledge may no longer be applicable. It is also necessary to check whether there is incomplete information, that is, whether some key parts are missing. Use automated tools to review each node and edge in the knowledge graph.

[0092] When it is found that the information in the knowledge graph needs to be updated, different measures need to be taken according to the scale and nature of the update. If it is a modification of a small amount of information, such as a change in the value of an attribute of an individual entity, or the relationship description of an edge needs to be fine-tuned, it can be edited directly in the knowledge graph. This method can quickly and conveniently make local adjustments to the knowledge graph to ensure the timeliness and accuracy of the knowledge graph. However, if a large amount of information is updated, such as a major change in the system architecture, or new components and services are introduced, the process of information extraction and matching needs to be repeated. This process is relatively complex and requires re-extraction of key information from the original data (such as log data, monitoring data, and fault reports, etc.), and then according to the standard process of building a knowledge graph, triple modeling, graph database storage and other operations are performed to ensure that the knowledge graph can accurately reflect the latest status of the system.

[0093] By continuously monitoring the system operation status, the knowledge graph is dynamically updated to ensure that it reflects the latest system architecture and fault information. An incremental update strategy is adopted to adjust only the affected parts to maintain the timeliness and accuracy of the knowledge graph.

[0094] Embodiment 2

[0095] Figure 3 This is a flowchart of a method for analyzing the root cause of a fault in a distributed software system provided in the second embodiment of the present invention. This embodiment further optimizes the above embodiment. For its specific implementation, please refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here. Figure 3 As shown, the method includes:

[0096] S210. Obtain fault data of the distributed software system, and extract key information from the fault data.

[0097] S220. Process the key information according to the triple model to obtain a knowledge graph corresponding to the key information.

[0098] S230. Determine the performance change sequence corresponding to each node in the knowledge graph, and calculate the weight of the edge between two adjacent nodes in the knowledge graph based on the performance change sequence.

[0099] Among them, each node in the knowledge graph refers to each entity in the graph, which can also be understood as services, applications, network devices, etc. in distributed software systems. The performance change sequence refers to the sequence of changes in the attributes corresponding to the nodes over time.

[0100] In some embodiments, calculating the weight of an edge between two adjacent nodes in a knowledge graph based on a performance change sequence may include: for an adjacent first node and a second node, calculating the Pearson correlation coefficient based on the performance change sequence of the first node and the performance change sequence of the second node; and determining the calculated Pearson correlation coefficient as the weight of the edge between the first node and the second node.

[0101] It is understandable that in the process of abnormal chain reasoning and root cause search based on the knowledge graph, the candidate root cause paths are mainly mined through weight sorting. In this process, a weight value is assigned to each edge X→Y in the knowledge graph to measure the impact of X on Y. There are two main steps in how to assign weights:

[0102] It effectively represents the value changes of all nodes in the time period t. The value change events of nodes can be represented as a binary sequence E.

[0103] Calculate two sequences E X and E Y The Pearson correlation coefficient between them is used as the edge weight, where E X is the value change of node X, E Y is the value change of node Y.

[0104] The specific calculation process of assigning a weight value to each edge X→Y is as follows:

[0105] To check whether the time series KPI has significant value changes, a t-test is used. Given an event type E, whose timestamp is T E =(t 1 ,t 2 ,...,t n ), the time series KPI set is S = (s 1 ,s 2 ,...,s m )

[0106] Calculate the event e separately i Subseries before occurrence and the subsequent subseries Then calculate t by t-test score , the formula is (where n is the window size, and is Γ front and Γ rear The mean of and is their variance).

[0107] If t score>v (v is the statistical test result under a specific degree of freedom), it means that the KPI has a significant change between the two windows. score The positive or negative value of determines whether the KPI has increased or decreased significantly. The performance change sequence can be converted into a ternary sequence, where "0" indicates that the performance indicator has no significant change, "1" indicates that the KPI value has increased, and "-1" indicates that the KPI value has decreased.

[0108] Finally, according to the performance change sequence E 1 and E 2 Calculate the weight of the edge in the causal graph, the formula is

[0109]

[0110] S240. Determine the target node in the knowledge graph according to the fault analysis task.

[0111] Among them, the target node can be a node corresponding to the fault analysis task. For example, if the fault analysis task is to perform fault analysis on an application corresponding to a node in the knowledge graph, then the node in the knowledge graph can be used as the target node.

[0112] like Figure 4 As shown, it is an example diagram of the knowledge graph constructed with the monitoring target KPI provided in the second embodiment of the present invention.

[0113] S250. Traverse the knowledge graph based on a breadth-first search algorithm to obtain at least two candidate paths associated with the target node, and determine the target fault path based on the weights of the edges in the candidate paths and the lengths of the candidate paths.

[0114] Specifically, the knowledge graph can be traversed based on the breadth-first search algorithm to obtain multiple candidate paths associated with the target node, that is, nodes and paths that may be related to the target node failure. Then, the weight of the edge in the candidate path and the length of the candidate path are calculated to obtain the total weight of each candidate path, and then the total weight is sorted to determine the target fault path.

[0115] In some implementations, the process of performing causal search on the knowledge graph adopts an algorithm based on breadth-first search. Its main purpose is to create a queue to store path vectors, initialize the queue with the first path starting from the source node, and then loop until the queue is empty. During the loop, get the path at the front of the queue, determine whether the last node of the path is the target node (a node without a predecessor), if so, print the path, and loop for all nodes connected to the current node. Sort the paths by combining the weight distribution method. According to the following two rules: Rule 1 The sum of the weights of all edges on the path, the larger the sum of the weights, the higher the priority. Rule 2 The length of the path, the shorter the path, the higher the priority. With this sorting method, the path that is most likely to be the root cause will get the highest priority. Such as Figure 5 , which is an example diagram of tracing the root cause of performance anomalies provided in Embodiment 2 of the present invention.

[0116] S260. Determine the node data corresponding to each node in the target fault path based on the knowledge graph, and generate a fault cause description text based on the node data.

[0117] Specifically, data such as edges and attributes corresponding to nodes in the target fault path may be integrated into a text form as a fault cause description text.

[0118] S270: Input the fault cause description text into the large model to obtain the fault analysis result inferred by the large model and corresponding to the fault analysis task.

[0119] In an embodiment of the present invention, a fault cause description text is input into a large model to obtain a fault analysis result corresponding to the fault analysis task inferred by the large model, which may include: determining a user question text according to the fault analysis task, filtering the user question text, and retrieving professional knowledge from a knowledge base; inputting the fault cause description text, the filtered user question text and professional knowledge into the large model to obtain a fault analysis result inferred by the large model.

[0120] Specifically, the detailed steps of large language model processing may include:

[0121] Text parsing: parse the key components, services, and failure modes in the fault cause description text, and extract relevant entities and relationships;

[0122] Semantic analysis: Analyze the semantics of the text, identify the core information in the fault description, and exclude redundant or irrelevant content;

[0123] Semantic merging and supplementing: For multiple fault cause descriptions, the model will perform semantic analysis, identify possible similar or repeated information, and merge the same or similar fault causes. At the same time, the large language model can automatically supplement the missing information based on the context to ensure the completeness and coherence of the text;

[0124] Generate concise text: After processing, the generated fault cause text is more concise and structured, highlighting the root cause, impact scope and related components of the system failure;

[0125] Output and storage: The final processed text will be output in a structured or semi-structured format and stored in the local fault knowledge base to provide support for subsequent root cause analysis, question-and-answer systems, or automated decision-making.

[0126] In some embodiments, Figure 6 The figure shows a question-answering flow chart provided by the second embodiment of the present invention. The user asks a question to the system, and after the question is filtered through information, it is combined with relevant professional knowledge in the large-scale distributed software root cause analysis knowledge base, and node data (i.e., or fault description text) to form a guide prompt information, which is input into the professional question-answering module to obtain the answer, that is, the fault analysis result inferred by the large model.

[0127] In another optional embodiment, the fault cause text information processed by the large language model will be stored in a local knowledge base in a structured or semi-structured format. This not only helps to effectively organize and manage fault information, but also facilitates subsequent fault analysis. By continuously incorporating new fault cause text information into the knowledge base, the knowledge base can be continuously expanded, thereby improving the integrity and richness of the knowledge base.

[0128] In addition, knowledge base maintenance requires version control of the knowledge base so that the change history of the knowledge base can be tracked. This is very important for rolling back to a previous version, comparing differences between different versions, and ensuring data consistency and traceability. You can use a version control system (such as Git) to manage the version of the knowledge base. Each time you update the knowledge base, create a new version and record the content and reasons for the update. In this way, you can easily restore to a previous version when needed.

[0129] In another preferred embodiment, in order to limit the scope of questions that can be answered by the large model, the large language model does not need to answer questions in other fields. A BERT-based information filter is added to filter the questions to limit the scope of questions that can be answered by the large model. Generally speaking, when using BERT in text classification tasks, the classification word vector H of the BERT result is usually combined with softmax to make a simple classifier to predict the probability of the category label L. The formula is P(L|H)=softmax(WH), where W is the parameter matrix of the classification task. All parameters in BERT and W are fine-tuned by maximizing the logarithmic probability of the correct label. Now it is modified to use the fully connected layer to obtain the probability of each label. The formula is P(L|H)=FC(H). During training, the vector dimension of the input to the fully connected layer is 768. There are two hidden layers with dimensions of 384 and 768 respectively. The output dimension is the number of categories, and the label with a larger probability is selected as the classification result. In the process of training the information filter, the training data is input into BERT, and then the result of BERT is input into the fully connected layer to obtain the classification result CLS. According to the labels in the data set, only the parameters of the fully connected layer need to be updated during training. As Figure 7 The figure shows the information filtering model training process diagram provided in the second embodiment of the present invention.

[0130] The technical solution of the embodiment of the present invention, by constructing a knowledge graph, comprehensively describes the dependencies between various components in the system, provides a global perspective, can intuitively display the structure of the system and the fault propagation path, and greatly improves the ability to parse complex dependencies. Combined with the reasoning ability of the knowledge graph, it is possible to deeply explore the fault path and potential root causes through technologies such as link walking.

[0131] With the powerful natural language processing capabilities of the big model, it automatically analyzes unstructured data such as fault logs and reports, generates and simplifies descriptions of fault causes, greatly reduces manual intervention, and improves analysis efficiency and accuracy. The big language model simplifies complex fault descriptions and extracts key elements, making the causes of faults clearer and easier to understand. Automated text processing and information simplification capabilities not only reduce data processing time, but also improve the readability of analysis results.

[0132] Through automated data extraction, knowledge graph construction, and text processing of large language models, manual intervention is greatly reduced. The system can automatically extract key information from logs and monitoring data and generate concise fault descriptions. Technicians only need to focus on the extracted core information, which greatly reduces the workload of analysis and reduces dependence on senior technicians, while greatly improving the efficiency and accuracy of fault analysis.

[0133] Embodiment 3

[0134] Figure 8This is a schematic diagram of the structure of a fault root cause analysis device for a distributed software system provided in Embodiment 3 of the present invention. Figure 8 As shown, the device comprises:

[0135] The key information extraction module 310 is used to obtain the fault data of the distributed software system and extract key information from the fault data;

[0136] A knowledge graph construction module 320 is used to process the key information according to a triple model to obtain a knowledge graph corresponding to the key information;

[0137] The exception chain reasoning module 330 is used to perform exception chain reasoning based on the knowledge graph when receiving a fault analysis task, and generate a fault cause description text of the distributed software system based on the exception chain reasoning result;

[0138] The fault result determination module 340 is used to input the fault cause description text into the large model to obtain the fault analysis result inferred by the large model and corresponding to the fault analysis task.

[0139] Based on the above technical solution, the key information extraction module 310 includes:

[0140] A fault data processing submodule is used to obtain log data, monitoring data and fault reports of the distributed software system, and determine the log data, monitoring data and fault reports as fault data;

[0141] Preprocess the fault data to obtain key information.

[0142] Based on the above technical solution, the knowledge graph construction module 320 includes:

[0143] The knowledge graph construction submodule is used to convert key information into triple-form information according to the triple model;

[0144] Convert triple-form information into a graph structure to obtain the knowledge graph corresponding to the key information;

[0145] Among them, the triple form information includes entities, attributes and relationships, and the knowledge graph includes nodes and edges;

[0146] The entity corresponds to a service, application, node, database or network device in a distributed software system; the attribute includes at least one of state, load and response time;

[0147] The relationship includes at least one of service dependency, interaction, causality, and data flow path.

[0148] Based on the above technical solution, the abnormal chain reasoning module 330 includes:

[0149] The weight calculation submodule is used to determine the performance change sequence corresponding to each node in the knowledge graph, and calculate the weight of the edge between two adjacent nodes in the knowledge graph based on the performance change sequence;

[0150] The target node determination submodule is used to determine the target node in the knowledge graph according to the fault analysis task;

[0151] A target fault path determination submodule is used to traverse the knowledge graph based on a breadth-first search algorithm to obtain at least two candidate paths associated with the target node, and determine the target fault path based on the weights of the edges in the candidate paths and the lengths of the candidate paths;

[0152] The fault cause description text determination submodule is used to determine the node data corresponding to each node in the target fault path based on the knowledge graph, and generate the fault cause description text based on the node data.

[0153] Based on the above technical solution, the weight calculation submodule is specifically used for:

[0154] For adjacent first and second nodes, calculating a Pearson correlation coefficient based on a performance change sequence of the first node and a performance change sequence of the second node;

[0155] The calculated Pearson correlation coefficient is determined as the weight of the edge between the first node and the second node.

[0156] On the basis of the above technical solution, the fault result determination module 340 is specifically used to:

[0157] Determine user question text according to the fault analysis task, filter the user question text, and retrieve professional knowledge from the knowledge base;

[0158] The fault cause description text, filtered user problem text and professional knowledge are input into the big model to obtain the fault analysis results inferred by the big model.

[0159] Optionally, the device further comprises a verification and matching module, which is used to extract the fault analysis result through the information extraction module after obtaining the fault analysis result corresponding to the fault analysis task inferred by the large model, so as to obtain triple information corresponding to the fault analysis result;

[0160] Match and verify the triple information with the knowledge graph.

[0161] Optionally, the device is also used for:

[0162] Based on the fault data, the knowledge graph is checked according to the preset period to determine the content to be updated in the knowledge graph;

[0163] The content to be updated is updated according to the preset update rules to obtain an updated knowledge graph.

[0164] The device for analyzing root cause of failure of a distributed software system provided in an embodiment of the present invention can execute the method for analyzing root cause of failure of a distributed software system provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0165] Embodiment 4

[0166] Fig. 9 A schematic diagram of the structure of an electronic device provided for Embodiment 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0167] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0168] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0169] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a method for analyzing the root cause of a fault in a distributed software system.

[0170] In some embodiments, the method for root cause analysis of faults in a distributed software system may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for root cause analysis of faults in a distributed software system described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for root cause analysis of faults in a distributed software system in any other appropriate manner (e.g., by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0173] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0176] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0177] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0178] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for analyzing root causes of failures in a distributed software system, characterized in that: include: Obtaining fault data of a distributed software system and extracting key information from the fault data; Process the key information according to the triple model to obtain a knowledge graph corresponding to the key information; When a fault analysis task is received, an exception chain reasoning is performed based on the knowledge graph, and a fault cause description text of the distributed software system is generated based on the exception chain reasoning result; The fault cause description text is input into the large model to obtain the fault analysis result inferred by the large model and corresponding to the fault analysis task.

2. The method according to claim 1, characterized in that The obtaining of fault data of the distributed software system and extracting key information from the fault data includes: Obtaining log data, monitoring data, and fault reports of the distributed software system, and determining the log data, monitoring data, and fault reports as the fault data; The fault data is preprocessed to obtain the key information.

3. The method according to claim 1, characterized in that The step of processing the key information according to the triple model to obtain a knowledge graph corresponding to the key information includes: Converting the key information into triple-form information according to the triple-form model; Convert the triple-form information into a graph structure to obtain a knowledge graph corresponding to the key information; The triple form information includes entities, attributes and relationships, and the knowledge graph includes nodes and edges; The entity corresponds to a service, application, node, database or network device in the distributed software system; the attribute includes at least one of state, load and response time; The relationship includes at least one of service dependency, interaction, causality, and data flow path.

4. The method according to claim 1, characterized in that: When receiving a fault analysis task, performing abnormal chain reasoning based on the knowledge graph, and generating a fault cause description text of the distributed software system based on the abnormal chain reasoning result, including: Determine a performance change sequence corresponding to each node in the knowledge graph, and calculate the weight of an edge between two adjacent nodes in the knowledge graph based on the performance change sequence; Determine a target node in the knowledge graph according to the fault analysis task; Traversing the knowledge graph based on a breadth-first search algorithm to obtain at least two candidate paths associated with the target node, and determining a target fault path based on the weights of the edges in the candidate paths and the lengths of the candidate paths; The node data corresponding to each node in the target fault path is determined based on the knowledge graph, and the fault cause description text is generated based on the node data.

5. The method according to claim 4, characterized in that The calculating the weight of the edge between two adjacent nodes in the knowledge graph based on the performance change sequence includes: For an adjacent first node and a second node, calculating a Pearson correlation coefficient based on a performance change sequence of the first node and a performance change sequence of the second node; The calculated Pearson correlation coefficient is determined as the weight of the edge between the first node and the second node.

6. The method according to claim 4, characterized in that The step of inputting the fault cause description text into the large model to obtain the fault analysis result corresponding to the fault analysis task inferred by the large model includes: Determining a user question text according to the fault analysis task, filtering the user question text, and retrieving professional knowledge from a knowledge base; The fault cause description text, the filtered user question text and the professional knowledge are input into the large model to obtain the fault analysis result inferred by the large model.

7. The method according to claim 6, characterized in that After obtaining the fault analysis result corresponding to the fault analysis task inferred by the large model, the method further includes: Extracting the fault analysis result through an information extraction module to obtain triplet information corresponding to the fault analysis result; The triple information is matched and verified with the knowledge graph.

8. The method according to claim 1, characterized in that The method comprises: Based on the fault data, the knowledge graph is checked according to a preset period to determine the content to be updated in the knowledge graph; The content to be updated is updated according to preset update rules to obtain the updated knowledge graph.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for root cause analysis of a distributed software system according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for root cause analysis of a distributed software system according to any one of claims 1 to 8 when executed.

Images