Fault location methods, devices, electronic equipment and software products

Abstract

This application provides a fault location method, apparatus, electronic device, and program product. The method includes: receiving target alarm events from a server cluster and analyzing and processing the target alarm events to obtain fault task information, which includes an alarm description, an initial fault node, and an initial fault time; based on the initial fault node and initial fault time, performing topological reasoning on the fault task information in the service dependency topology graph of the server cluster to construct a fault propagation chain; determining multiple association verification sets for multiple alarm pairs corresponding to the fault propagation chain; and generating fault analysis results corresponding to the target alarm event based on the alarm description and the association verification sets of each alarm pair. The method of this application improves the reliability of fault location.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a fault location method, apparatus, electronic device, and program product. Background Technology

[0002] As data centers continue to expand in scale, a single faulty node can trigger a cascading alarm effect through complex dependencies during operation.

[0003] In existing technologies, when quickly locating the initial fault node in massive, heterogeneous, and noisy alarm data, a model can be built based on the dependencies between indicators to capture the statistical dependencies between data, thereby locating the fault node. However, using only the initial alarm event as the analysis object limits the analysis results to a single data source, resulting in low reliability of fault location. Summary of the Invention

[0004] This application provides a fault location method, apparatus, electronic device, and program product to solve the reliability problem of fault location.

[0005] Firstly, this application provides a fault location method, including:

[0006] Receive target alarm events from the server cluster, analyze and process the target alarm events to obtain fault task information, which includes alarm description, initial fault node and initial fault time.

[0007] Based on the initial fault node and the initial fault time, topological reasoning is performed on the fault task information in the service dependency topology graph of the server cluster to construct a fault propagation chain;

[0008] Determine multiple associated verification sets corresponding to multiple alarm pairs in the fault propagation chain;

[0009] Based on the alarm description and the associated verification set of each alarm pair, the fault analysis result corresponding to the target alarm event is generated.

[0010] In this embodiment, based on the initial fault node and the initial fault time, a fault propagation chain is constructed by performing topological reasoning on the fault task information in the business dependency topology graph of the server cluster. The association verification of multiple alarm pairs in the fault propagation chain is completed, and the fault analysis results are obtained. This breaks through the limitation of a single data source and improves the reliability of fault location.

[0011] Optionally, based on the initial fault node and the initial fault time, topological reasoning is performed on the fault task information in the service dependency topology graph of the server cluster to construct a fault propagation chain, including:

[0012] Based on the initial fault node, determine the fault dependency subgraph corresponding to the target alarm event in the service dependency topology graph;

[0013] Based on the initial fault time of the initial fault node, the fault dependency subgraph is cut to obtain the fault propagation chain.

[0014] In this embodiment, timing pruning is performed by combining alarm timestamps, which can eliminate nodes that are not related to normal operation and abnormal events that are not caused by the current fault, effectively filtering invalid interference information in the topology and improving the reliability of fault location.

[0015] Optionally, based on the initial fault node, a fault dependency subgraph corresponding to the target alarm event is determined in the service dependency topology graph, including:

[0016] Starting from the initial fault node, the business dependency topology graph is traversed based on upstream dependencies to determine the upstream dependency subgraph of a preset number of layers.

[0017] Starting from the initial fault node, the downstream dependency subgraph of the preset number of layers is determined by traversing the business dependency topology graph based on downstream calls.

[0018] The upstream dependency subgraph and the downstream dependency subgraph are concatenated to obtain the fault dependency subgraph.

[0019] In this embodiment, the method of splitting the upstream and downstream, traversing them separately, and then merging them can distinguish between the source of the fault and the spread of the fault's impact, which facilitates subsequent targeted analysis and improves the reliability of fault location.

[0020] Optionally, based on the initial failure time of the initial fault node, the fault dependency subgraph is pruned to obtain the fault propagation chain, including:

[0021] Filter multiple candidate fault nodes in the fault dependency subgraph, where the candidate fault nodes are nodes with fault events;

[0022] Based on the temporal relationship between the candidate fault times of each candidate fault node and the initial fault time of the initial fault node, multiple selected fault nodes are determined from the multiple candidate fault nodes.

[0023] Based on the selected fault nodes, the fault dependency subgraph is cut to obtain the fault propagation chain.

[0024] In this embodiment, nodes can be filtered by temporal relationships to eliminate irrelevant and abnormal nodes from the time dimension, ensuring that the generated fault propagation chain has a smooth temporal logic. Furthermore, the fault dependency subgraph is cut to eliminate invalid and redundant links, compress the scope of fault analysis, and improve the reliability of fault location.

[0025] Optionally, based on the temporal relationship between the candidate failure times of each candidate failure node and the initial failure time of the initial failure node, multiple selected failure nodes are determined from the plurality of candidate failure nodes, including:

[0026] For any candidate fault node, determine whether the candidate fault time of the candidate fault node is within the preset propagation time corresponding to the initial fault time;

[0027] If so, the candidate fault node is determined as the selected fault node to obtain the plurality of selected fault nodes.

[0028] In this embodiment, by using time window threshold filtering, irrelevant isolated faults that occur after a long period of time can be effectively intercepted, avoiding cross-time period anomalies from interfering with the analysis results, which can improve the reliability of fault location and improve fault location efficiency.

[0029] Optionally, determining multiple associated verification sets for multiple alarm pairs corresponding to the fault propagation chain includes:

[0030] Determine the multiple alarm pairs corresponding to the fault propagation chain, and the alarm type of each alarm pair;

[0031] Based on the alarm pairs corresponding to each alarm type, determine the fault window duration corresponding to multiple alarm types;

[0032] Based on the fault window duration corresponding to each alarm type, determine the associated verification set for each alarm pair.

[0033] In this embodiment, matching the fault window duration according to the alarm type can adapt to the different fault propagation rates. Furthermore, limiting alarm pairs through the propagation chain reduces unnecessary calculations and improves the computational efficiency and accuracy of alarm correlation analysis.

[0034] Optionally, based on the alarm pairs corresponding to each alarm type, determine the fault window duration corresponding to multiple alarm types, including:

[0035] For any given alarm type, determine the alarm interval duration for each alarm pair in at least one alarm pair corresponding to that alarm type;

[0036] For any given alarm type, the fault window duration corresponding to the alarm type is determined based on the alarm interval duration of each alarm pair.

[0037] In this embodiment, the window duration of each alarm type is configured differently, which can adapt to the different propagation rates of different fault types, improve the accuracy of subsequent correlation verification, and thus improve the reliability of fault location.

[0038] Optionally, the association verification set includes frequency statistics parameters, probability feature parameters, and significance parameters; based on the fault window duration corresponding to each alarm type, the association verification set for each alarm pair is determined, including:

[0039] Based on the fault window duration corresponding to each alarm type, determine the frequency statistics parameters for each alarm pair;

[0040] Based on the frequency statistics parameters of each alarm pair, determine the probability characteristic parameters of each alarm pair;

[0041] The significance parameter of each alarm pair is determined based on the frequency statistics parameter of each alarm pair and the probability characteristic parameter of each alarm pair.

[0042] In this embodiment, statistics are performed based on the window duration corresponding to each alarm type, which conforms to the fault propagation law and improves the authenticity of the statistical results.

[0043] Optionally, based on the alarm description and the association verification set of each alarm pair, a fault analysis result corresponding to the target alarm event is generated, including:

[0044] Based on the alarm description, obtain the historical fault information corresponding to the alarm event;

[0045] Based on the historical fault information and the associated verification set of each alarm pair, the fault analysis result corresponding to the target alarm event is determined.

[0046] In this embodiment, historical fault information is introduced as empirical data to make up for the blind spots in business scenarios that exist in pure data statistical analysis, so that the root cause determination is closer to the actual operation and maintenance scenario and the fault location accuracy is improved.

[0047] Optionally, based on the historical fault information and the association verification set of each alarm pair, the fault analysis result corresponding to the target alarm event is determined, including:

[0048] Get the prompt word template;

[0049] The prompt word template is updated using the historical fault information and the associated verification set of each alarm pair to obtain decision prompt words;

[0050] The decision prompt words are analyzed and processed by a pre-set large language model to obtain the fault analysis results corresponding to the target alarm event.

[0051] In this embodiment, quantitative statistical verification data is integrated with real historical operation and maintenance failure experience to achieve a combination of data quantitative analysis and domain experience analysis. This can avoid the scenario limitations of single algorithm reasoning and improve the reliability of fault root cause localization.

[0052] Secondly, this application provides a fault location device, including a receiving module, an analysis and processing module, a topology reasoning module, a determining module, and a generating module:

[0053] The receiving module is used to receive target alarm events from the server cluster;

[0054] The analysis and processing module is used to analyze and process the target alarm event to obtain fault task information, which includes alarm description, initial fault node and initial fault time.

[0055] The topology reasoning module is used to perform topology reasoning on the fault task information in the business dependency topology graph of the server cluster based on the initial fault node and the initial fault time, and to construct a fault propagation chain.

[0056] The determining module is used to determine multiple associated verification sets for multiple alarm pairs corresponding to the fault propagation chain;

[0057] The generation module is used to generate fault analysis results corresponding to the target alarm event based on the alarm description and the associated verification set of each alarm pair.

[0058] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0059] The memory stores computer-executed instructions;

[0060] The processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects.

[0061] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0062] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the second aspects.

[0063] The fault location method, apparatus, electronic device, and program product provided in this application can receive target alarm events from a server cluster, analyze and process the target alarm events to obtain fault task information, including alarm description, initial fault node, and initial fault time. Based on the initial fault node and initial fault time, topological reasoning is performed on the fault task information in the service dependency topology graph of the server cluster to construct a fault propagation chain. Multiple association verification sets for multiple alarm pairs corresponding to the fault propagation chain are determined. Based on the alarm description and the association verification sets of each alarm pair, a fault analysis result corresponding to the target alarm event is generated. By performing topological reasoning on the fault task information in the service dependency topology graph of the server cluster based on the initial fault node and initial fault time to construct a fault propagation chain, and then verifying the association of multiple alarm pairs in the fault propagation chain to obtain fault analysis results, this method overcomes the limitations of a single data source and improves the reliability of fault location. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application;

[0066] Figure 2 A flowchart illustrating a fault location method provided in an embodiment of this application;

[0067] Figure 3 A flowchart illustrating another fault location method provided in an embodiment of this application;

[0068] Figure 4 A schematic diagram of the architecture of a fault location method provided in an embodiment of this application;

[0069] Figure 5 A flowchart illustrating the fault location method provided in this application embodiment;

[0070] Figure 6 This is a schematic diagram of the structure of a fault location device provided in an embodiment of this application;

[0071] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0072] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0074] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 This application is applicable to server clusters in complex IT system operation and maintenance scenarios characterized by modern distributed, microservice, and containerized architectures. Specifically, the application scenario includes server cluster 101 and fault analysis device 102, wherein server cluster 101 includes multiple service nodes.

[0075] The fault analysis device 102 can receive target alarm events from the server cluster 101. The fault analysis device 102 can analyze and process the target alarm events to obtain fault task information, which includes alarm description, initial fault node, and initial fault time.

[0076] The fault analysis device 102 can perform topological reasoning on fault task information in the business dependency topology graph of the server cluster based on the initial fault node and the initial fault time, construct a fault propagation chain, determine multiple association verification sets of multiple alarm pairs corresponding to the fault propagation chain, and generate fault analysis results corresponding to the target alarm event based on the alarm description and the association verification sets of each alarm pair.

[0077] In related technologies, when quickly locating the initial fault node in massive, heterogeneous, and noisy alarm data, a model can be built based on the dependencies between indicators to capture the statistical dependencies between data, thereby locating the fault node. However, using only the initial alarm event as the analysis object limits the analysis results to a single data source, resulting in low reliability of fault location.

[0078] The fault location method provided in this application can receive target alarm events from a server cluster, analyze and process the target alarm events to obtain fault task information, which includes alarm description, initial fault node, and initial fault time. Based on the initial fault node and initial fault time, topological reasoning is performed on the fault task information in the business dependency topology graph of the server cluster to construct a fault propagation chain. Multiple association verification sets of multiple alarm pairs corresponding to the fault propagation chain are determined. Based on the alarm description and the association verification sets of each alarm pair, fault analysis results corresponding to the target alarm event are generated.

[0079] In the above process, based on the initial fault node and the initial fault time, topological reasoning is performed on the fault task information in the business dependency topology graph of the server cluster to construct a fault propagation chain. On this basis, the association verification of multiple alarm pairs in the fault propagation chain is completed to obtain the fault analysis results, which breaks through the limitation of a single data source and improves the reliability of fault location.

[0080] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0081] Figure 2 This is a flowchart illustrating a fault location method provided in an embodiment of this application. Please refer to... Figure 2 The method may include:

[0082] S201, Receive target alarm events from the server cluster.

[0083] The execution subject of this application embodiment can be a fault analysis device or a fault location device installed in the fault analysis device. The fault location device can be implemented by software or by a combination of software and hardware.

[0084] It can uniformly connect to various service nodes, middleware, databases and operation and maintenance monitoring platforms within the server cluster, and collect various raw heterogeneous alarm data such as log errors, abnormal indicators, abnormal service calls, and resource overruns generated during cluster operation in real time.

[0085] A unified data access protocol can be used to aggregate alarm data, perform preliminary filtering on scattered and disordered original alarm messages, remove redundant information such as duplicate reports and invalid heartbeats, and select valid events with actual faults as target alarm events to complete the data entry access for the fault analysis process.

[0086] S202. Analyze and process the target alarm events to obtain fault task information.

[0087] Fault task information may include alarm description, initial fault node, and initial fault time.

[0088] Alarm descriptions are used to indicate standardized textual descriptions of faults formed after normalizing and organizing the original alarm content, including information such as the type of abnormality, the content of the error, and the abnormal behavior of the business.

[0089] The initial fault node is used to indicate the unique identifier of the business service entity node that first failed and triggered the alarm from the unstructured alarm data.

[0090] The initial fault time is the standard timestamp corresponding to the first report of the target alarm event and the actual occurrence of the fault.

[0091] It can be executed by the data parsing agent, and can extract algorithms and embed semantic vectorization technology based on the Drain log template to perform structured decomposition of the original heterogeneous target alarm events.

[0092] By using a sliding time window, discrete alarm time sequence aggregation is completed, redundant and invalid fields are removed, and alarm text features, the service node to which the fault belongs (i.e., the initial fault node) and the time when the alarm was first generated are automatically extracted.

[0093] The original unstructured target alarm events are transformed into standardized field data, and the resulting data is organized to generate complete fault task information, including alarm description, initial fault node, and initial fault time.

[0094] S203. Based on the initial fault node and the initial fault time, perform topological reasoning on the fault task information in the business dependency topology graph of the server cluster to construct the fault propagation chain.

[0095] The business dependency topology graph is built on the Configuration Management Database (CMDB). It is a directed acyclic graph (DAG) consisting of various information technology (IT) resource entities as service nodes and service calls and business dependencies as directed edges. It can completely record the upstream and downstream call logic and hierarchical dependency relationships of all service nodes in the cluster.

[0096] The fault propagation chain is used to indicate the order in which a fault spreads from the initial fault node along the business dependency flow, the propagation path, and all related abnormal nodes affected.

[0097] In some embodiments, based on the initial fault node, the fault dependency subgraph corresponding to the target alarm event can be determined in the service dependency topology graph; based on the initial fault time, the fault dependency subgraph can be cut to obtain the fault propagation chain.

[0098] It can be executed by the topology reasoning agent. Using the initial fault node obtained from the parsing as the core base point, the breadth-first search (BFS) algorithm can be used to extract nodes and related links of a specified level range from the global business dependency topology graph, generate a local fault dependency subgraph, and complete the preliminary delineation of the fault investigation scope.

[0099] Using the initial fault time as the timing benchmark, the alarm timestamps of all nodes in the fault dependency subgraph are matched, and path pruning is performed to remove irrelevant nodes and redundant links that have no fault alarms, alarm timing is reversed, or exceed the reasonable fault propagation timing range. Finally, valid links that conform to the fault propagation timing and dependency logic are selected and retained to form a complete fault propagation chain.

[0100] In this embodiment, timing pruning is performed by combining alarm timestamps, which can eliminate nodes that are not related to normal operation and abnormal events that are not caused by the current fault, effectively filtering invalid interference information in the topology and improving the reliability of fault location. In addition, by extracting subgraphs, the scope of fault investigation can be quickly narrowed down without traversing the entire cluster topology, thus improving processing efficiency.

[0101] S204. Determine multiple associated verification sets for multiple alarm pairs corresponding to the fault propagation chain.

[0102] Alarm pairs can be combinations of fault events formed by pairing based on topological dependencies and temporal order within the fault propagation chain. They can include pre-alarm events of preceding fault nodes and post-alarm events of subsequent fault nodes. After a pre-alarm event occurs at a preceding fault node, a post-alarm event occurs at a subsequent fault node.

[0103] The association validation set can include frequency statistics parameters, probability feature parameters, and significance parameters.

[0104] The frequency statistics parameter is used to indicate the total number of co-occurrences of alarm events within the same dynamic time window, the number of first-time secondary alarms, the frequency of independent occurrence of the two types of alarms, and the timing statistics results such as the event occurrence time delay.

[0105] The probability feature parameters are used to indicate the forward conditional transition probability and the reverse tracing probability after Laplace smoothing correction, as well as the upper and lower confidence limits of the probability calculated by Wilson intervals, characterizing the credibility of causal transmission between alarms.

[0106] The significance parameter indicates the false discovery rate obtained after the original hypothesis test p-value is obtained through binomial test and corrected by the Benjamin-Hochberg (FDR-BH) multiple test. The value and significance determination results distinguish between genuine causal relationships and random, irrelevant alarms.

[0107] In some embodiments, multiple alarm pairs corresponding to the fault propagation chain and the alarm type of each alarm pair are determined; the fault window duration corresponding to the multiple alarm types is determined based on the alarm pairs corresponding to each alarm type; and the associated verification set of each alarm pair is determined based on the fault window duration corresponding to each alarm type.

[0108] This can be performed by a statistical analysis agent. Based on the established fault propagation chain, it can batch split and combine all compliant alarm pairs according to the topological flow and timing rules of upstream and downstream, and classify the fault types of all alarm pairs.

[0109] The estimate_window algorithm can be called to count the historical occurrence intervals of similar alarms, and the 90th percentile of the interval data can be used as the base duration. At the same time, a hard upper limit of 30 minutes can be set to adaptively generate a dynamic fault window duration for different alarm types.

[0110] Within the corresponding fault window duration, frequency statistics are performed using both Any_within_window and First_subsequent modes. Combining smoothing algorithms, interval estimation methods, hypothesis testing, and multiple test correction processes, frequency statistics parameters, probability feature parameters, and significance parameters are calculated and generated layer by layer. Finally, these parameters are integrated and summarized to form the association verification set for each alarm pair.

[0111] In this embodiment, matching the fault window duration according to the alarm type can adapt to the different fault propagation rates. Furthermore, limiting alarm pairs through the propagation chain reduces unnecessary calculations and improves the computational efficiency and accuracy of alarm correlation analysis.

[0112] S205. Based on the alarm description and the associated verification set of each alarm pair, generate the fault analysis results corresponding to the target alarm event.

[0113] The fault analysis results are used to indicate the true root cause of the target alarm event failure, the complete propagation path, the scope of the abnormal impact, the basis for causal relationship, and the corresponding operation and maintenance handling plan.

[0114] Fault analysis results can include root cause conclusions, fault propagation paths, multidimensional evidence chains, scope of anomaly impact, fault occurrence sequence, and standardized repair recommendations.

[0115] In some embodiments, historical fault information corresponding to the alarm event can be obtained based on the alarm description; and the fault analysis result corresponding to the target alarm event can be determined based on the historical fault information and the association verification set of each alarm pair.

[0116] Based on standardized alarm descriptions, retrieval query vectors can be constructed. In a vector knowledge base containing pre-stored historical fault cases, operation and maintenance manuals, and troubleshooting experience, Top-K similar historical fault information can be retrieved through cosine similarity matching.

[0117] The retrieved historical fault handling experience and root cause patterns of similar faults are integrated and analyzed with the statistical frequency, causal probability, and significance verification conclusions of the correlation verification set of each alarm pair.

[0118] It can integrate topology propagation logic, temporal correlation evidence, statistical causal basis, and industry operation and maintenance experience to eliminate interfering abnormal events, identify the core fault causes, and finally, organize and output complete and applicable fault analysis results.

[0119] In this embodiment, historical fault information is introduced as empirical data to make up for the blind spots in business scenarios that exist in pure data statistical analysis, so that the root cause determination is closer to the actual operation and maintenance scenario and the fault location accuracy is improved.

[0120] The root cause reasoning agent can integrate the decision-making process and work in conjunction with the knowledge retrieval agent to complete the task.

[0121] The decision-making hub, built on the ReAct (Reasoning and Action Execution) model, refers to an intelligent fault decision-making core architecture centered on a root cause reasoning agent. This architecture first autonomously analyzes the inherent causal relationships of faults logically, then, as needed, coordinates various functional agents to perform practical actions such as data retrieval, parameter calculation, and experience searching. This achieves bidirectional linkage between reasoning and business execution, iteratively refining the judgment conclusions. For details, please refer to [link to relevant documentation]. Figure 4 .

[0122] The fault location method provided in this application can receive target alarm events from a server cluster, analyze and process the target alarm events to obtain fault task information, including alarm description, initial fault node, and initial fault time. Based on the initial fault node and initial fault time, topological reasoning is performed on the fault task information in the service dependency topology graph of the server cluster to construct a fault propagation chain. Multiple association verification sets of multiple alarm pairs corresponding to the fault propagation chain are determined. Based on the alarm description and the association verification sets of each alarm pair, a fault analysis result corresponding to the target alarm event is generated. In the above process, based on the initial fault node and initial fault time, topological reasoning is performed on the fault task information in the service dependency topology graph of the server cluster to construct a fault propagation chain. On this basis, the association verification of multiple alarm pairs in the fault propagation chain is completed to obtain the fault analysis result, breaking through the limitation of a single data source and improving the reliability of fault location.

[0123] Figure 3 This is a flowchart illustrating another fault location method provided in an embodiment of this application. Please refer to... Figure 3 The method may include:

[0124] S301, Receive target alarm events from the server cluster.

[0125] S302. Analyze and process the target alarm events to obtain fault task information.

[0126] The execution process of S301-S302 can be found in the execution process of S201-S202, and will not be repeated here.

[0127] S303. Based on the initial fault node, determine the fault dependency subgraph corresponding to the target alarm event in the business dependency topology graph.

[0128] The fault dependency subgraph is used to indicate the set of local nodes with business calls and resource dependencies within a specified range above and below the initial fault node, as well as the complete directed links between nodes, defining the topological boundary range that needs to be investigated for this fault.

[0129] In some embodiments, starting from the initial fault node, the upstream dependency subgraph with a preset number of layers is determined by traversing the business dependency topology graph based on upstream dependencies; starting from the initial fault node, the downstream dependency subgraph with a preset number of layers is determined by traversing the business dependency topology graph based on downstream calls; and the upstream dependency subgraph and the downstream dependency subgraph are concatenated to obtain the fault dependency subgraph.

[0130] Traversal operations can be performed on the global business dependency topology DAG graph built based on CMDB.

[0131] The initial fault node obtained from the analysis is locked as the traversal origin. The breadth-first search algorithm is used to traverse upwards layer by layer, tracing back all upstream nodes that provide services and support the operation of the node. The topological range that reaches the preset upper limit of the level is extracted to form an upstream dependency subgraph.

[0132] Then, taking the same initial fault node as the origin, traverse the downstream business nodes that the node calls out to and directly affects layer by layer, and extract the same preset level range to form a downstream dependency subgraph.

[0133] Finally, duplicate and overlapping nodes and common links in the two subgraphs are removed to complete the seamless splicing of the upper and lower topologies, ultimately forming a fault dependency subgraph with complete scope and controllable hierarchy.

[0134] For example, assuming the preset traversal depth is 2 layers, and the initial fault node is the order service node, traverse upwards through 2 layers of upstream dependencies: Layer 1 upstream consists of product service and user service, and Layer 2 upstream consists of inventory service and account authentication service, combining them to obtain the upstream dependency subgraph; traverse downwards through 2 layers of downstream calls: Layer 1 downstream consists of payment service, and Layer 2 downstream consists of accounting and settlement service, combining them to obtain the downstream dependency subgraph; combine the upper and lower subgraphs to obtain the final complete fault dependency subgraph with the order service as the core, covering all related nodes and call chains in the two layers of upstream and downstream.

[0135] In this embodiment, the method of splitting the upstream and downstream processes, traversing them separately, and then merging them can distinguish between the source of the fault and the spread of its impact, facilitating subsequent targeted analysis and improving the reliability of fault location. Furthermore, the hierarchical bidirectional traversal method allows for customizable investigation levels, preventing both overly narrow investigation scopes from missing related fault points and overly broad scopes from introducing massive amounts of irrelevant topology data.

[0136] S304. Based on the initial fault time, the fault dependency subgraph is cut to obtain the fault propagation chain.

[0137] In some embodiments, multiple candidate fault nodes are screened in the fault dependency subgraph; based on the temporal relationship between the candidate fault time of each candidate fault node and the initial fault time of the initial fault node, multiple selected fault nodes are determined among the multiple candidate fault nodes; based on the multiple selected fault nodes, the fault dependency subgraph is cut to obtain the fault propagation chain.

[0138] Candidate faulty nodes are nodes that have experienced fault events.

[0139] Selected fault nodes are valid abnormal nodes whose fault occurrence time conforms to the fault propagation time sequence and falls within a reasonable propagation time range, and have a temporal linkage relationship with the initial fault.

[0140] From the fault dependency subgraph, all nodes that have generated alarms or have abnormal events are selected in batches as candidate fault nodes.

[0141] Using the fault occurrence time of the initial fault node as the timeline benchmark, the abnormal occurrence times of all candidate fault nodes are compared one by one. Based on the business logic of faults being transmitted from upstream to downstream in a step-by-step delay, nodes with reversed alarm timing or occurrence times that deviate from the fault transmission pattern are filtered out.

[0142] Valid nodes with matching time sequences are retained, while blank nodes and redundant connection edges without alarms or time sequences in the subgraph are deleted to complete the topology pruning and ultimately form a fault propagation chain with smooth time sequences and effective links.

[0143] For example, assume the initial fault node is the order service node, and the initial fault time is 10:00; the candidate fault nodes in the fault dependency subgraph include the product service node, payment service node, user service node, and accounting service node.

[0144] Among them, the alarm time of the user service node is 9:50, which is earlier than the initial fault time, and the node is directly removed if the time sequence is reversed; the alarm time of the product service node is 10:02, the alarm time of the payment service node is 10:05, and the alarm time of the accounting service node is 10:08, and the alarm time sequence is lagging behind, so they are determined to be selected fault nodes; irrelevant nodes and links are cut off and removed to form a fault propagation chain of order service, product service, payment service and accounting service.

[0145] In this embodiment, nodes can be filtered by temporal relationships to conform to the forward propagation of faults. Irrelevant and abnormal nodes are eliminated from the time dimension to ensure that the temporal logic of the generated fault propagation chain is smooth. Furthermore, the fault dependency subgraph is cut to eliminate invalid and redundant links, further compressing the scope of fault analysis and improving the reliability of fault location.

[0146] Specifically, when determining the selected fault node, for any candidate fault node, it can be determined whether the candidate fault time of the candidate fault node is within the preset propagation time corresponding to the initial fault time; if so, the candidate fault node is determined as the selected fault node, so as to obtain multiple selected fault nodes.

[0147] The preset propagation duration is a reasonable range of fault propagation time defined by taking the initial fault time as a reference point and combining the normal call latency of business services.

[0148] For example, the preset propagation duration can be set to a maximum limit of 30 minutes.

[0149] In some embodiments, the propagation duration corresponding to the target time period corresponding to the initial fault time can be determined as the preset propagation duration.

[0150] A day can be divided into multiple business runtime segments. For any given business runtime segment, the greater the business traffic and the higher the service call pressure, the longer it takes for the fault to propagate step by step, and the longer its corresponding propagation time will be.

[0151] For example, the window duration can be appropriately increased during peak hours and high concurrency periods, while the window duration can be shortened during off-peak hours to match the actual business transmission rhythm.

[0152] By dynamically adapting the window size to different business time periods, the accuracy of time-series filtering can be further improved, thereby enhancing the reliability of fault location.

[0153] Iterate through each candidate fault node, extract the candidate fault time when the fault occurred, and determine whether the time falls within the preset propagation time interval. If it is within the interval, it means that the node's abnormality is caused by the chain propagation after the fault occurred, and it is determined to be a selected fault node. If it is outside the time window, it is determined to be an independent abnormal event that is not related to the fault and is directly removed. In this way, all valid selected fault nodes are selected in batches.

[0154] For example, assuming the initial fault time is 10:00 and the preset propagation time is 10:00~10:30; candidate node A has a fault time of 10:10, is within the window, and is selected as the fault node; candidate node B has a fault time of 10:35, exceeds the 30-minute time limit, is not within the window, and is directly eliminated.

[0155] In this embodiment, by using time window threshold filtering, irrelevant isolated faults that occur after a long period of time can be effectively intercepted, avoiding cross-time period anomalies from interfering with the analysis results, which can improve the reliability of fault location and improve fault location efficiency.

[0156] S305. Determine the multiple alarm pairs corresponding to the fault propagation chain, and the alarm type of each alarm pair.

[0157] Event pairing can be performed along the already trimmed and logically coherent fault propagation chain, following the topological directed dependency relationship and the temporal sequence.

[0158] Alarm events generated by earlier nodes in the propagation chain are used as pre-alarm A, and alarm events from later nodes in the chain that have business transmission relationships are used as post-alarm B. Compliant alarm pairs are generated in batches by combining them.

[0159] Based on the alarm anomaly attributes, fault triggering source, and business anomaly dimension, each generated alarm pair is uniformly classified and divided to distinguish different alarm types such as resource alarms, call anomaly alarms, database anomaly alarms, and network timeout alarms, thus completing the alarm pair assembly and type tag binding.

[0160] S306. Determine the fault window duration corresponding to multiple alarm types based on the alarm pairs corresponding to each alarm type.

[0161] In some embodiments, for any alarm type, the alarm interval duration of each alarm pair in at least one alarm pair corresponding to the alarm type is determined; for any alarm type, the fault window duration corresponding to the alarm type is determined based on the alarm interval duration of each alarm pair.

[0162] This can be achieved by the statistical analysis agent executing the estimate_window algorithm.

[0163] For any alarm type, calculate the time difference between the occurrence time of the preceding alarm and the occurrence time of the following alarm for all alarm pairs to obtain the total alarm interval duration for that alarm type.

[0164] Statistical calculations are performed on all alarm interval durations under this alarm type, and the 90th percentile Q90 is taken as the base window duration. This base window duration can be determined as the fault window duration for this alarm type.

[0165] In this embodiment, the window duration of each alarm type is configured differently, which can adapt to the different propagation rates of different fault types, improve the accuracy of subsequent correlation verification, and thus improve the reliability of fault location.

[0166] In some embodiments, a threshold constraint rule can be executed: if the calculated basic window duration is less than the preset window duration, the basic window duration is directly determined as the fault window duration of the corresponding alarm type; if the basic window duration is greater than or equal to the preset window duration, the preset window duration is determined as the fault window duration of the alarm type.

[0167] For example, the preset window duration can be 30 minutes.

[0168] In this embodiment, setting a hard upper limit for the preset window duration can effectively avoid redundant data influx caused by excessively long time intervals, reduce irrelevant alarms, and lower data analysis noise.

[0169] For example, suppose the alarm interval durations of multiple alarm pairs corresponding to the service call timeout class are 20s, 35s, 50s, 1min, and 1.5min respectively; determine the 90th percentile of this set of duration data, and obtain the basic window duration of 55 seconds; since the basic window duration is less than the upper limit of 30 minutes, determine the fault window duration corresponding to the service call timeout class alarm as 55 seconds.

[0170] S307. Determine the associated verification set for each alarm pair based on the fault window duration corresponding to each alarm type.

[0171] In some embodiments, the frequency statistics parameter of each alarm pair is determined based on the fault window duration corresponding to each alarm type; the probability characteristic parameter of each alarm pair is determined based on the frequency statistics parameter of each alarm pair; and the significance parameter of each alarm pair is determined based on the frequency statistics parameter of each alarm pair and the probability characteristic parameter of each alarm pair.

[0172] The frequency statistics parameters may include the total number of independent occurrences of the preceding fault node, the total number of independent occurrences of the following fault node, the total number of times the following fault node co-occurs after the preceding fault node is triggered within the fault window duration, the number of times the following fault node first appears after the preceding fault node is triggered within the fault window duration, and the average time delay difference between the preceding and following fault nodes.

[0173] The probability feature parameters may include the forward conditional probability P(B|A) after Laplace smoothing correction, the reverse source probability P(A|B), and the lower and upper confidence limits of the probability calculated from the Wilson interval.

[0174] Significance parameters may include the original p-value obtained from the binomial test and the p-value corrected by the FDR-BH multiple test. The value and alarm result determine whether there is a strong statistical correlation.

[0175] Based on the fault window duration of the alarm for the corresponding alarm type, data statistics are completed within the time window range, and various frequency and latency-related data are obtained in sequence to generate complete frequency statistics parameters.

[0176] Based on the frequency statistics parameter, a Laplace smoothing mechanism is introduced to eliminate the probability failure problem caused by zero frequency, calculate the bidirectional causal transition probability, and combine the sample size to solve the probability confidence interval to generate probability feature parameters.

[0177] Finally, based on the frequency statistics and probability characteristic parameters, a binomial test was used to complete the independence hypothesis test and obtain the original P-value. Then, the FDR-BH algorithm was used to correct the multiple test error for batch alarms, resulting in the corrected value. Values ​​and significance determination results.

[0178] The three types of parameters are integrated to form a single alarm pair with a complete set of associated verification.

[0179] For example, taking the alarm pair (service connection timeout A, interface call failure B) as an example, statistics are completed based on the fault window duration of the corresponding alarm type: A occurred 80 times, B occurred 56 times in the window following A, and B occurred 49 times as the first occurrence, forming the frequency statistics parameter; substituting into the formula and calculating with Laplace smoothing, P(B|A) = 0.68 is obtained, and the corresponding confidence interval is calculated as [0.61, 0.75], completing the solution of the probability feature parameter; a binomial test is performed to obtain the original P value, which is then corrected by FDR-BH to obtain the result. A value less than 0.05 is considered a significant strong association, and a significance parameter is generated.

[0180] In this embodiment, statistics are performed based on the window duration corresponding to each alarm type, which conforms to the fault propagation law and improves the authenticity of the statistical results.

[0181] Meanwhile, the introduction of smoothing algorithms and interval estimation effectively solves the problem of probability calculation distortion in small sample scenarios and improves the reliability of probability reference. By constructing an association verification set with multi-dimensional parameters, cross-verification is completed from multiple perspectives such as frequency, probability, and statistical significance, making the basis for judging the causal relationship of alarms more reliable, thereby improving the reliability of fault location.

[0182] S308. Based on the alarm description, obtain the historical fault information corresponding to the alarm event.

[0183] Semantic extraction and feature condensation can be performed on alarm descriptions to obtain semantic features; the extracted semantic features are transformed into retrieval vectors, and similarity matching retrieval is performed in the knowledge base. K historical fault data with matching degree are selected and identified as historical fault information.

[0184] Semantic features include fault / anomaly type, error keywords, business scenario, and anomaly behavior.

[0185] The knowledge base stores past similar failure cases, operation and maintenance troubleshooting experience, classic failure root causes, standard handling procedures, and historical analysis conclusions for corresponding abnormal scenarios.

[0186] The cosine similarity ratio can be used to filter the top-K historical fault data with the highest matching degree with the retrieved vector in the knowledge base, where K is an integer greater than or equal to 1.

[0187] By accurately retrieving historical fault information from similar scenarios, we can provide data support at the experience level for subsequent data fusion and statistical conclusions, industry operation and maintenance experience, and fault analysis results.

[0188] S309. Based on historical fault information and the correlation verification set of each alarm pair, determine the fault analysis result corresponding to the target alarm event.

[0189] In some embodiments, a prompt word template is obtained; the prompt word template is updated using historical fault information and the association verification set of each alarm pair to obtain decision prompt words; the decision prompt words are analyzed and processed using a preset large language model to obtain the fault analysis results corresponding to the target alarm event.

[0190] The retrieved text fragments are used as context and injected into the prompt word template of the pre-defined large language model for decision-making.

[0191] The default large language model can be a large language model (LLM), or you can choose a general open source large model, a large model specifically tuned for the operation and maintenance field, or a large model for intelligent analysis in a vertical industry.

[0192] The prompt word template is a pre-defined, standardized, structured reasoning framework with built-in fault analysis logic, analysis dimensions, and output format requirements. It also reserves data fields and experience fields for use in standardizing the reasoning direction and output paradigm of large models.

[0193] The decision prompt is a complete instruction text that is formed by filling in historical fault case experience, alarm frequency statistics, causal probability values, significance verification conclusions, fault propagation links, and other actual business data on the prompt template. It has complete analytical basis and can be directly input into large models for intelligent reasoning.

[0194] Retrieve the unified fault analysis prompt word template; fill the reserved spaces in the prompt word template with the historical fault information corresponding to the target alarm event, as well as the correlation verification set, fault propagation chain, and time-series correlation characteristics of each alarm pair, and complete the template content update to generate exclusive decision prompt words with complete evidence.

[0195] The integrated decision prompts are input into a pre-defined large language model. Relying on the powerful semantic understanding, logical reasoning, and experience induction capabilities of the large model, and combined with professional knowledge in the operation and maintenance field, a comprehensive analysis is completed.

[0196] The pre-set large model determines the core root cause of the fault, the fault propagation path, the fault impact range, the risk level, and the standardized operation and maintenance repair plan according to the pre-set output format, and outputs the fault analysis results corresponding to the standardized target alarm events.

[0197] In this embodiment, quantitative statistical verification data is integrated with real historical operation and maintenance failure experience to combine data quantitative analysis and domain experience analysis. This avoids the scenario limitations of single-algorithm reasoning and improves the reliability of fault root cause localization. Furthermore, decision prompts can be constructed using template filling to unify reasoning logic and output specifications, avoiding disorganized and illogical output from large models and ensuring consistent and clear analysis results.

[0198] The fault location method provided in this application can automate the following processes: alarm data access, structured information extraction, local topology delineation, time-series link simplification, alarm event classification and pairing, adaptive time window configuration, multi-dimensional alarm correlation quantitative verification, historical operation and maintenance experience matching and retrieval, and large-scale model fusion intelligent reasoning. By performing layer-by-layer filtering and verification from data source standardization to intelligent decision output, it solves the problems of misjudgment and omission, excessive investigation scope, and low analysis efficiency caused by relying solely on single alarm data and fixed analysis rules. It achieves more precise fault investigation scope, time-series fault transmission analysis, data-driven alarm causal verification, and intelligent root cause decision-making experience, effectively improving the accuracy, real-time performance, and reliability of fault location in complex cluster scenarios. At the same time, it reduces the workload of manual operation and maintenance troubleshooting, shortens the online fault emergency response time, and comprehensively improves the overall operation and maintenance management efficiency of server clusters.

[0199] Figure 4 This is a schematic diagram illustrating the architecture of a fault location method provided in an embodiment of this application. Please refer to... Figure 4 The architecture is divided into an external environment data flow layer and a multi-agent collaboration layer, which realizes the processing from alarm event access to fault analysis result output.

[0200] The external environment data flow layer provides basic data support for the entire system, including monitoring data / log streams, a CMDB asset repository, and a knowledge base (vector database). The monitoring data / log streams are used to collect real-time operational metrics, log errors, and alarm events from the server cluster. The CMDB asset repository stores the business dependency topology information of all service nodes within the cluster, providing a basis for dependency reasoning. The knowledge base (vector database) stores historical fault cases, operational troubleshooting experience, and standard handling procedures, structured and stored in vector form to support subsequent semantic retrieval and experience matching.

[0201] The multi-agent collaboration layer uses the root cause reasoning agent built on the ReAct model as the decision-making center, linking four functional agents—data parsing, topological reasoning, statistical analysis, and knowledge retrieval—to work together.

[0202] In the data parsing agent, alarm events are received from the data stream of the external environment. Unstructured alarm content is parsed in a standardized manner, and key information such as alarm description, initial fault node, and initial fault time are extracted to generate structured fault task information and feed it back to the root cause reasoning agent.

[0203] In the topology reasoning agent, the dependency query command is received from the root cause reasoning agent. Based on the business dependency topology graph of the CMDB asset library, the fault dependency subgraph of the upstream and downstream specified levels is extracted with the initial fault node as the center. The timing pruning is completed in combination with the initial fault time to construct the fault propagation chain and feed the fault dependency subgraph back to the root cause reasoning agent.

[0204] In the statistical analysis agent, the causal verification instruction is received from the root cause reasoning agent. Based on the alarm pairs in the fault propagation chain, the time-series co-occurrence statistics are completed using a dynamic time window. The association verification set is generated through methods such as Laplace smoothing, Wilson interval estimation, and FDR-BH correction, and then fed back to the root cause reasoning agent.

[0205] In the knowledge retrieval agent, it receives case retrieval instructions from the root cause reasoning agent, matches similar historical fault information in the knowledge base based on alarm descriptions, and feeds back similar cases and handling experience to the root cause reasoning agent.

[0206] After receiving fault task information, fault dependency subgraphs, association validation sets and historical fault information returned by various agents, the root cause reasoning agent completes multi-source data fusion and finally generates standardized fault analysis results. The result generation module outputs the root cause of the fault, propagation path, scope of impact and repair suggestions.

[0207] Figure 5 This is a flowchart illustrating a fault location method provided in an embodiment of this application. Please refer to... Figure 5 It receives and parses alarm events, generating fault event information that includes alarm description, initial fault node, and initial fault time; at the same time, based on the business dependency topology graph, it extracts upstream and downstream related links with the initial fault node as the core, generating a fault dependency subgraph.

[0208] Then, based on the initial fault time in the fault event information, the fault dependency subgraph is pruned temporally to remove irrelevant nodes and links that are out of order or exceed the reasonable propagation window, and a fault propagation chain is constructed. Then, along the fault propagation chain, multiple alarm pairs are generated in batches according to the topology flow and temporal order rules.

[0209] Next, for each alarm pair, based on the fault window duration corresponding to its alarm type, frequency statistics, probability feature calculation and significance verification are completed to generate an association verification set containing frequency statistics parameters, probability feature parameters and significance parameters; at the same time, historical fault information is retrieved according to the alarm description to obtain operation and maintenance experience and handling cases of similar scenarios.

[0210] Finally, the fault event information, associated validation set and historical fault information are integrated and analyzed. By combining topological dependencies, temporal transmission patterns, statistical causal evidence and historical operation and maintenance experience, the final fault analysis results are generated, which include the root cause of the fault, the propagation path, the scope of impact and the repair suggestions.

[0211] Figure 6 This is a schematic diagram of a fault location device provided in an embodiment of this application. Please refer to... Figure 6The fault location device 600 may include a receiving module 601, an analysis and processing module 602, a topology reasoning module 603, a determination module 604, and a generation module 605.

[0212] The receiving module 601 is used to receive target alarm events from the server cluster;

[0213] The analysis and processing module 602 is used to analyze and process the target alarm event to obtain fault task information, which includes alarm description, initial fault node and initial fault time.

[0214] The topology reasoning module 603 is used to perform topology reasoning on fault task information in the business dependency topology graph of the server cluster based on the initial fault node and the initial fault time, and to build a fault propagation chain.

[0215] The determination module 604 is used to determine multiple associated verification sets corresponding to multiple alarm pairs in the fault propagation chain;

[0216] The generation module 605 is used to generate fault analysis results corresponding to the target alarm event based on the alarm description and the associated verification set of each alarm pair.

[0217] Optionally, the topology inference module 603 is specifically used for:

[0218] Based on the initial fault node, determine the fault dependency subgraph corresponding to the target alarm event in the business dependency topology graph;

[0219] Based on the initial failure time of the initial failure node, the failure dependency subgraph is cut to obtain the failure propagation chain.

[0220] Optionally, the topology inference module 603 is specifically used for:

[0221] Starting with the initial fault node, the business dependency topology graph is traversed based on upstream dependencies to determine the upstream dependency subgraph of a preset number of layers.

[0222] Starting from the initial fault node, the downstream dependency topology is traversed based on downstream calls to determine the downstream dependency subgraph with a preset number of layers.

[0223] By concatenating the upstream dependency subgraph and the downstream dependency subgraph, a fault dependency subgraph is obtained.

[0224] Optionally, the topology inference module 603 is specifically used for:

[0225] Filter multiple candidate fault nodes in the fault dependency subgraph. Candidate fault nodes are nodes that have fault events.

[0226] Based on the temporal relationship between the candidate fault time of each candidate fault node and the initial fault time of the initial fault node, multiple selected fault nodes are determined from among the multiple candidate fault nodes.

[0227] Based on multiple selected fault nodes, the fault dependency subgraph is cut to obtain the fault propagation chain.

[0228] Optionally, the topology inference module 603 is specifically used for:

[0229] For any candidate fault node, determine whether the candidate fault time of the candidate fault node is within the preset propagation time corresponding to the initial fault time;

[0230] If so, the candidate fault node is selected as the selected fault node, thus obtaining multiple selected fault nodes.

[0231] Optionally, module 604 is specifically used for:

[0232] Identify the multiple alarm pairs corresponding to the fault propagation chain, and the alarm type of each alarm pair;

[0233] Based on the alarm pairs corresponding to each alarm type, determine the fault window duration corresponding to multiple alarm types;

[0234] Based on the fault window duration corresponding to each alarm type, determine the associated verification set for each alarm pair.

[0235] Optionally, module 604 is specifically used for:

[0236] For any given alarm type, determine the alarm interval duration for each alarm pair in at least one alarm pair corresponding to the alarm type;

[0237] For any given alarm type, determine the corresponding fault window duration based on the alarm interval duration of each alarm pair.

[0238] Optionally, the association validation set includes frequency statistics parameters, probability feature parameters, and significance parameters; the determination module 604 is specifically used for:

[0239] Based on the fault window duration corresponding to each alarm type, determine the frequency statistics parameters for each alarm pair;

[0240] Based on the frequency statistics parameters of each alarm pair, determine the probability characteristic parameters of each alarm pair;

[0241] Based on the frequency statistics parameters and probability characteristic parameters of each alarm pair, the significance parameters of each alarm pair are determined.

[0242] Optionally, the generation module 605 is specifically used for:

[0243] Based on the alarm description, obtain the historical fault information corresponding to the alarm event;

[0244] Based on historical fault information and the correlation verification set of each alarm pair, the fault analysis results corresponding to the target alarm event are determined.

[0245] Optionally, the generation module 605 is specifically used for:

[0246] Get the prompt word template;

[0247] By updating the prompt word template using historical fault information and the associated verification set of each alarm pair, decision prompt words are obtained.

[0248] By analyzing and processing decision prompt words using a pre-set large language model, the fault analysis results corresponding to the target alarm event are obtained.

[0249] The fault location and detection device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0250] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 7 The electronic device 700 may include a processor 701 and a memory 702 communicatively connected to the processor 701. Exemplarily, the processor 701 and the memory 702 are interconnected via a bus 703.

[0251] Memory 702 stores instructions executed by the computer;

[0252] The processor 701 executes computer execution instructions stored in the memory 702, causing the processor 701 to perform the fault location and detection method as shown in the above method embodiment.

[0253] Accordingly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the fault location and detection method of the above-described method embodiments.

[0254] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the fault location and detection method shown in the above method embodiments.

[0255] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0256] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0257] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0258] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0259] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0260] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0261] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0262] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0263] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0264] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A fault location method, characterized in that, include: Receive target alarm events from the server cluster, analyze and process the target alarm events to obtain fault task information, which includes alarm description, initial fault node and initial fault time. Based on the initial fault node and the initial fault time, topological reasoning is performed on the fault task information in the service dependency topology graph of the server cluster to construct a fault propagation chain; Determine multiple associated verification sets corresponding to multiple alarm pairs in the fault propagation chain; Based on the alarm description and the associated verification set of each alarm pair, the fault analysis result corresponding to the target alarm event is generated.

2. The method according to claim 1, characterized in that, Based on the initial fault node and the initial fault time, topological reasoning is performed on the fault task information in the service dependency topology graph of the server cluster to construct a fault propagation chain, including: Based on the initial fault node, determine the fault dependency subgraph corresponding to the target alarm event in the service dependency topology graph; Based on the initial fault time of the initial fault node, the fault dependency subgraph is cut to obtain the fault propagation chain.

3. The method according to claim 2, characterized in that, Based on the initial fault node, in the service dependency topology graph, determine the fault dependency subgraph corresponding to the target alarm event, including: Starting from the initial fault node, the business dependency topology graph is traversed based on upstream dependencies to determine the upstream dependency subgraph of a preset number of layers. Starting from the initial fault node, the downstream dependency subgraph of the preset number of layers is determined by traversing the business dependency topology graph based on downstream calls. The upstream dependency subgraph and the downstream dependency subgraph are concatenated to obtain the fault dependency subgraph.

4. The method according to claim 2, characterized in that, Based on the initial failure time of the initial fault node, the fault dependency subgraph is pruned to obtain the fault propagation chain, including: Filter multiple candidate fault nodes in the fault dependency subgraph, where the candidate fault nodes are nodes with fault events; Based on the temporal relationship between the candidate fault times of each candidate fault node and the initial fault time of the initial fault node, multiple selected fault nodes are determined from the multiple candidate fault nodes. Based on the selected fault nodes, the fault dependency subgraph is cut to obtain the fault propagation chain.

5. The method according to claim 4, characterized in that, Based on the temporal relationship between the candidate failure times of each candidate failure node and the initial failure time of the initial failure node, multiple selected failure nodes are determined from the plurality of candidate failure nodes, including: For any candidate fault node, determine whether the candidate fault time of the candidate fault node is within the preset propagation time corresponding to the initial fault time; If so, the candidate fault node is determined as the selected fault node to obtain the plurality of selected fault nodes.

6. The method according to any one of claims 1-5, characterized in that, Determine multiple associated verification sets corresponding to multiple alarm pairs in the fault propagation chain, including: Determine the multiple alarm pairs corresponding to the fault propagation chain, and the alarm type of each alarm pair; Based on the alarm pairs corresponding to each alarm type, determine the fault window duration corresponding to multiple alarm types; Based on the fault window duration corresponding to each alarm type, determine the associated verification set for each alarm pair.

7. The method according to claim 6, characterized in that, Based on the alarm pairs corresponding to each alarm type, determine the fault window duration for multiple alarm types, including: For any given alarm type, determine the alarm interval duration for each alarm pair in at least one alarm pair corresponding to that alarm type; For any given alarm type, the fault window duration corresponding to the alarm type is determined based on the alarm interval duration of each alarm pair.

8. The method according to claim 6, characterized in that, The association verification set includes frequency statistics parameters, probability feature parameters, and significance parameters; based on the fault window duration corresponding to each alarm type, the association verification set for each alarm pair is determined, including: Based on the fault window duration corresponding to each alarm type, determine the frequency statistics parameters for each alarm pair; Based on the frequency statistics parameters of each alarm pair, determine the probability characteristic parameters of each alarm pair; The significance parameter of each alarm pair is determined based on the frequency statistics parameter and the probability characteristic parameter of each alarm pair.

9. The method according to any one of claims 1-5, characterized in that, Based on the alarm description and the correlation verification set of each alarm pair, generate the fault analysis results corresponding to the target alarm event, including: Based on the alarm description, obtain the historical fault information corresponding to the alarm event; Based on the historical fault information and the associated verification set of each alarm pair, the fault analysis result corresponding to the target alarm event is determined.

10. The method according to claim 9, characterized in that, Based on the historical fault information and the correlation verification set of each alarm pair, the fault analysis result corresponding to the target alarm event is determined, including: Get the prompt word template; The prompt word template is updated using the historical fault information and the associated verification set of each alarm pair to obtain decision prompt words; The decision prompt words are analyzed and processed by a pre-set large language model to obtain the fault analysis results corresponding to the target alarm event.

11. A fault location device, characterized in that, It includes a receiving module, an analysis and processing module, a topology reasoning module, a determination module, and a generation module: The receiving module is used to receive target alarm events from the server cluster; The analysis and processing module is used to analyze and process the target alarm event to obtain fault task information, which includes alarm description, initial fault node and initial fault time. The topology reasoning module is used to perform topology reasoning on the fault task information in the business dependency topology graph of the server cluster based on the initial fault node and the initial fault time, and to construct a fault propagation chain. The determining module is used to determine multiple associated verification sets for multiple alarm pairs corresponding to the fault propagation chain; The generation module is used to generate fault analysis results corresponding to the target alarm event based on the alarm description and the associated verification set of each alarm pair.

12. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

Images