A method and system for failure root cause localization for server clusters

Abstract

The application discloses a kind of methods and systems for fault root cause positioning of server cluster, it is related to network fault diagnosis technical field;The present application realizes microservice call chain, container index, nanosecond level synchronous collection of physical node and network data by precise time protocol, constructs consistent data set after entity association and standardization processing;Based on this, service-resource topology graph is dynamically constructed, and the inter-service call edge weight model is innovatively designed: the real-time load factor is fused, and the historical fault index attenuation and processed by Sigmoid function are superposed, and the weight is periodically updated to accurately quantify the influence strength between nodes;Topology graph is converted into Bayesian network;When fault occurs, three-stage pipeline is used to compress alarm, frequent item set is mined by bitmap index and parallel FP-Growth, and strong causal association item set is screened by combining topology edge weight and KL divergence;Strong causal alarm is taken as evidence, and probabilistic root cause ranking is output by reverse random walk sampling, to realize high-precision positioning of complex distributed system fault.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network fault diagnosis, and particularly relates to a fault root cause positioning method and system for a server cluster. BACKGROUND

[0002] With the wide popularity of cloud computing and micro-service architecture, the scale of large data center server clusters has broken through the level of 100,000 nodes. The complex service dependency chain and hardware-software coupling relationship lead to cascading and cross-layer characteristics of fault propagation. The traditional monitoring system, when facing composite faults such as network jitter, storage performance degradation or container resource contention, lacks the ability to dynamically model the service-resource three-dimensional dependency topology, and can only output discrete threshold alarms, causing an "alarm storm" that obscures the real root cause. The industry still needs an intelligent root cause positioning solution that integrates multi-source data real-time analysis, dynamic causal reasoning and self-optimization capabilities to address the serious challenges of high false alarm rate, low cross-layer fault positioning efficiency, and heavy reliance on human experience, and to meet the rigid demand for high-availability services in the fields of finance, e-commerce, cloud computing, etc.

[0003] The existing technology has the problems of high false alarm rate, which obscures the real fault signal, cannot reflect the real-time service dependency relationship, has low positioning efficiency, long average repair time, and lacks a closed-loop optimization mechanism; SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the problems in the related art, the application provides a fault root cause positioning method for a server cluster to overcome the above technical problems existing in the prior art.

[0006] (II) Technical solutions

[0007] To solve the above technical problems, the application is realized by the following technical solutions:

[0008] S1, collect a multi-source server cluster dataset; standardize and process abnormal data of the multi-source server cluster dataset to obtain a processed multi-source server cluster dataset;

[0009] S2, construct a topology graph; based on the processed multi-source server cluster dataset, obtain a basic node set, a state attribute set, an inter-service call edge and a service deployment relationship edge of the topology graph;

[0010] S3, dynamically calculate the weight of the inter-service call edge by a dynamic edge weight calculation model to obtain a real-time edge weight; update the topology graph by a topology real-time updating algorithm combined with the real-time edge weight to obtain a dynamic topology graph;

[0011] S4, convert the dynamic topology graph into a causal reasoning network and remove useless edges to obtain a key causal reasoning network; calculate the dynamic edge weight of the key causal reasoning network and the fault condition probability table to obtain a Bayesian network;

[0012] S5, when a fault alarm occurs, collect original fault alarm data; process the original fault alarm data through a three-stage processing pipeline operation and an FP-Growth algorithm to obtain a frequent item set with strong causal association; based on the Bayesian network and the frequent item set with strong causal association, obtain a final fault root cause probability distribution through a random walk model;

[0013] The application constructs a global view through multi-source data standardization, creates a service-resource dynamic topology graph, and innovatively designs a real-time edge weight calculation model; when a fault occurs, a three-stage pipeline is used to compress the alarm storm, and an FP-Growth algorithm and KL divergence causal analysis are combined to screen a strong association item set, the topology is converted into a Bayesian network, the probability is propagated through edge weight mapping, and the probability root cause is output through random walk reasoning; the problems of multi-source data dispersion, dependence on dynamic changes, alarm interference and inaccurate root cause positioning in the cloud native environment are solved, the fault diagnosis precision and efficiency are significantly improved, and the average repair time is reduced.

[0014] Preferably, the S1 comprises the following steps:

[0015] S11, obtain a multi-source server cluster dataset by collecting microservice call chain data, container resource index data, physical node index data and network state data of the server cluster;

[0016] S12, synchronize and align all data points in the multi-source server cluster dataset to nanoseconds based on a precise time protocol server to obtain an aligned multi-source server cluster dataset;

[0017] Analyze the relationship between the data entities in the aligned multi-source server cluster dataset to obtain an analyzed multi-source server cluster dataset;

[0018] Uniform the same type index values of different sources in the analyzed multi-source server cluster dataset to a standard unit to obtain a uniformed multi-source server cluster dataset;

[0019] Perform format standardization and write processing on the uniformed multi-source server cluster dataset to obtain a formatted multi-source server cluster dataset;

[0020] S13, process the formatted multi-source server cluster dataset through a long-short-time missing and conflict resolution algorithm to obtain a processed multi-source server cluster dataset;

[0021] The application solves the problem of scattered and dislocated multi-source heterogeneous data by collecting four types of data of micro-service call chain, container, physical node and network, performing nanosecond timestamp alignment, entity relationship analysis, unit unification and format standardization, adopting linear interpolation to fill short-time missing, health check to cut path to fill long-time missing and version vector to solve conflict, and builds a global consistent high-quality data set to lay a solid foundation for fault analysis.

[0022] Preferably, the S2 comprises the following steps:

[0023] S21, a topology graph is constructed; the unique identifiers and type information of different entities in the processed multi-source server cluster data set are identified to obtain a basic node set of the topology graph; the basic node set of the topology graph comprises service nodes, container nodes, physical nodes and network nodes;

[0024] By continuously receiving real-time monitoring index data of each type of node in the processed multi-source server cluster data set and performing association, the state attributes attached to each node are dynamically updated to obtain a dynamically updated state attribute set;

[0025] S22, the micro-service call chain data in the processed multi-source server cluster data set is analyzed to identify the calling relationship and the calling relationship of the service name, and the edges representing the calling between services are automatically created or updated to obtain dynamically updated calling edges between services;

[0026] By combining the service name and container identifier in the call chain data in the processed multi-source server cluster data set and the container-host mapping established by the standardization engine, edges representing service deployment relationship are automatically created to obtain dynamically updated service deployment relationship edges;

[0027] The application constructs a topology node set based on the four types of entities of standardized data recognition service, container, physical node and network, dynamically associates real-time monitoring index to update node state attributes; automatically generates calling edges between services through call chain analysis, and creates service deployment edges in combination with container-host mapping to realize dynamic visualization of system resources and dependency relationship, accurately map complex architecture, and provide real-time topology basis for fault propagation analysis.

[0028] Preferably, the S3 comprises the following steps:

[0029] S31, the weight of the dynamically updated service deployment relationship edge is set as a fixed weight to obtain a fixed service deployment relationship edge weight;

[0030] S32, a dynamic edge weight calculation model is constructed; the weight of the calling edge between services is calculated through the dynamic variable weight model to obtain real-time edge weight;

[0031] S33, updating the topology graph through a topology real-time updating algorithm to obtain a dynamically updated topology graph;

[0032] The application fuses real-time load factors and historical fault Sigmoid conversion influence factors to obtain a service inter-call edge dynamic weight model, fixes the deployment edge weight, and periodically updates the topology edge weight through a scheduler; accurately quantifies the real-time influence strength between nodes, upgrades the static topology to a "dynamic blood vessel", and significantly improves the accuracy of fault propagation path analysis.

[0033] Preferably, the S32 comprises the following steps:

[0034] S321, collecting historical fault events of service inter-call edges and their occurrence times to obtain historical edge fault data;

[0035] Setting a decay coefficient, calculating the exponential decay sum of historical edge faults through an exponential decay sum formula;

[0036] S322, based on the exponential decay sum of historical edge faults, calculating a historical negative influence factor of historical faults on current weight through a Sigmoid function;

[0037] Setting a maximum capacity threshold of service inter-call edges; based on the processed multi-source server cluster data set, obtaining real-time traffic metrics on service inter-call edges at the current time;

[0038] Comparing the real-time traffic metrics on service inter-call edges at the current time with the maximum capacity threshold of service inter-call edges to obtain a normalized real-time load factor;

[0039] Respectively setting the weights of the historical negative influence factor and the real-time load factor; based on the historical negative influence factor, the real-time load factor, and the weights of the historical negative influence factor and the real-time load factor, calculating a dynamic weight value of the service inter-call edge;

[0040] The application quantifies the influence of historical faults through an exponential decay sum formula, converts it into a dynamic negative influence factor through a Sigmoid function, calculates a normalized load factor combining real-time traffic and capacity threshold, and fuses the double-factor linear weighting to generate a service inter-call edge dynamic weight; synchronously reflects the system real-time pressure and historical fault memory, accurately depicts the influence strength change between nodes, and provides a quantitative basis for fault propagation analysis.

[0041] Preferably, the S33 comprises the following steps:

[0042] S331, checking whether the entity ID corresponding to the data point in the processed multi-source server cluster data set exists in the graph, and when it does not exist, creating a new node containing its type and initial state attributes;

[0043] S332, set the relationship discovery rule to obtain a preset relationship rule; by identifying the type of the data points in the processed multi-source server cluster data set, a new edge is created or the association information of the existing edge is updated according to the preset relationship discovery rule;

[0044] S333, deploy an independently running scheduler and set a traversal period; by the independently running scheduler, all edges in the topology graph are traversed in each traversal period, and the weight value of each edge is updated;

[0045] The application realizes real-time dynamic evolution of the topology graph, ensures real-time and accurate mapping of system state changes and dependency relationships, and provides adaptive and updated core data support for fault analysis.

[0046] Preferably, the S4 comprises the following steps:

[0047] S41, converting the dynamic topology graph into a causal reasoning network to obtain an initial causal reasoning network;

[0048] A weight threshold is set; the edges with a dynamic edge weight less than the weight threshold in the initial causal reasoning network are regarded as useless edges; the useless edges are removed to obtain a key edge set and a key causal reasoning network;

[0049] S42, converting the dynamic edge weight of each edge in the key edge set into a propagation intensity to obtain a propagation intensity set;

[0050] Based on the propagation intensity set, the fault propagation probability of the key edge is calculated through a conditional probability formula to obtain a fault conditional probability table;

[0051] S43, obtaining a Bayesian network through S41 and S42;

[0052] The application prunes the dynamic topology by filtering weak edges through a weight threshold, converts it into a key causal reasoning network, maps the edge weight into a propagation intensity through a Sigmoid function, and derives a fault conditional probability table to construct a Bayesian network.

[0053] Preferably, the S5 comprises the following steps:

[0054] S51, collecting original fault alarm data when a fault alarm occurs; the original fault alarm data is compressed through a three-stage processing pipeline to obtain compressed fault alarm data;

[0055] S52, processing the compressed fault alarm data through a vertical data layout converter to obtain cache-friendly bitmap index structure data;

[0056] The highly compressed frequent pattern prefix tree data is obtained through the bit map index structure data careful processing of the Trie tree compression storage technology + 8 thread pool processing;

[0057] The frequent item set set is obtained by traversing the highly compressed frequent pattern prefix tree data through the MapReduce mode miner;

[0058] S53, set the causal strength threshold value; based on the dynamic weight value of the edge of the inter-service call and the exponential decay sum of the historical edge fault, the causal strength value of each frequent item set in the frequent item set set is obtained through KL divergence processing;

[0059] The causal strength value of each frequent item set is compared with the causal strength threshold value, and the frequent item set with a causal strength less than the causal strength threshold value is removed, and the strong causal correlation frequent item set is obtained;

[0060] S54, based on the alarm node in the strong causal correlation frequent item set, the node access frequency score is obtained by traversing the Bayesian network through the reverse random walk sampler;

[0061] Based on the node access frequency score and the likelihood estimation of the Bayesian network, the candidate fault root cause probability distribution is obtained through the joint probability calculator;

[0062] S55, the root cause probability in the candidate fault root cause probability distribution is standardized by the probability normalization processor, and the final fault root cause probability distribution is obtained;

[0063] The original alarm is compressed through a three-stage pipeline, the compressed FP-Tree is constructed by using the bit map index and the multi-threaded Trie tree, the frequent item set is mined in parallel FP-Growth, the KL divergence causal strength is calculated based on the topological edge weight and the historical fault exponential decay sum, and the strong causal correlation item set is screened; the traditional association rule limitation is broken through, the real fault causal chain is accurately identified, the alarm analysis efficiency and reliability are significantly improved; the strong causal alarm item set is taken as evidence, the node access frequency is calculated by using the reverse random walk sampling, the candidate root cause distribution is generated by combining the Bayesian likelihood estimation and the joint probability, and the probability sorting is finally normalized and output; high-precision root cause positioning in a complex dependent network is realized, the diagnosis accuracy is improved, and the average repair time is reduced.

[0064] Preferably, the S51 is compressed through a three-stage processing pipeline to obtain the compressed fault alarm data, and the method comprises the following steps:

[0065] S511, count and merge the same type of infrastructure alarms to obtain the merged infrastructure alarms;

[0066] S512, based on the service-resource topology relationship, the container OOM alarm and the host memory pressure alarm are associated as a container resource squeeze combination event alarm;

[0067] S513, through service call chain analysis, the payment API delay, order DB timeout and Redis cache hit are fused as transaction link degradation semantic event alarms;

[0068] S514, through S511, S512 and S513, compressed fault alarm data is obtained;

[0069] The application combines alarms through a time window, generates combined events through topology association, and aggregates semantic events through call chains to compress original alarms; the alarm storm is suppressed, cross-layer business semantics are refined, and a high-value event chain is provided for root cause positioning.

[0070] A fault root cause positioning system for a server cluster is used to implement the above-mentioned fault root cause positioning method for the server cluster, comprising a multi-source data acquisition and standardization processing module, a dynamic topology graph construction and updating module, a causal reasoning network construction module and a fault causal association mining and root cause positioning module.

[0071] The multi-source data acquisition and standardization processing module is used to collect original and heterogeneous monitoring data from the server cluster environment, and to strictly preprocess the data; based on the precise time protocol timestamp nanosecond level synchronization alignment, the relationship between data entities is analyzed, the index unit is unified, the data format is standardized, and the data quality problem is handled; the final output is a processed multi-source server cluster dataset that is strictly cleaned, aligned, standardized and consistent, providing reliable basic data for subsequent analysis.

[0072] The dynamic topology graph construction and updating module uses the processed multi-source server cluster dataset to construct and maintain a topology graph reflecting the structure and state of the entire server cluster system in real time; first, the basic nodes are identified and created, and the state attribute set of each node is continuously associated with real-time monitoring indicators to dynamically update the health and performance; edges representing the calling relationship between services are automatically created and updated by analyzing the call chain data, and edges representing the service deployment relationship are created in combination with the container mapping information; the dynamic edge weight is calculated according to the real-time traffic load factor and the exponential decay and dynamic calculation of historical fault events; the independent scheduler periodically traverses all edges to update the weight, ensuring that the topology graph can dynamically track the latest state of the system, and obtaining a dynamic topology graph containing real-time state and dynamic weight;

[0073] The cause-effect reasoning network construction module converts the dynamic topology graph into a cause-effect network for root cause reasoning;According to the set weight threshold, filter out the edges with too low dynamic weight, and obtain a simplified key cause-effect reasoning network;The dynamic weight value of the key edge is mapped to a parameter representing the fault propagation strength through a conversion function, and the conditional fault probability table of the key edge is calculated, so a Bayesian network with propagation probability is constructed;

[0074] The fault cause-effect correlation mining and root cause positioning module is used for receiving original alarm data stream when a fault occurs, and counting and merging the same type of infrastructure alarms in a short time window;Based on the service-resource topology relationship, the related alarms are associated to form a combined event;Through business call chain analysis, multiple service / component alarms affecting the same business link are fused into events with business semantics;The compressed alarm data is converted into a bitmap index structure, and a highly compressed frequent pattern prefix tree is constructed using Trie tree compression technology and multi-thread processing;The MapReduce mode miner is used to mine the frequent co-occurring alarm patterns from the tree;The edge weight and historical fault exponential decay in the dynamic topology graph are used to quantify the causal influence between elements in the term set by the KL divergence method, and the frequent item set with strong causal correlation is obtained by removing weakly associated items;The strong causal correlation frequent item set is used as input, and the process of simulating fault reverse propagation from alarm nodes to potential root cause nodes is used to record the access frequency of each node as its root cause score;The joint probability calculation is combined with the likelihood estimation of the Bayesian network to obtain the probability distribution of the candidate root cause;Through probability normalization processing, the final fault root cause probability distribution is obtained, which clearly indicates the most likely fault source and its probability.

[0075] (Three) beneficial effects

[0076] The present application has the following beneficial effects:

[0077] Through the whole-chain innovation of "multi-source fusion-dynamic topology-cause-effect mining-probability reasoning", the present application significantly improves the automation, intelligence and precision level of cloud native system fault diagnosis, effectively reduces the average repair time, and guarantees business continuity.

[0078] At the data level, the present application constructs a high-quality, strongly consistent multi-source fusion data set by nanosecond-level timestamp synchronization, entity relationship analysis, unit standardization and intelligent processing of data missing conflicts, lays a reliable foundation for upper-layer analysis, and solves the "blind man touching an elephant" problem caused by scattered and misplaced data;

[0079] The application innovatively constructs a dynamic topology graph of service-resource dual-dimension dependence at the system modeling level. The graph not only contains full-factor nodes such as services, containers, physical nodes, network devices and real-time state attributes thereof, but also distinguishes dynamic calling edges between services and static deployment edges of services / containers, and accurately depicts the system structure. A weight calculation model based on historical fault exponential decay and real-time normalized load factor is designed. The historical negative influence factor is converted into an influence value in the interval (0, 1) by a Sigmoid function, so that the weight can reflect the current traffic pressure and also remember the distance and severity of historical faults. The weight is periodically updated by an independent scheduler to ensure that the topology always tracks the latest system state. This enables the topology graph to upgrade from a “static skeleton” to a “dynamic blood vessel”, and accurately quantifies the real-time influence strength between components.

[0080] At the alarm processing level, the application innovatively constructs a three-stage pipeline compressor mechanism. Redundancy is reduced by time window counting, combined events are constructed by topology association, and semantic events are generated by calling chain fusion, which significantly suppresses alarm storms and improves semantic value. Frequent item sets are efficiently mined by bitmap indexing, Trie tree compression and parallel FP-Growth, and the causal strength is innovatively calculated by combining topology edge weight and KL divergence, so as to screen strong causal correlation item sets and break through the limitation of traditional association rules that only mine co-occurrence and ignore causality.

[0081] At the root cause positioning level, the dynamic topology is converted into a causal reasoning network, and weak edges are retained by pruning according to the weight threshold to reserve key causal paths. The edge weight is converted into propagation strength by a function, and a conditional probability table is derived to construct a Bayesian network with quantifiable propagation probability. The strong causal alarm item set is taken as evidence, and the candidate root cause distribution is generated by combining Bayesian likelihood estimation and joint probability calculation through reverse random walk sampling to simulate fault propagation. Finally, the probability-based root cause ranking is normalized and output. The experience-driven is upgraded to data probability-driven, and high-precision and interpretable root cause positioning is realized in a complex network topology.

[0082] Of course, implementing any product of the application does not necessarily need to achieve all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0084] Figure 1 A flowchart of a fault root cause positioning method for a server cluster according to the application;

[0085] Figure 2 A flowchart for original fault alarm compression in a fault root cause positioning method for a server cluster of the present application is shown in the figure;

[0086] Figure 3 A module diagram of a fault root cause positioning system for a server cluster of the present application is shown in the figure. DETAILED DESCRIPTION

[0087] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the application.

[0088] In the description of the application, it should be understood that the terms "opening", "upper", "lower", "top", "middle", "inner" and the like indicate the orientation or positional relationship and are only used to facilitate the description of the application and simplify the description, and do not indicate or imply that the indicated component or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the application.

[0089] Embodiment one: please refer to Figure 1 , Figure 2 The application discloses a fault root cause positioning method for a server cluster, comprising the following steps:

[0090] S1, collect a multi-source server cluster dataset; standardize and process the multi-source server cluster dataset to obtain a processed multi-source server cluster dataset;

[0091] The S1 comprises the following steps:

[0092] S11, collect microservice call chain data, container resource index data, physical node index data and network state data of the server cluster to obtain a multi-source server cluster dataset;

[0093] The S11 comprises the following steps:

[0094] Collect full-amount call chains through OpenTelemetry Agent deployed on each microservice; the captured key information includes unique tracking identification, call relationship, service name and call delay to obtain microservice call chain data;

[0095] Collect container-level resource usage through cAdvisor combined with Kubernetes API; the captured key information includes container identification, CPU usage and memory working set size to obtain container resource index data;

[0096] Collect hardware and operating system metrics of the host machine through Node Exporter and IPMI tools; key information captured includes host IP, network send traffic, disk I / O latency, and physical node metrics;

[0097] Receive log events using Syslog forwarding and capture key state change events of network devices using SNMP Trap mechanism; key information captured includes device IP, port status, and packet loss rate, resulting in network state data;

[0098] S12, synchronize and align the timestamps of all data points in the multi-source server cluster dataset to nanosecond level based on the precise time protocol server, resulting in the aligned multi-source server cluster dataset; ensure the time consistency of cross-source data.

[0099] Parse the relationships between data entities in the aligned multi-source server cluster dataset, resulting in the parsed multi-source server cluster dataset; for example, for container metric data, it will be mapped to the host machine it is in by querying the Kubernetes API, establishing the association between the container and the physical resource;

[0100] Uniform the same type of metric values from different sources in the parsed multi-source server cluster dataset to standard units, resulting in the unified multi-source server cluster dataset; for example, all memory-related metrics will be converted to megabytes;

[0101] Perform format standardization and write processing on the unified multi-source server cluster dataset, resulting in the formatted multi-source server cluster dataset;

[0102] S13, process the formatted multi-source server cluster dataset through long and short time missing and conflict resolution algorithm, resulting in the processed multi-source server cluster dataset;

[0103] Short-term missing is compensated by estimating the missing value based on the valid data points before and after the missing point in the formatted multi-source server cluster dataset using linear interpolation algorithm, maintaining the continuity of the data stream;

[0104] Long-term missing triggers a data source health check process, the system automatically diagnoses the problem source and attempts to switch to the pre-configured backup collector or path to restore the missing data stream of the formatted multi-source server cluster dataset;

[0105] In the scenario of distributed collection that may cause version conflicts in the formatted multi-source server cluster dataset (such as multiple collection points reporting different states of the same entity at the same time), the system adopts a conflict resolution algorithm based on version vectors to effectively identify the causality and concurrency of events, intelligently resolve conflicts, and ensure the consistency of the final data view.

[0106] S2, constructing a topology graph; based on the processed multi-source server cluster dataset, obtaining a basic node set, a state attribute set, an inter-service calling edge, and a service deployment relationship edge of the topology graph;

[0107] The S2 includes the following steps:

[0108] S21, constructing a topology graph; by identifying the unique identifiers and type information of different entities in the processed multi-source server cluster dataset, obtaining a basic node set of the topology graph; the basic node set of the topology graph includes service nodes, container nodes, physical nodes, and network nodes;

[0109] By continuously receiving real-time monitoring index data (such as service QPS / error rate / P99 delay, container CPU / memory / network usage, physical node CPU load / disk utilization / temperature, network port packet loss / RTT) of various nodes in the processed multi-source server cluster dataset and performing association, the "state attribute set" attached to each node is dynamically updated to obtain a dynamically updated state attribute set; the node reflects the health and performance status of the system components in real time;

[0110] S22, by analyzing the micro-service calling chain data in the processed multi-source server cluster dataset, identifying the calling relationship and the service name calling relationship, automatically creating or updating the edges representing the inter-service calling, and obtaining a dynamically updated inter-service calling edge; the source node of the dynamically updated edge is the parent service, and the target node is the called service;

[0111] By combining the service name and container identifier in the calling chain data in the processed multi-source server cluster dataset, and the container-host mapping established by the standardization engine, the service deployment relationship edge is automatically created to obtain a dynamically updated service deployment relationship edge; the source node is the service, and the target node is the container Pod (the weight is fixed to 1.0, indicating a strong association; the edges between physical nodes and containers, and network nodes and physical nodes / containers are usually automatically associated through resource ownership information, which is not detailed in the example code but has similar logic);

[0112] S3, performing dynamic weight calculation on the inter-service calling edge through a dynamic edge weight calculation model to obtain real-time edge weights;

[0113] By combining the real-time edge weights, the topology is updated through a topology real-time updating algorithm to obtain a dynamic topology graph;

[0114] S31, set the weight of the edge of the dynamically updated service deployment relationship as a fixed weight to obtain a fixed service deployment relationship edge weight; the service invocation edge needs to reflect real-time business traffic (such as a sudden increase in QPS leading to an increase in delay), and the deployment edge represents a static attribution relationship;

[0115] S32, construct a dynamic edge weight calculation model; calculate the weight of the service invocation edge through the dynamic variable weight model to obtain a real-time edge weight;

[0116] The S32 includes the following steps:

[0117] S321, collect historical fault events of the service invocation edge and their occurrence times to obtain historical edge fault data;

[0118] Set the decay coefficient as λ, and calculate the exponential decay sum of the historical edge fault through an exponential decay sum formula; the exponential decay sum formula is as follows,

[0119] ;

[0120] Wherein, F ( t ) represents the exponential decay sum of the historical edge fault (i.e. the cumulative value of the historical fault influence), t represents the current time, t i represents the occurrence time of the historical i time fault event related to the edge, delta i represents the weight of the set i time fault event (used to represent the severity or influence of the fault);

[0121] S322, based on the exponential decay sum of the historical edge fault, calculate the historical negative influence factor of the historical fault on the current weight through a Sigmoid function (the more and the closer the historical faults, the greater the negative influence); the Sigmoid function calculation formula is as follows,

[0122] ;

[0123] Wherein, d represents the negative influence factor of the historical fault on the current weight (the value range is (0, 1), and the greater the value represents the greater the influence of the historical fault), k represents the slope parameter of the set Sigmoid function (used to control the steepness of the curve), e represents the base number of the natural logarithm;

[0124] Set the maximum capacity threshold of the inter-service invocation edge; based on the processed multi-source server cluster data set, obtain the real-time traffic metric (such as the number of invocations, bandwidth) on the inter-service invocation edge at the current time;

[0125] Compare the real-time traffic metric on the inter-service invocation edge at the current time with the maximum capacity threshold of the inter-service invocation edge to obtain a normalized real-time load factor (the higher the real-time load, the larger the factor);

[0126] Set the weights of the historical negative impact factor and the real-time load factor respectively; based on the historical negative impact factor, the real-time load factor, and the weights of the historical negative impact factor and the real-time load factor, calculate the dynamic weight value of the inter-service invocation edge; the calculation formula is as follows,

[0127] ;

[0128] Wherein, w represents the dynamic weight value of the dynamically updated inter-service invocation edge, f represents the normalized load factor, α , β respectively represent the weights of the historical negative impact factor and the real-time load factor;

[0129] S33, update the topology graph by the topology real-time updating algorithm to obtain a dynamically updated topology graph;

[0130] The S33 includes the following steps:

[0131] S331, check whether the entity ID corresponding to the data point in the processed multi-source server cluster data set exists in the graph, and when it does not exist, create a new node containing its type and initial state attribute;

[0132] S332, set the relationship discovery rule (such as service invocation chain, service-container deployment) to obtain a preset relationship rule; by identifying the type (such as TRACE_SPAN) of the data point in the processed multi-source server cluster data set, create a new edge or update the association information (especially the dynamic weight, which is calculated by calling the CalculateWeight(edge) function to apply the mathematical model) of the existing edge according to the preset relationship discovery rule;

[0133] S333, deploy an independently running scheduler and set a traversal period; by the independently running scheduler, traverse all edges in the topology graph in each traversal period (such as one hour) to update the weight value of each edge (this ensures that the influence of historical failures F ( t ) gradually decreases over time, so that the topology weight can dynamically track the latest state of the system);

[0134] S4, converting the dynamic topology graph into a causal reasoning network and removing useless edges to obtain a key causal reasoning network; calculating dynamic edge weights of the key causal reasoning network and a fault condition probability table to obtain a Bayesian network;

[0135] The S4 includes the following steps:

[0136] S41, converting the dynamic topology graph into a causal reasoning network to obtain an initial causal reasoning network;

[0137] Setting a weight threshold; regarding edges with dynamic edge weights less than the weight threshold in the initial causal reasoning network as useless edges; removing the useless edges to obtain a key edge set and a key causal reasoning network;

[0138] S42, converting the dynamic edge weight of each edge in the key edge set into a propagation intensity to obtain a propagation intensity set; the conversion formula is as follows,

[0139] ;

[0140] Wherein, theta represents the propagation intensity of each edge in the key edge set, and w(t) represents the dynamic weight value of each edge in the key edge set;

[0141] Based on the propagation intensity set, the fault propagation probability of the key edge is calculated through a conditional probability formula to obtain a fault condition probability table; the conditional probability formula is as follows,

[0142] ;

[0143] Wherein, P ( a | b ) represents the fault a The fault propagation probability of the parent node b in the key causal reasoning network, theta mi represents the fault propagation intensity of the first parent node in the key causal reasoning network to the node i ; m

[0144] S43, obtaining the Bayesian network through S41 and S42;

[0145] S5, when a fault alarm occurs, collecting original fault alarm data; processing the original fault alarm data through a three-stage processing pipeline operation and an FP-Growth algorithm to obtain a frequent item set with strong causal association; based on the Bayesian network and the frequent item set with strong causal association, obtaining a final fault root cause probability distribution through a random walk model;

[0146] The S5 includes the following steps: ​

[0147] S51, collecting original fault alarm data when a fault alarm occurs; compressing the original fault alarm data through a three-stage processing pipeline to obtain compressed fault alarm data;

[0148] The S51 includes the following steps:

[0149] S511, counting and merging infrastructure alarms of the same type (such as continuous CPU over-limit and disk high load) in a 5-second window to obtain merged infrastructure alarms;

[0150] S512, based on the service-resource topology relationship, associating the container OOM alarm and the host memory pressure alarm as a container resource squeeze combined event alarm;

[0151] S513, through business call chain analysis, merging payment API delay, order DB timeout, and Redis cache hit into a transaction link degradation semantic event alarm;

[0152] S514, through S511, S512, and S513, obtaining compressed fault alarm data;

[0153] S52, processing the compressed fault alarm data through a vertical data layout converter to obtain cache-friendly bitmap index structure data;

[0154] Through Trie tree compression storage technology + 8 thread pool processing, the cache-friendly bitmap index structure data is carefully processed to obtain highly compressed frequent pattern prefix tree data;

[0155] Through a MapReduce mode miner, the highly compressed frequent pattern prefix tree data is traversed to obtain a frequent item set collection;

[0156] S53, setting a causal strength threshold; based on the dynamic weight value of the edge of the inter-service call and the exponential decay of the historical edge fault, the causal strength value of each frequent item set in the frequent item set collection is obtained through KL divergence processing;

[0157] The causal strength value of each frequent item set is compared with the causal strength threshold, and the frequent item set with a causal strength less than the causal strength threshold is removed to obtain a strongly causally related frequent item set;

[0158] S54, based on the alarm nodes in the strongly causally related frequent item set, traversing the Bayesian network through a reverse random walk sampler to obtain node access frequency scores;

[0159] Based on the node access frequency scores and the Bayesian network likelihood estimate, and through a joint probability calculator, a candidate fault root cause probability distribution is obtained;

[0160] S55, standardize the root cause probabilities in the candidate fault root cause probability distribution through a probability normalization processor to obtain a final fault root cause probability distribution.

[0161] Embodiment two: please refer to Figure 3 A fault root cause positioning system for a server cluster, configured to implement the above-mentioned fault root cause positioning method for a server cluster, comprising a multi-source data acquisition and standardization processing module, a dynamic topology graph construction and updating module, a causal reasoning network construction module, and a fault causal correlation mining and root cause positioning module.

[0162] The multi-source data acquisition and standardization processing module is configured to collect raw and heterogeneous monitoring data from a server cluster environment and perform strict preprocessing thereon; based on precise time protocol timestamp nanosecond level synchronization alignment, data entity relationship analysis, index unit unification, data format standardization, and data quality processing; the final output is a processed multi-source server cluster dataset that is strictly cleaned, aligned, standardized, and consistent, providing reliable basic data for subsequent analysis.

[0163] The dynamic topology graph construction and updating module uses the processed multi-source server cluster dataset to construct and maintain a topology graph reflecting the structure and state of the entire server cluster system in real time; first, identify and create basic nodes, and continuously associate real-time monitoring indicators to dynamically update the state attribute set of each node to reflect its health and performance; automatically create and update edges representing service-to-service call relationships by analyzing call chain data, and create edges representing service deployment relationships in combination with container mapping information; dynamically calculate the dynamic edge weights according to real-time traffic load factors and historical fault events; periodically traverse all edges through an independent scheduler to update the weights, ensuring that the topology graph can dynamically track the latest state of the system, and obtain a dynamic topology graph containing real-time state and dynamic weights;

[0164] The causal reasoning network construction module converts the dynamic topology graph into a causal network for root cause reasoning; according to the set weight threshold, filter out edges with excessively low dynamic weights to obtain a simplified key causal reasoning network; map the dynamic weight values of the key edges to a parameter representing the fault propagation intensity through a conversion function, and calculate the conditional fault probability table of the key edges, thus constructing a Bayesian network with propagation probability;

[0165] The fault cause-effect correlation mining and root cause positioning module is configured to receive original alarm data streams when a fault occurs, and to combine alarms of the same type of infrastructure in a short time window by counting; to form combined events by correlating related alarms based on service-resource topology relationships; to fuse multiple service / component alarms affecting the same service link into events with service semantics through service call chain analysis; to convert the compressed alarm data into a bitmap index structure, and to construct a highly compressed frequent pattern prefix tree using Trie tree compression technology and multi-thread processing; to mine frequent co-occurring alarm patterns from the tree using a MapReduce pattern miner; to obtain frequent item sets with strong cause-effect correlations by removing weakly correlated items using edge weights and historical fault exponential decay in a dynamic topology graph, and quantifying the cause-effect influence between elements in the item set using a KL divergence method; to take the strong cause-effect correlation frequent item sets as input, and to record the access frequency of each node as its score of becoming a root cause by simulating the process of fault back-propagation from alarm nodes to potential root cause nodes; to calculate the joint probability using the likelihood estimation of a Bayesian network to obtain the probability distribution of candidate root causes; and to obtain the final fault root cause probability distribution by probability normalization processing, and to clearly indicate the most likely fault source and its probability.

[0166] In the description of the present specification, the description referring to the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the invention. In the present specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0167] The above-disclosed preferred embodiments of the invention are only used to help explain the invention. The preferred embodiments do not describe all the details of the invention, nor limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the invention, so that those skilled in the art can well understand and utilize the invention.

Claims

1. A method for locating the root cause of a fault in a server cluster, characterized in that, Includes the following steps: S1. Collect multi-source server cluster datasets; perform standardization and outlier processing on the multi-source server cluster datasets to obtain the processed multi-source server cluster datasets. S2. Construct the topology graph; Based on the processed multi-source server cluster dataset, the basic node set, state attribute set, service call edges, and service deployment relationship edges of the topology graph are obtained. S3. The dynamic edge weight calculation model is used to dynamically calculate the weight of the service call edge to obtain the real-time edge weight. A dynamic topology graph is obtained by updating the topology graph using a real-time topology update algorithm combined with real-time edge weights. S4. Transform the dynamic topology graph into a causal inference network and remove useless edges to obtain the key causal inference network; calculate the dynamic edge weights and fault condition probability table of the key causal inference network to obtain the Bayesian network. S5. When a fault alarm occurs, collect the original fault alarm data; process the original fault alarm data through a three-level processing pipeline operation and the FP-Growth algorithm to obtain frequent itemsets with strong causal association; based on Bayesian networks and frequent itemsets with strong causal association, obtain the final fault root cause probability distribution through a random walk model.

2. The method for locating the root cause of a server cluster fault according to claim 1, characterized in that, S1 includes the following steps: S11. By collecting microservice call chain data, container resource indicator data, physical node indicator data, and network status data of the server cluster, a multi-source server cluster dataset is obtained. S12. Synchronize and align the timestamps of all data points in the multi-source server cluster dataset at the nanosecond level based on the precise time protocol server to obtain the aligned multi-source server cluster dataset. The relationships between data entities in the aligned multi-source server cluster dataset are analyzed to obtain the parsed multi-source server cluster dataset. The values ​​of the same type of indicators from different sources in the parsed multi-source server cluster dataset are unified to a standard unit to obtain a unified multi-source server cluster dataset. The unified multi-source server cluster dataset is subjected to format standardization and writing processes to obtain a formatted multi-source server cluster dataset. S13. The formatted multi-source server cluster dataset is processed using long and short time missing data and conflict resolution algorithms to obtain the processed multi-source server cluster dataset.

3. The method for locating the root cause of a server cluster fault according to claim 1, characterized in that, S2 includes the following steps: S21. Construct a topology graph; by identifying the unique identifiers and type information of different entities in the processed multi-source server cluster dataset, the basic node set of the topology graph is obtained; the basic node set of the topology graph includes service nodes, container nodes, physical nodes, and network nodes; By continuously receiving and associating real-time monitoring index data of various nodes in the multi-source server cluster dataset, the status attributes of each node are dynamically updated to obtain a dynamically updated set of status attributes. S22. By parsing the microservice call chain data in the processed multi-source server cluster dataset, identify the call relationship and the call relationship between service names, automatically create or update the edges representing inter-service calls, and obtain dynamically updated inter-service call edges. By combining the service name and container identifier in the centralized call chain data of the processed multi-source server cluster data, and the container-host mapping established by the standardized engine, service deployment relationship edges are automatically created, resulting in dynamically updated service deployment relationship edges.

4. The method for locating the root cause of a server cluster fault according to claim 1, characterized in that, S3 includes the following steps: S31. Set the weight of the edge of the dynamically updated service deployment relationship to a fixed weight to obtain the fixed service deployment relationship edge weight; S32. Construct a dynamic edge weight calculation model; calculate the weight of the inter-service call edge using the dynamic edge weight calculation model to obtain the real-time edge weight; S33. Update the topology graph using a real-time topology update algorithm to obtain a dynamically updated topology graph.

5. The method for locating the root cause of a server cluster fault according to claim 4, characterized in that, S32 includes the following steps: S321. Collect historical fault events and their occurrence times of inter-service call edges to obtain historical edge fault data; Set an attenuation coefficient and calculate the exponentially decayed sum of historical edge faults using the exponential decay summation formula; S322. Based on the exponential decay sum of historical edge faults, the historical negative impact factor of historical faults on the current weights is calculated using the Sigmoid function. Set the maximum capacity threshold for inter-service call edges; based on the processed multi-source server cluster dataset, obtain the real-time traffic measurement of the inter-service call edges at the current moment; The real-time traffic metric on the inter-service call edge at the current moment is compared with the maximum capacity threshold of the inter-service call edge to obtain the normalized real-time load factor. The weights of the historical negative impact factor and the real-time load factor are set separately; based on the weights of the historical negative impact factor, the real-time load factor, and the weights of the historical negative impact factor and the real-time load factor, the dynamic weight value of the inter-service call edge is calculated.

6. The method for locating the root cause of a server cluster fault according to claim 4, characterized in that, S33 includes the following steps: S331. Check whether the entity ID corresponding to the data point in the processed multi-source server cluster dataset already exists in the graph. If it does not exist, create a new node containing its type and initial state attributes. S332. Set relationship discovery rules to obtain pre-set relationship rules; by identifying the type of data points in the processed multi-source server cluster dataset, create new edges or update the association information of existing edges according to the pre-set relationship discovery rules; S333. Deploy an independently running scheduler and set the traversal period; using the independently running scheduler, traverse all edges in the topology graph in each traversal period and update the weight value of each edge.

7. The method for locating the root cause of a server cluster failure according to claim 1, characterized in that, S4 includes the following steps: S41. Transform the dynamic topology graph into a causal inference network to obtain the initial causal inference network; Set a weight threshold; designate edges in the initial causal inference network whose dynamic edge weights are less than the weight threshold as useless edges; remove useless edges to obtain the key edge set and the key causal inference network; S42. Convert the dynamic edge weight of each edge in the key edge set into propagation intensity to obtain the propagation intensity set; Based on the propagation intensity set, the failure propagation probability of the critical edge is calculated using the conditional probability formula to obtain the failure conditional probability table. S43. Obtain the Bayesian network through S41 and S42.

8. The method for locating the root cause of a server cluster fault according to claim 1, characterized in that, S5 includes the following steps: S51. When a fault alarm occurs, collect the original fault alarm data; compress the original fault alarm data through a three-level processing pipeline to obtain compressed fault alarm data. S52. Process the compressed fault alarm data through the vertical data layout converter to obtain cache-friendly bitmap index structure data. By carefully processing the cache-friendly bitmap index structure data using Trie tree compression storage technology and an 8-thread pool, highly compressed frequent pattern prefix tree data is obtained. The MapReduce pattern miner traverses the highly compressed frequent pattern prefix tree data to obtain a set of frequent itemsets; S53. Set the causal strength threshold; based on the dynamic weight values ​​of the edges in the dynamically updated inter-service calls and the exponential decay sum of historical edge failures, the causal strength value of each frequent itemset in the frequent itemset set is obtained through KL divergence processing. The causal strength value of each frequent itemset is compared with the causal strength threshold, and frequent itemsets with causal strength less than the causal strength threshold are removed to obtain frequent itemsets with strong causal associations. S54. Based on the alarm nodes in the frequent itemset with strong causal association, the Bayesian network is traversed by a reverse random walk sampler to obtain the node access frequency score. Based on the node access frequency score and the likelihood estimation of the Bayesian network, the probability distribution of candidate fault root causes is obtained through a joint probability calculator. S55. The root cause probabilities in the candidate fault root cause probability distribution are standardized by a probability normalization processor to obtain the final fault root cause probability distribution.

9. A method for locating the root cause of a server cluster fault according to claim 8, characterized in that, In step S51, the original fault alarm data is compressed through a three-stage processing pipeline to obtain the compressed fault alarm data, which includes the following steps: S511. Count and merge infrastructure alarms of the same type to obtain merged infrastructure alarms; S512. Based on the service-resource topology relationship, the container OOM alarm and the host memory pressure alarm are associated as a container resource squeeze combined event alarm; S513. By analyzing the business call chain, payment API delays, order DB timeouts, and Redis cache hits are integrated into transaction chain degradation semantic event alarms. S514. Compressed fault alarm data is obtained through S511, S512, and S513.

10. A fault root cause localization system for server clusters, characterized in that, The system implements a fault root cause localization method for server clusters as described in any one of claims 1-9, comprising a multi-source data acquisition and standardization processing module, a dynamic topology graph construction and update module, a causal reasoning network construction module, and a fault causal association mining and root cause localization module.

Images