Fault root cause positioning method and system for server cluster

Abstract

The invention discloses a fault root cause positioning method and system for a server cluster, and relates to the technical field of network fault diagnosis. According to the method, nanosecond-level synchronous acquisition of micro-service call chains, container indexes, physical nodes and network data is realized through a precise time protocol, and a consistent data set is constructed through entity association and standardized processing; a service-resource topological graph is dynamically constructed, and an inter-service calling edge weight model is innovatively designed: a real-time load factor and a historical fault index attenuation sum processed by a Sigmoid function are fused, and the weight is periodically updated to accurately quantify the inter-node influence intensity; converting the topological graph into a Bayesian network; when a fault occurs, a three-level assembly line compression alarm is adopted, frequent item sets are mined through bitmap indexes and parallel FP-Growth, and strong causal association item sets are screened in combination with topological edge weights and KL divergence; strong causal alarm is taken as evidence, probabilistic root cause sorting is output through reverse random walk sampling, and high-precision positioning of complex distributed system faults is achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network fault diagnosis, and particularly relates to a fault root cause positioning method and system for a server cluster. BACKGROUND

[0002] With the wide popularity of cloud computing and micro-service architecture, the scale of large data center server clusters has broken through the level of 100,000 nodes. The complex service dependency chain and hardware-software coupling relationship cause the fault propagation to present the characteristics of cascading and cross-layer. When facing the composite faults such as network jitter, storage performance degradation or container resource contention, the traditional monitoring system can only output discrete threshold alarms due to the lack of dynamic modeling capability of service-resource three-dimensional dependency topology, which causes the "alarm storm" to cover the real root cause. The industry still needs an intelligent root cause positioning scheme that integrates multi-source data real-time analysis, dynamic causal reasoning and self-optimization capability to solve the serious challenges such as high false alarm rate of massive alarms, low cross-layer fault positioning efficiency, and serious dependence on manual experience, and to meet the rigid demand for high-availability services in the fields of finance, e-commerce, cloud computing and the like.

[0003] The prior art has the problems of high false alarm rate, which submerges the real fault signal, cannot reflect the real-time service dependency relationship, low positioning efficiency, long average repair time and lack of closed-loop optimization mechanism; SUMMARY

[0004] (I) Technical problems solved In view of the problems in the related art, the application provides a fault root cause positioning method for a server cluster to overcome the above technical problems existing in the prior art.

[0005] (II) Technical solutions To solve the above technical problems, the application is implemented by the following technical solutions: S1, collect a multi-source server cluster dataset; standardize and process abnormal data of the multi-source server cluster dataset to obtain a processed multi-source server cluster dataset; S2, construct a topology graph; based on the processed multi-source server cluster dataset, obtain a basic node set, a state attribute set, a service inter-call edge and a service deployment relationship edge of the topology graph; S3, calculate the dynamic weight of the service inter-call edge by a dynamic edge weight calculation model to obtain a real-time edge weight; update the topology graph by a topology real-time updating algorithm combined with the real-time edge weight to obtain a dynamic topology graph; S4, convert the dynamic topology graph into a causal reasoning network and remove useless edges to obtain a key causal reasoning network; calculate the dynamic edge weight and the fault condition probability table of the key causal reasoning network to obtain a Bayesian network; S5, when a fault alarm occurs, collecting original fault alarm data; processing the original fault alarm data through a three-stage processing pipeline operation and an FP-Growth algorithm to obtain a strong causal correlation frequent item set; based on a Bayesian network and the strong causal correlation frequent item set, obtaining a final fault root cause probability distribution through a random walk model; The application constructs a global view through multi-source data standardization, creates a service-resource dynamic topology graph, and innovatively designs a real-time edge weight calculation model; when a fault occurs, a three-stage pipeline is used to compress an alarm storm, an FP-Growth algorithm and KL divergence causal analysis are combined to screen a strong correlation item set, a topology is converted into a Bayesian network, a probability is propagated through edge weight mapping, and a probability root cause is output through random walk reasoning; the problems of multi-source data dispersion, dynamic change dependence, alarm interference and inaccurate root cause positioning in a cloud native environment are solved, the fault diagnosis precision and efficiency are significantly improved, and the average repair time is reduced.

[0006] Preferably, the S1 comprises the following steps: S11, obtaining a multi-source server cluster dataset by collecting microservice call chain data, container resource index data, physical node index data and network state data of the server cluster; S12, synchronizing and aligning all data points in the multi-source server cluster dataset to nanoseconds based on a precision time protocol server to obtain an aligned multi-source server cluster dataset; analyzing the relationship between the data entities in the aligned multi-source server cluster dataset to obtain an analyzed multi-source server cluster dataset; unifying the same type index values of different sources in the analyzed multi-source server cluster dataset to a standard unit to obtain a unified multi-source server cluster dataset; performing format standardization and writing processing on the unified multi-source server cluster dataset to obtain a formatted multi-source server cluster dataset; S13, processing the formatted multi-source server cluster dataset through a long-short-time missing and conflict resolution algorithm to obtain a processed multi-source server cluster dataset; The application collects four types of data of microservice call chain, container, physical node and network, performs nanosecond timestamp alignment, entity relationship analysis, unit unification and format standardization, and uses linear interpolation to fill short-time missing, health check to cut path to fill long-time missing, and version vector to resolve conflicts, solves the problem of multi-source heterogeneous data dispersion and misplacement, and constructs a globally consistent high-quality dataset, laying a solid foundation for fault analysis.

[0007] Preferably, the S2 comprises the following steps: S21, construct a topology graph; obtain a basic node set of the topology graph by identifying unique identifiers and type information of different entities in the processed multi-source server cluster data set; the basic node set of the topology graph includes service nodes, container nodes, physical nodes and network nodes; By continuously receiving real-time monitoring index data of various nodes in the processed multi-source server cluster data set and performing association, the state attributes attached to each node are dynamically updated, and a dynamically updated state attribute set is obtained; S22, by analyzing the micro-service call chain data in the processed multi-source server cluster data set, the call relationship and the service name are identified, and the edges representing the service call are automatically created or updated, and the dynamically updated service call edges are obtained. By combining the service name and the container identifier in the call chain data in the processed multi-source server cluster data set and the container-host mapping established by the standardization engine, the service deployment relationship edge is automatically created, and the dynamically updated service deployment relationship edge is obtained. The application constructs a topology node set based on the identification of four types of entities, namely services, containers, physical nodes and networks, dynamically associates real-time monitoring indexes to update node state attributes, automatically generates service call edges through call chain analysis, creates service deployment edges in combination with container-host mapping, realizes dynamic visualization of system resources and dependency relationship, accurately maps complex architecture, and provides real-time topology basis for fault propagation analysis.

[0008] Preferably, the S3 comprises the following steps: S31, set the weight of the dynamically updated service deployment relationship edge to a fixed weight, and obtain a fixed service deployment relationship edge weight; S32, construct a dynamic edge weight calculation model; calculate the weight of the service call edge through the dynamic variable weight model, and obtain a real-time edge weight; S33, update the topology graph through a topology real-time updating algorithm, and obtain a dynamically updated topology graph; The application obtains a service call edge dynamic weight model by fusing real-time load factors and historical fault Sigmoid conversion influence factors, fixes the deployment edge weight, and updates the topology edge weight through a scheduler periodically; accurately quantifies the real-time influence intensity between nodes, upgrades the static topology to "dynamic blood vessels", and significantly improves the accuracy of fault propagation path analysis.

[0009] Preferably, the S32 comprises the following steps: S321, collect historical fault events of the service call edge and their occurrence times, and obtain historical edge fault data; Set a decay coefficient, and calculate the exponential decay sum of the historical edge fault through an exponential decay sum formula; S322, calculate a historical negative influence factor of the current weight based on historical edge failure exponential decay and through a Sigmoid function; Set a maximum capacity threshold of the inter-service call edge; based on the processed multi-source server cluster data set, obtain a real-time traffic metric on the inter-service call edge at the current time; Compare the real-time traffic metric on the inter-service call edge at the current time with the maximum capacity threshold of the inter-service call edge to obtain a normalized real-time load factor; Set weights of the historical negative influence factor and the real-time load factor respectively; based on the historical negative influence factor, the real-time load factor and the weights of the historical negative influence factor and the real-time load factor, calculate a dynamic weight value of the inter-service call edge; The application quantifies the influence of historical failures through an exponential decay formula, converts it into a dynamic negative influence factor through a Sigmoid function, calculates a normalized load factor by combining real-time traffic and a capacity threshold, and fuses the two factors by linear weighting to generate a dynamic weight of the inter-service call edge; the system real-time pressure and historical failure memory are reflected synchronously, the change of the influence strength between nodes is accurately described, and a quantitative basis is provided for failure propagation analysis.

[0010] Preferably, the S33 comprises the following steps: S331, check whether the entity ID corresponding to the data point in the processed multi-source server cluster data set exists in the graph, and when it does not exist, create a new node containing the type and initial state attributes; S332, set a relationship discovery rule to obtain a preset relationship rule; identify the type of the data point in the processed multi-source server cluster data set, and create a new edge or update the association information of the existing edge according to the preset relationship discovery rule; S333, deploy an independently running scheduler and set a traversal period; through the independently running scheduler, traverse all edges in the topology graph in each traversal period, and update the weight value of each edge; The application dynamically adds and deletes nodes through entity ID checking, automatically creates and updates edge association based on preset rules, and deploys an independent scheduler to periodically traverse and update the weight of all edges; realize real-time dynamic evolution of the topology graph, ensure real-time and accurate mapping of system state changes and dependency relationships, and provide adaptive updated core data support for failure analysis.

[0011] Preferably, the S4 comprises the following steps: S41, convert the dynamic topology graph into a causal reasoning network to obtain an initial causal reasoning network; Set a weight threshold; edges with a dynamic edge weight less than the weight threshold in the initial causal reasoning network are regarded as useless edges; remove the useless edges to obtain a key edge set and a key causal reasoning network. S42, convert the dynamic edge weight of each edge in the key edge set into a propagation intensity to obtain a propagation intensity set; Based on the propagation intensity set, the fault propagation probability of the key edge is calculated through the conditional probability formula to obtain a fault conditional probability table; S43, obtain the Bayesian network through S41 and S42; The application prunes the dynamic topology by filtering weak edges through a weight threshold, converts it into a key causal reasoning network, maps the edge weight to the propagation intensity through a Sigmoid function, and derives a fault conditional probability table to construct a Bayesian network.

[0012] Preferably, the S5 comprises the following steps: S51, when a fault alarm occurs, collect original fault alarm data; compress the original fault alarm data through a three-stage processing pipeline to obtain compressed fault alarm data; S52, process the compressed fault alarm data through a vertical data layout converter to obtain cache-friendly bitmap index structure data; Carefully process the cache-friendly bitmap index structure data through Trie tree compression storage technology + 8 thread pool processing to obtain highly compressed frequent pattern prefix tree data; Iterate the highly compressed frequent pattern prefix tree data through a MapReduce mode miner to obtain a frequent item set collection; S53, set a causal strength threshold; based on the dynamic weight value of the dynamically updated inter-service call edge and the exponential decay sum of historical edge faults, obtain the causal strength value of each frequent item set in the frequent item set collection through KL divergence processing; Compare the causal strength value of each frequent item set with the causal strength threshold, remove the frequent item sets with causal strength less than the causal strength threshold, and obtain the frequent item sets with strong causal correlation; S54, based on the alarm nodes in the frequent item sets with strong causal correlation, iterate the Bayesian network through a reverse random walk sampler to obtain node access frequency scores; Based on the node access frequency scores and the likelihood estimation of the Bayesian network, and through a joint probability calculator, obtain a candidate fault root cause probability distribution; S55, standardize the root cause probability in the candidate fault root cause probability distribution through a probability normalization processor to obtain a final fault root cause probability distribution; The application compresses original alarms through a three-stage pipeline, constructs a compressed FP-Tree by using a bitmap index and a multi-threaded Trie tree, mines frequent item sets in parallel FP-Growth, screens strong causal correlation item sets based on topological edge weight and historical fault index decay and calculates KL divergence strength, breaks through the limitations of traditional association rules, accurately identifies real fault causal chains, and significantly improves the efficiency and reliability of alarm analysis; takes the strong causal alarm item set as evidence, calculates node access frequency by using reverse random walk sampling, generates a candidate root cause distribution by combining Bayesian likelihood estimation and joint probability, and finally outputs a normalized probability ranking; high-precision root cause positioning in a complex dependency network is realized, the diagnostic accuracy is improved, and the mean time to repair is reduced.

[0013] Preferably, the compression of the original fault alarm data in S51 through a three-stage processing pipeline to obtain the compressed fault alarm data comprises the following steps: S511, counting and merging alarms of the same type of infrastructure to obtain merged infrastructure alarms; S512, based on the service-resource topology relationship, associating the container OOM alarm and the host memory pressure alarm as a container resource squeeze combination event alarm; S513, by analyzing the business call chain, fusing the payment API delay, order DB timeout and Redis cache hit into a transaction link degradation semantic event alarm; S514, through S511, S512 and S513, obtaining the compressed fault alarm data; The application merges alarms in a time window, generates combined events based on topology association, and compresses original alarms by aggregating semantic events in call chains; suppresses alarm storms, refines cross-layer business semantics, and provides high-value event chains for root cause positioning.

[0014] A fault root cause positioning system for a server cluster is used to implement the above-mentioned fault root cause positioning method for a server cluster, comprising a multi-source data acquisition and standardization processing module, a dynamic topology graph construction and updating module, a causal reasoning network construction module, and a fault causal correlation mining and root cause positioning module. The multi-source data acquisition and standardization processing module is used to collect original and heterogeneous monitoring data from a server cluster environment and strictly preprocess the data; based on the precise time protocol timestamp nanosecond level synchronization alignment, the relationship between data entities is analyzed, the index unit is unified, the data format is standardized, and the data quality problem is handled; the final output is a processed multi-source server cluster dataset that is strictly cleaned, aligned, standardized and consistent, providing reliable basic data for subsequent analysis.

[0015] The dynamic topology graph construction and updating module utilizes the processed multi-source server cluster data set to construct and maintain a topology graph reflecting the structure and state of the entire server cluster system in real time; first, basic nodes are identified and created, and real-time monitoring indicators are continuously associated to dynamically update the state attribute set of each node to reflect its health and performance; edges representing the calling relationship between services are automatically created and updated by analyzing the calling chain data, and edges representing the service deployment relationship are created in combination with the container mapping information; the dynamic edge weight is calculated according to the real-time traffic load factor and the exponential decay and dynamic calculation of historical fault events; the topology graph is dynamically tracked to obtain a dynamic topology graph containing real-time state and dynamic weight by periodically traversing all edges to update the weight through an independent scheduler; The causal reasoning network construction module converts the dynamic topology graph into a causal network for root cause reasoning; according to the set weight threshold, edges with excessively low dynamic weights are filtered out to obtain a simplified key causal reasoning network; the dynamic weight value of the key edges is mapped to a parameter representing the fault propagation intensity through a conversion function, and the conditional fault probability table of the key edges is calculated, thus constructing a Bayesian network with propagation probability; The fault causal correlation mining and root cause positioning module is used to receive the original alarm data stream when a fault occurs, and count and merge the same type of infrastructure alarms in a short time window; related alarms are associated to form a combined event based on the service-resource topology relationship; multiple service / component alarms affecting the same business link are fused into events with business semantics through business calling chain analysis; the compressed alarm data is converted into a bitmap index structure, and a highly compressed frequent pattern prefix tree is constructed using Trie tree compression technology and multi-thread processing; the MapReduce pattern miner is used to mine the frequently co-occurring alarm patterns from the tree; the edge weights in the dynamic topology graph and the historical fault exponential decay and are used to quantify the causal influence between elements in the item set through the KL divergence method, obtaining a frequent item set that removes weakly associated items and retains strong causal correlations; the strong causal correlation frequent item set is input, and the process of simulating the fault from the alarm node to the potential root cause node is recorded, and the access frequency of each node is recorded as its root cause score; the joint probability calculation is performed in combination with the likelihood estimation of the Bayesian network to obtain the probability distribution of the candidate root cause; through probability normalization processing, the final fault root cause probability distribution is obtained, which clearly indicates the most likely fault source and its probability.

[0016] (Three) Beneficial effects The present application has the following beneficial effects: Through the whole-chain innovation of "multi-source fusion-dynamic topology-cause mining-probability reasoning", the present application significantly improves the automation, intelligence and precision level of cloud native system fault diagnosis, effectively reduces the average repair time and guarantees business continuity.

[0017] At the data level, the application synchronizes through nanosecond timestamps, analyzes entity relationships, normalizes units, and intelligently handles data missing conflicts, thereby constructing a high-quality, strongly consistent multi-source fusion dataset, laying a reliable foundation for upper-layer analysis and solving the "blind man touching an elephant" problem caused by scattered and misplaced data. At the system modeling level, the application innovatively constructs a dynamic topology graph with service-resource dual-dimension dependence. This graph not only includes full-factor nodes such as services, containers, physical nodes, network devices, and their real-time state attributes, but also distinguishes between dynamic call edges between services and static deployment edges of services / containers, accurately depicting the system structure. A weight calculation model based on historical fault exponential decay and real-time normalized load factors is designed. The historical negative impact factor is converted into an impact value in the (0, 1) interval through a Sigmoid function, allowing the weight to reflect not only the current traffic pressure but also the proximity and severity of historical faults. The weight is periodically updated by an independent scheduler to ensure that the topology always tracks the latest system state. This upgrades the topology graph from a "static skeleton" to a "dynamic blood vessel", accurately quantifying the real-time influence strength between components.

[0018] At the alarm processing level, the application innovatively constructs a three-stage pipeline compressor mechanism. Redundancy is reduced by time window counting, combined events are constructed by topology association, and semantic events are generated by calling chain fusion, significantly suppressing alarm storms and improving semantic value. Frequent item sets are efficiently mined through bitmap indexing, Trie tree compression, and parallel FP-Growth, and the causal strength is innovatively calculated by combining topology edge weights and KL divergence, filtering strong causal correlation item sets, and breaking through the limitations of traditional association rules that only mine co-occurrence and ignore causality.

[0019] At the root cause positioning level, the dynamic topology is converted into a causal reasoning network, weak edges are pruned according to weight thresholds to retain key causal paths, edge weights are converted into propagation strengths through a function, and a conditional probability table is derived to construct a Bayesian network with quantifiable propagation probabilities. Strong causal alarm item sets are used as evidence to simulate fault propagation through reverse random walk sampling, and candidate root cause distribution is generated by combining Bayesian likelihood estimation and joint probability calculation. Finally, the probability-based root cause ranking is normalized and output. This upgrades experience-driven to data probability-driven, achieving high-precision and interpretable root cause positioning in complex network topologies.

[0020] Of course, implementing any product of the application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments. Obviously, the drawings described in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0022] Figure 1 A flowchart of a fault root cause positioning method for a server cluster according to the application; Figure 2 A flowchart of original fault alarm compression in a fault root cause positioning method for a server cluster according to the application; Figure 3 A module diagram of a fault root cause positioning system for a server cluster according to the application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, but not all the embodiments. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0024] In the description of the application, it should be understood that the terms "opening", "upper", "lower", "top", "middle", "inner" and the like indicate the orientation or positional relationship, which are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the indicated component or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the application.

[0025] Embodiment one: please refer to Figure 1 、 Figure 2 The application discloses a fault root cause positioning method for a server cluster, comprising the following steps: S1, collecting a multi-source server cluster dataset; performing standardization processing and abnormal data processing on the multi-source server cluster dataset to obtain a processed multi-source server cluster dataset; The S1 comprises the following steps: S11, obtaining the multi-source server cluster dataset by collecting micro-service call chain data, container resource index data, physical node index data and network state data of the server cluster; The S11 comprises the following steps: Full call chain collection is performed by OpenTelemetry Agent deployed on each microservice. Key information captured includes unique trace identifier, call relationship, service name, and call delay, resulting in microservice call chain data. Container-level resource usage is collected using cAdvisor combined with Kubernetes API. Key information captured includes container identifier, CPU usage, and memory working set size, resulting in container resource indicator data. Hardware and operating system indicators of the host are collected through Node Exporter and IPMI tools. Key information captured includes host IP, network transmission traffic, disk I / O latency, and physical node indicators. Syslog forwarding is used to receive log events, and SNMP Trap mechanism is used to capture key state change events of network equipment. Key information captured includes device IP, port state, and packet loss rate, resulting in network state data. S12. The timestamps of all data points in the multi-source server cluster dataset are synchronized to nanosecond level based on the precise time protocol server, resulting in the aligned multi-source server cluster dataset. This ensures the time consistency of cross-source data.

[0026] The relationships between data entities in the aligned multi-source server cluster dataset are analyzed, resulting in the parsed multi-source server cluster dataset. For example, for container indicator data, it is mapped to the host it belongs to by querying the Kubernetes API, establishing the association between the container and the physical resource. The same type of indicator values from different sources in the parsed multi-source server cluster dataset are unified to standard units, resulting in the unified multi-source server cluster dataset. For example, all memory-related indicators are converted to megabytes. Format standardization and writing processing are performed on the unified multi-source server cluster dataset, resulting in the formatted multi-source server cluster dataset. S13. The formatted multi-source server cluster dataset is processed through a long-short-time missing and conflict resolution algorithm, resulting in the processed multi-source server cluster dataset. Short-time missing is compensated by estimating the missing value based on the valid data points before and after the missing point in the formatted multi-source server cluster dataset using linear interpolation algorithm, maintaining the continuity of the data stream. Long-time missing triggers a data source health check process. The system automatically diagnoses the problem source and attempts to switch to a pre-configured backup collector or path to restore the missing data stream of the formatted multi-source server cluster dataset. In the scenario of distributed collection that may cause version conflicts in the formatted multi-source server cluster dataset (such as multiple collection points reporting different states of the same entity at the same time), the system adopts a conflict resolution algorithm based on version vectors to effectively identify the causality and concurrency of events, intelligently resolve conflicts, and ensure the consistency of the final data view.

[0027] S2, constructing a topology graph; based on the processed multi-source server cluster dataset, obtaining a basic node set, a state attribute set, an inter-service calling edge, and a service deployment relationship edge of the topology graph; The S2 includes the following steps: S21, constructing a topology graph; by identifying the unique identifiers and type information of different entities in the processed multi-source server cluster dataset, obtaining a basic node set of the topology graph; the basic node set of the topology graph includes service nodes, container nodes, physical nodes, and network nodes; By continuously receiving real-time monitoring index data (such as QPS / error rate / P99 delay of services, CPU / memory / network usage of containers, CPU load / disk utilization / temperature of physical nodes, and packet loss / RTT of network ports) of various nodes in the processed multi-source server cluster dataset and performing association, the "state attribute set" attached to each node is dynamically updated to obtain a dynamically updated state attribute set; the nodes can reflect the health and performance status of system components in real time; S22, by analyzing the micro-service calling chain data in the processed multi-source server cluster dataset, identifying the calling relationship and the service name calling relationship, automatically creating or updating edges representing inter-service calling, and obtaining dynamically updated inter-service calling edges; the source node of the dynamically updated edge is the parent service, and the target node is the called service; By combining the service name and container identifier in the calling chain data in the processed multi-source server cluster dataset and the container-host mapping established by the standardization engine, automatically creating edges representing service deployment relationship, and obtaining dynamically updated service deployment relationship edges; the source node is the service, and the target node is the container Pod (the weight is fixed at 1.0, indicating a strong association; the edges between physical nodes and containers, and network nodes and physical nodes / containers are usually automatically associated through resource ownership information, which is not detailed in the example code but has similar logic); S3, dynamically calculating the weight of the inter-service calling edge through a dynamic edge weight calculation model to obtain real-time edge weight; By combining the real-time edge weight, the topology is updated through a topology real-time updating algorithm to obtain a dynamic topology graph; S31, set the weight of the edge of the dynamically updated service deployment relationship to a fixed weight to obtain a fixed service deployment relationship edge weight; the service calling edge needs to reflect real-time business traffic (such as a sudden increase in QPS leading to an increase in delay), and the deployment edge represents a static attribution relationship; S32, construct a dynamic edge weight calculation model; calculate the weight of the service calling edge through the dynamic variable weight model to obtain a real-time edge weight; The S32 includes the following steps: S321, collect historical fault events of the service calling edge and their occurrence times to obtain historical edge fault data; Set the decay coefficient as λ, and calculate the exponential decay sum of the historical edge fault through an exponential decay sum formula; the exponential decay sum formula is as follows, ; Wherein, F ( t ) represents the exponential decay sum of the historical edge fault (that is, the cumulative value of the historical fault influence), t represents the current time, t i represents the occurrence time of the historical i th fault event related to the edge, delta i represents the weight of the set i th fault event (used to represent the severity or influence of the fault); S322, based on the exponential decay sum of the historical edge fault, calculate the historical negative influence factor of the historical fault on the current weight through a Sigmoid function (the more and the closer the historical faults, the greater the negative influence); the Sigmoid function calculation formula is as follows, ; Wherein, d represents the negative influence factor of the historical fault on the current weight (the value range is (0, 1), and the greater the value represents the greater the influence of the historical fault), k represents the slope parameter of the set Sigmoid function (used to control the steepness of the curve), e represents the base number of natural logarithm; Set the maximum capacity threshold of the service calling edge; based on the processed multi-source server cluster data set, obtain the real-time traffic metric (such as the number of calls, bandwidth) on the service calling edge at the current time; Compare the real-time traffic metric on the service calling edge at the current time with the maximum capacity threshold of the service calling edge to obtain a normalized real-time load factor (the higher the real-time load, the greater the factor); respectively, and the weight of the historical negative influence factor and the real-time load factor is set respectively; a dynamic weight value of the inter-service calling edge is calculated based on the historical negative influence factor, the real-time load factor, and the weight of the historical negative influence factor and the real-time load factor; the calculation formula is as follows, ; wherein, w represents the dynamic weight value of the dynamically updated inter-service calling edge, f represents the normalized load factor, α , β respectively, and the weight of the historical negative influence factor and the real-time load factor is set respectively; a dynamic weight value of the inter-service calling edge is calculated based on the historical negative influence factor, the real-time load factor, and the weight of the historical negative influence factor and the real-time load factor; the calculation formula is as follows, S33, updating the topology graph by the topology real-time updating algorithm to obtain a dynamically updated topology graph; The S33 comprises the following steps: S331, checking whether the entity ID corresponding to the data point in the processed multi-source server cluster data set exists in the graph, and creating a new node containing the type and initial state attributes when it does not exist; S332, setting a relationship discovery rule (such as a service calling chain, a service-container deployment) to obtain a preset relationship rule; identifying the type (such as a TRACE_SPAN) of the data point in the processed multi-source server cluster data set, and creating a new edge or updating the association information (especially the dynamic weight, which is calculated by calling the CalculateWeight(edge) function by applying the aforementioned mathematical model) of the existing edge according to the preset relationship discovery rule; S333, deploying an independently running scheduler and setting a traversal period; traversing all edges in the topology graph under each traversal period (such as one hour) by the independently running scheduler, and updating the weight value of each edge (which ensures that the influence of historical failures gradually weakens and decreases over time, so that the topology weight can dynamically track the latest state of the system); F t ); S4, converting the dynamic topology graph into a causal reasoning network and removing useless edges to obtain a key causal reasoning network; calculating the dynamic edge weight and the fault condition probability table of the key causal reasoning network to obtain a Bayesian network; The S4 comprises the following steps: S41, converting the dynamic topology graph into a causal reasoning network to obtain an initial causal reasoning network; setting a weight threshold; regarding the edges with a dynamic edge weight less than the weight threshold in the initial causal reasoning network as useless edges; removing the useless edges to obtain a key edge set and a key causal reasoning network; S42, converting the dynamic edge weight of each edge in the key edge set into a propagation intensity to obtain a propagation intensity set; the conversion formula is as follows,​ ; in, theta The propagation strength of each edge in the key edge set is represented by w(t), and the dynamic weight value of each edge in the key edge set is represented by w(t). Based on the propagation intensity set, the failure propagation probability of the critical edge is calculated using the conditional probability formula, resulting in a failure conditional probability table. The conditional probability formula is as follows: ; in, P ( a | b ) indicates a fault a Parent node in key causal reasoning network b The probability of fault propagation, theta mi This represents the first key causal reasoning network. i Parent node to node m The intensity of fault propagation; S43. Obtain the Bayesian network through S41 and S42; S5. When a fault alarm occurs, collect the original fault alarm data; process the original fault alarm data through a three-level processing pipeline operation and the FP-Growth algorithm to obtain frequent itemsets with strong causal association; based on Bayesian network and frequent itemsets with strong causal association, obtain the final fault root cause probability distribution through a random walk model. S5 includes the following steps: S51. When a fault alarm occurs, collect the original fault alarm data; compress the original fault alarm data through a three-level processing pipeline to obtain compressed fault alarm data. S51 includes the following steps: S511. Count and merge the same type of infrastructure alarms (such as continuous CPU overload and high disk load) within a 5-second window to obtain the merged infrastructure alarms. S512. Based on the service-resource topology relationship, the container OOM alarm and the host memory pressure alarm are associated as a container resource squeeze combined event alarm; S513. By analyzing the business call chain, payment API delays, order DB timeouts, and Redis cache hits are integrated into transaction chain degradation semantic event alarms. S514. Compressed fault alarm data is obtained through S511, S512, and S513. S52. Process the compressed fault alarm data through the vertical data layout converter to obtain cache-friendly bitmap index structure data. Through the Trie tree compression storage technology + 8 thread pool processing, the cache-friendly bitmap index structure data is carefully processed, and the highly compressed frequent pattern prefix tree data is obtained. Through the MapReduce mode miner, the highly compressed frequent pattern prefix tree data is traversed, and the frequent item set collection is obtained. S53, set the causal strength threshold; based on the dynamic weight value of the edge of the inter-service call and the exponential decay of the historical edge fault, the causal strength value of each frequent item set in the frequent item set collection is obtained through KL divergence processing; Compare the causal strength value of each frequent item set with the causal strength threshold, remove the frequent item set with the causal strength less than the causal strength threshold, and obtain the frequent item set with strong causal correlation; S54, based on the alarm node in the frequent item set with strong causal correlation, the node access frequency score is obtained by traversing the Bayesian network through the reverse random walk sampler; Based on the node access frequency score and the Bayesian network likelihood estimation, and through the joint probability calculator, the candidate fault root cause probability distribution is obtained; S55, the root cause probability in the candidate fault root cause probability distribution is standardized by the probability normalization processor, and the final fault root cause probability distribution is obtained.

[0028] Embodiment two: please refer to Figure 3 A fault root cause positioning system for a server cluster, for implementing the above-mentioned fault root cause positioning method for a server cluster, comprising a multi-source data acquisition and standardization processing module, a dynamic topology graph construction and update module, a causal reasoning network construction module, and a fault causal correlation mining and root cause positioning module. The multi-source data acquisition and standardization processing module is used to collect original and heterogeneous monitoring data from the server cluster environment, and to strictly preprocess it; based on the time stamp nanosecond level synchronization alignment of the precision time protocol, the relationship analysis between data entities, the index unit unification, the data format standardization, and the data quality problem processing; the final output is the processed multi-source server cluster data set after strict cleaning, alignment, standardization and consistency, providing reliable basic data for subsequent analysis.

[0029] The dynamic topology graph construction and updating module uses the processed multi-source server cluster data set to construct and maintain a topology graph reflecting the entire server cluster system structure and state in real time; first, the basic nodes are identified and created, and the real-time monitoring indicators are continuously associated to dynamically update the state attribute set of each node to reflect its health and performance; the edges representing the service-to-service calling relationship are automatically created and updated by analyzing the calling chain data, and the edges representing the service deployment relationship are created in combination with the container mapping information; the dynamic edge weights are calculated according to the real-time traffic load factor and the exponential decay and dynamic calculation of historical fault events; the independent scheduler periodically traverses all edges to update the weights, ensuring that the topology graph can dynamically track the latest state of the system, and a dynamic topology graph containing real-time state and dynamic weights is obtained; The causal reasoning network construction module converts the dynamic topology graph into a causal network for root cause reasoning; according to the set weight threshold, the edges with too low dynamic weights are filtered out to obtain a simplified key causal reasoning network; the dynamic weight values of the key edges are mapped to parameters representing the fault propagation intensity through a conversion function, and the conditional fault probability table of the key edges is calculated, thus constructing a Bayesian network with propagation probability; The fault causal correlation mining and root cause positioning module is used to receive the original alarm data stream when a fault occurs, and count and merge the same type of infrastructure alarms in a short time window; based on the service-resource topology relationship, the related alarms are associated to form a combined event; through business calling chain analysis, multiple service / component alarms affecting the same business link are fused into an event with business semantics; the compressed alarm data is converted into a bitmap index structure, and a highly compressed frequent pattern prefix tree is constructed using Trie tree compression technology and multi-thread processing; the MapReduce pattern miner is used to mine the frequent co-occurring alarm patterns from the tree; using the edge weights in the dynamic topology graph and the historical fault exponential decay and, the causal influence between elements in the item set is quantified by the KL divergence method, and the frequent item set with strong causal correlation is obtained by removing weakly associated items; the strong causal correlation frequent item set is used as input, and the process of simulating the fault from the alarm node to the potential root cause node is recorded, and the access frequency of each node is recorded as its root cause score; the joint probability calculation is performed in combination with the likelihood estimation of the Bayesian network to obtain the probability distribution of the candidate root cause; through probability normalization processing, the final fault root cause probability distribution is obtained, which clearly indicates the most likely fault source and its probability.

[0030] In the description of the specification, reference to "one embodiment", "an example", "a specific example" or the like means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The appearances of the phrases "in one embodiment", "an example", "a specific example" or the like in various places in the specification are not necessarily referring to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0031] The preferred embodiments of the application disclosed above are only to help explain the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the specification. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and use the application.

Claims

1. A method for locating the root cause of a fault in a server cluster, characterized in that, The method comprises the following steps: S1, collecting a multi-source server cluster dataset; performing standardization processing and abnormal data processing on the multi-source server cluster dataset to obtain a processed multi-source server cluster dataset; S2, constructing a topology graph; Based on the processed multi-source server cluster dataset, the base node set, the state attribute set, the inter-service calling edge and the service deployment relationship edge of the topology graph are obtained; S3, performing dynamic weight calculation on the inter-service calling edge through a dynamic edge weight calculation model to obtain real-time edge weights; The topology graph is updated through a topology real-time updating algorithm combined with the real-time edge weights to obtain a dynamic topology graph; S4, converting the dynamic topology graph into a causal reasoning network and removing useless edges to obtain a key causal reasoning network; calculating the dynamic edge weights and the fault condition probability table of the key causal reasoning network to obtain a Bayesian network; S5, when a fault alarm occurs, collecting original fault alarm data; processing the original fault alarm data through a three-stage processing pipeline operation and an FP-Growth algorithm to obtain a frequent item set with strong causal association; based on the Bayesian network and the frequent item set with strong causal association, an ultimate fault root cause probability distribution is obtained through a random walk model.

2. The method for fault root cause localization for a server cluster of claim 1, wherein, The S1 comprises the following steps: S11, obtaining a multi-source server cluster dataset by collecting micro-service calling chain data, container resource index data, physical node index data and network state data of a server cluster; S12, synchronously aligning all data points in the multi-source server cluster dataset at a nanosecond level based on a precision time protocol server to obtain an aligned multi-source server cluster dataset; Analyzing the relationship between data entities in the aligned multi-source server cluster dataset to obtain an analyzed multi-source server cluster dataset; Uniforming the same type index values of different sources in the analyzed multi-source server cluster dataset to a standard unit to obtain a uniformed multi-source server cluster dataset; Performing format standardization and writing processing on the uniformed multi-source server cluster dataset to obtain a formatted multi-source server cluster dataset; S13, processing the formatted multi-source server cluster dataset through a long-short-time missing and conflict resolution algorithm to obtain a processed multi-source server cluster dataset.

3. The method of claim 1, wherein, The S2 comprises the following steps: S21, constructing a topology graph; obtaining a base node set of the topology graph by identifying unique identifiers and type information of different entities in the processed multi-source server cluster dataset; the base node set of the topology graph comprises service nodes, container nodes, physical nodes and network nodes; Dynamically updating state attributes attached to each node by continuously receiving real-time monitoring index data of various nodes in the processed multi-source server cluster dataset and performing association to obtain a dynamically updated state attribute set; S22, identifying calling relationships and service names by analyzing micro-service calling chain data in the processed multi-source server cluster dataset, automatically creating or updating edges representing inter-service calling to obtain dynamically updated inter-service calling edges; By combining the service name and container identifier in the centralized invocation chain data of the processed multi-source server cluster data set and the container-host mapping established by the standardization engine, a service deployment relationship edge is automatically created, and a dynamically updated service deployment relationship edge is obtained.

4. The method of claim 1, wherein, The S3 includes the following steps: S31, setting the weight of the dynamically updated service deployment relationship edge as a fixed weight to obtain a fixed service deployment relationship edge weight; S32, constructing a dynamic edge weight calculation model; calculating the weight of the inter-service invocation edge through the dynamic variable weight model to obtain a real-time edge weight; S33, updating the topology graph through a topology real-time updating algorithm to obtain a dynamically updated topology graph.

5. The method for fault root cause localization for a server cluster of claim 4, wherein, The S32 includes the following steps: S321, collecting historical fault events of the inter-service invocation edge and their occurrence times to obtain historical edge fault data; Setting a decay coefficient, calculating the exponential decay sum of the historical edge fault through an exponential decay sum formula; S322, calculating the historical negative influence factor of the current weight based on the exponential decay sum of the historical edge fault through a Sigmoid function; Setting a maximum capacity threshold of the inter-service invocation edge; based on the processed multi-source server cluster data set, obtaining a real-time traffic metric on the inter-service invocation edge at the current time; Comparing the real-time traffic metric on the inter-service invocation edge at the current time with the maximum capacity threshold of the inter-service invocation edge to obtain a normalized real-time load factor; Respectively setting the weights of the historical negative influence factor and the real-time load factor; based on the historical negative influence factor, the real-time load factor, and the weights of the historical negative influence factor and the real-time load factor, calculating a dynamic weight value of the inter-service invocation edge.

6. The method for fault root cause localization for a server cluster of claim 4, wherein, The S33 includes the following steps: S331, checking whether the entity ID corresponding to the data point in the processed multi-source server cluster data set exists in the graph, and creating a new node containing the type and initial state attributes when it does not exist; S332, setting relationship discovery rules to obtain preset relationship rules; identifying the types of data points in the processed multi-source server cluster data set, and creating new edges or updating the association information of existing edges according to the preset relationship discovery rules; S333, deploying an independently running scheduler and setting a traversal period; through the independently running scheduler, traversing all edges in the topology graph in each traversal period, and updating the weight value of each edge.

7. The method of claim 1, wherein, The S4 includes the following steps: S41, converting the dynamic topology graph into a causal reasoning network to obtain an initial causal reasoning network; Setting a weight threshold; regarding the edges with a dynamic edge weight less than the weight threshold in the initial causal reasoning network as useless edges; removing the useless edges to obtain a key edge set and a key causal reasoning network; S42, converting the dynamic edge weight of each edge in the key edge set into a propagation intensity to obtain a propagation intensity set; Based on the propagation intensity set, calculating the fault propagation probability of the key edge through a conditional probability formula to obtain a fault conditional probability table; S43, obtaining a Bayesian network through S41 and S42.

8. The method of claim 1, wherein, The S5 includes the following steps: S51, collecting original fault alarm data when a fault alarm occurs; compressing the original fault alarm data through a three-stage processing pipeline to obtain compressed fault alarm data; S52, processing the compressed fault alarm data through a vertical data layout converter to obtain cache-friendly bitmap index structure data; carefully processing the cache-friendly bitmap index structure data through a Trie tree compression storage technology + 8 thread pool processing to obtain highly compressed frequent pattern prefix tree data; traversing the highly compressed frequent pattern prefix tree data through a MapReduce mode miner to obtain a frequent item set collection; S53, setting a causal strength threshold; based on the dynamic weight values of the edges of the dynamically updated inter-service calls and the exponential decay sum of historical edge faults, obtaining the causal strength value of each frequent item set in the frequent item set collection through KL divergence processing; comparing the causal strength value of each frequent item set with the causal strength threshold, removing the frequent item sets with a causal strength less than the causal strength threshold, and obtaining strongly causally related frequent item sets; S54, based on the alarm nodes in the strongly causally related frequent item sets, traversing the Bayesian network through a reverse random walk sampler to obtain node access frequency scores; based on the node access frequency scores and the likelihood estimation of the Bayesian network, and through a joint probability calculator, obtaining a candidate fault root cause probability distribution; S55, standardizing the root cause probability in the candidate fault root cause probability distribution through a probability normalization processor to obtain a final fault root cause probability distribution.

9. The method for fault root cause localization for a server cluster of claim 8, wherein, The S51 of compressing the original fault alarm data through a three-stage processing pipeline to obtain compressed fault alarm data includes the following steps: S511, counting and merging the same type of infrastructure alarms to obtain merged infrastructure alarms; S512, based on the service-resource topology relationship, associating the container OOM alarm and the host memory pressure alarm as a container resource squeeze combination event alarm; S513, through business call chain analysis, merging payment API delay, order DB timeout and Redis cache hit into transaction link degradation semantic event alarms; S514, obtaining the compressed fault alarm data through S511, S512 and S513.

10. A fault root cause localization system for server clusters, characterized in that, Implement a fault root cause positioning method for a server cluster as claimed in any one of claims 1-9, the system comprising a multi-source data acquisition and standardization processing module, a dynamic topology graph construction and update module, a causal reasoning network construction module, and a fault causal correlation mining and root cause positioning module.

Images