Method for fault locating of a baseboard management controller and electronic device

Abstract

The application discloses a fault positioning method of a baseboard management controller and electronic equipment, relates to the technical field of baseboard management controllers, and comprises collecting multi-source heterogeneous logs and extracting key events, breaking through the limitation of traditional technologies which only rely on a single log data source; through time sequence correlation analysis on the key events and construction of a time sequence correlation network, the occurrence time sequence logic of various fault events is sorted out, the inherent causal relationship in events is mined and identified from the time sequence correlation network, the defect that traditional time sequence machine learning methods cannot determine causality is abandoned, and a preliminary causal diagram reflecting fault propagation paths is constructed; the preliminary causal diagram is fused and enhanced with a predefined fault knowledge base, and the knowledge short board of simple causal reasoning is made up; the fault causal diagram is deduced by a fault alarm event, and a fault propagation link is completely restored, so that the core technical problems of the traditional fault positioning technology, i.e., incomplete restoration of fault propagation chains and low fault positioning accuracy, are fundamentally solved.

Description

Technical Field

[0001] This application relates to the field of substrate management controller technology, and in particular to a fault location method and electronic device for substrate management controller. Background Technology

[0002] Baseboard Management Controller (BMC) failures often propagate across modules and levels. A single fault can trigger multiple types of abnormal events. Accurately tracing the complete fault propagation chain and pinpointing the root cause is a core technical requirement for achieving efficient server operation and maintenance and avoiding business interruptions. Currently, mainstream BMC fault location technologies in the industry mostly employ solutions such as single log analysis, manual rule matching, pure time-series correlation analysis, or static knowledge base matching for fault location, but these cannot restore the hardware. firmware system The complete fault propagation chain in the business leads to a high rate of misjudgment in fault location and insufficient location accuracy. Summary of the Invention

[0003] This application provides a fault location method and electronic device for a baseboard management controller, to at least solve the problem of hardware incomprehensibility in related technologies. firmware system The complete fault propagation chain in the business leads to technical problems such as a high rate of misjudgment in fault location and insufficient location accuracy.

[0004] This application provides a fault location method for a baseboard management controller, comprising: acquiring multi-source heterogeneous logs and extracting key events from the multi-source heterogeneous logs; performing temporal correlation analysis on the key events to construct a temporal correlation network between events; identifying causal relationships from the temporal correlation network to generate a preliminary causal graph reflecting the fault propagation path; aligning and fusing the preliminary causal graph with a predefined fault knowledge base to obtain a knowledge-enhanced fault causal graph; and determining the fault propagation link based on the fault alarm event and the fault causal graph when a fault alarm event occurs.

[0005] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the fault location method of any of the above-described baseboard management controllers.

[0006] This application also provides a fault location device for a baseboard management controller, comprising: The acquisition and extraction module is used to acquire multi-source heterogeneous logs and extract key events from them. The temporal correlation analysis module is used to perform temporal correlation analysis on key events and construct a temporal correlation network between events; The causal relationship identification module is used to identify causal relationships from the temporal correlation network and generate a preliminary causal graph reflecting the fault propagation path; The knowledge enhancement module is used to align and merge the initial cause-effect graph with a predefined fault knowledge base to obtain a knowledge-enhanced fault cause-effect graph. The location module is used to determine the fault propagation path based on the fault alarm event and the fault cause-effect graph when a fault alarm event occurs.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the fault location method of the aforementioned baseboard management controller.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the fault location method for the aforementioned baseboard management controller.

[0009] This application provides a fault location method for a baseboard management controller (BMC). This method overcomes the limitations of traditional technologies that rely solely on a single log data source by collecting multi-source heterogeneous logs and extracting categorized key events. It achieves full-dimensional fault data coverage across modules and avoids the problem of misjudgment of cross-module faults caused by single hardware log analysis. By conducting time-series correlation analysis on key events and constructing a time-series correlation network, the method accurately sorts out the occurrence sequence logic of various fault events. Furthermore, it mines and identifies the inherent causal relationships of events from the time-series correlation network, overcoming the shortcomings of traditional time-series machine learning methods that can only identify correlations but cannot determine causality. It constructs a preliminary causal graph that can reflect the fault propagation path. By integrating the preliminary causal graph with a predefined fault knowledge base, it compensates for the knowledge shortcomings of simple causal reasoning and the inability of static knowledge bases to combine with real-time log dynamic analysis and adapt to new faults. Finally, by injecting fault alarm events to drive the fault causal graph deduction, it completely restores the fault propagation chain at all levels from hardware, firmware, system to business. This fundamentally solves the core technical problems of traditional BMC fault location technology, such as the inability to integrate multi-source logs, the inability to distinguish causal relationships of events, incomplete restoration of the fault propagation chain, and low fault location accuracy. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1A schematic diagram of the specific application environment architecture on which the execution of a fault location method for a baseboard management controller provided in this application depends; Figure 2 A flowchart illustrating a fault location method for a baseboard management controller provided in an embodiment of this application; Figure 3 A flowchart illustrating another fault location method for a baseboard management controller provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of a fault location device for a baseboard management controller provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] The Server Baseboard Management Controller (BMC) is the core control unit for server hardware management. It undertakes core functions such as server hardware status monitoring, anomaly alarms, and fault control. Its operational stability directly determines the reliability of the entire server and upper-layer business services.

[0015] The generation and propagation of BMC (Browser Control Center) failures involve multi-level and multi-module interactions, including hardware components, firmware, operating systems, and upper-layer application services. The failure propagation path is complex, and the causes are often concealed, making it a typical cross-level and cross-module complex failure scenario. Accurate and efficient BMC failure localization is a key technical support for quickly troubleshooting server anomalies, ensuring the stable operation of data center server clusters, and reducing the risk of business interruption. Therefore, the industry's demand for intelligent and precise root cause localization technology for BMC failures is becoming increasingly urgent.

[0016] Current BMC fault location technologies still have many shortcomings and are difficult to adapt to the needs of server fault diagnosis in complex scenarios. Mainstream technologies can be divided into four categories. The first category is analysis solutions based on single BMC hardware logs, which only collect and parse BMC hardware log data for fault analysis, without incorporating related log data from the operating system, upper-layer application services, etc. The second category is fault location solutions based on manually defined rule matching, relying on a manually compiled fault rule base to complete fault identification and judgment. The third category is fault analysis solutions based on temporal correlation machine learning, which achieves anomaly identification by mining the temporal correlation features of fault events. The fourth category is fault location solutions based on static knowledge base matching, relying on a fixedly constructed fault pattern knowledge base to complete fault comparison and location. In addition, some targeted BMC log processing and fault diagnosis technologies can only achieve basic functions such as centralized storage and management of BMC logs, structured parsing of multi-source heterogeneous logs, and root cause matching using preset indicator dependency graphs, covering only basic aspects such as log storage, format parsing, and simple pattern matching.

[0017] The aforementioned technical solutions have significant defects and limitations in practical applications, failing to meet the requirements for high-precision, full-scenario, and intelligent root cause localization of BMC hardware failures. First, single log analysis technologies have a limited data source dimension, covering only BMC hardware logs and lacking support from system and business log data. This makes them unsuitable for cross-module fault investigation scenarios and highly prone to misjudgment. For example, when application anomalies cause high CPU load and trigger BMC alarms, the business-level cause may be incorrectly identified as a hardware failure. Second, rule-matching technologies heavily rely on manually predefined rules, resulting in limited rule coverage and poor universality. They completely fail when faced with new and unknown faults, and the manual costs of rule iteration and maintenance increase dramatically with the expansion of server clusters, leading to extremely high operational costs. Third, temporal correlation machine learning technologies can only uncover the temporal correlation of fault events, failing to identify causal logic between events. This results in serious root cause determination bias. When multiple abnormal events are triggered by the same underlying cause, the causal relationship between events is easily misjudged, leading to incorrect root cause localization. Meanwhile, static knowledge base matching technology relies on fixed fault modes for comparison, which cannot dynamically update the fault knowledge system by combining real-time log data. It is difficult to adapt to the scenarios of server hardware configuration iteration and new faults, and the problem of fault knowledge update lag is prominent.

[0018] Furthermore, related technologies generally fail to address the challenges of event correlation caused by heterogeneous formats and inconsistent time bases in multi-source logs within the BMC (Browser Control Center). This makes it difficult to effectively integrate and align multi-level log data, and to accurately distinguish the temporal correlations and causal relationships of fault events. Moreover, most causal graph root cause localization technologies focus on pure business scenarios such as distributed software, microservices, and cloud-native environments, leaving a gap in specialized causal reasoning technologies for server BMC hardware faults. Related BMC technologies are limited to basic functions such as log collection, storage, and simple rule-based alarms, failing to achieve deep integration of multi-source log temporal alignment, heterogeneous fusion, and causal graph reasoning technologies. This makes it impossible to reconstruct the entire fault propagation chain across hardware, firmware, system, and business, resulting in low overall fault localization efficiency and accuracy. It heavily relies on manual maintenance experience and cannot meet the intelligent fault diagnosis needs of large-scale, highly reliable server clusters.

[0019] To address all or part of the aforementioned technical problems, this application provides a fault location method for a baseboard management controller. This method overcomes the limitations of traditional technologies that rely solely on a single log data source by collecting multi-source heterogeneous logs and extracting key events. It achieves full-dimensional fault data coverage across modules, avoiding the problem of misjudgment of cross-module faults caused by single hardware log analysis. By conducting time-series correlation analysis on key events and constructing a time-series correlation network, it accurately sorts out the occurrence sequence logic of various fault events, and further mines and identifies the inherent causal relationships of events from the time-series correlation network. This method abandons the limitations of traditional time-series machine learning methods, which can only identify correlations but cannot... By addressing the shortcomings of causal analysis, a preliminary causal graph that reflects the fault propagation path is constructed. This preliminary causal graph is enhanced by integrating it with a predefined fault knowledge base, compensating for the limitations of simple causal reasoning and the inability of static knowledge bases to dynamically analyze real-time logs and adapt to new types of faults. Finally, through fault alarm events driving the fault causal graph deduction, the entire fault propagation chain from hardware, firmware, system to business is fully reconstructed. This fundamentally solves the core technical problems of traditional BMC fault location technology, such as its inability to integrate multi-source logs, distinguish causal relationships between events, incomplete fault propagation chain reconstruction, and low fault location accuracy.

[0020] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Figure 1 This is a schematic diagram of the specific application environment architecture upon which the fault location method for a baseboard management controller provided in this application depends. For example... Figure 1 As shown, the application environment architecture upon which this execution depends consists of two main parts: the target server-side component and the analysis server-side component. The specific functions of each component are as follows: A lightweight data acquisition agent is deployed on the target server side. This agent acts as a data acquisition front-end, connecting to the Hardware Baseboard Management Controller (BMC) interface, the operating system kernel log interface, and various application service log output interfaces to complete the real-time acquisition of raw log data in a streaming or polling manner. The acquisition agent is locally configured with a high-precision system clock to record the reception time of each log entry. The acquisition agent then transmits the acquired raw logs and reception time information to the analysis service side.

[0022] The BMC hardware baseboard management controller serves as the hardware layer data source, outputting raw logs related to server hardware and providing raw observation data on hardware faults and hardware operating status. The operating system kernel provides the system layer log data source, outputting logs related to kernel operation and system anomalies. The application service serves as the business layer data source, outputting log data on business operation and service anomalies. These three heterogeneous data sources together constitute the raw data source for fault analysis.

[0023] The analysis service side includes a base time server, a log preprocessing module, a time series correlation analysis module, a causal inference module, a knowledge base module, a graph fusion calibration module, a probabilistic reasoning module, a report generation module, and a monitoring closed-loop control module.

[0024] The base time server provides a globally unified base clock, and the latency offset model corresponding to each data source is dynamically estimated by interacting with the base time server, providing a time base for the synchronization of timestamps in the full log.

[0025] The log preprocessing module receives raw logs reported by the collection agent, performs timestamp synchronization and correction, character encoding conversion, illegal character filtering, regular expression field parsing, key event extraction and classification, and outputs structured log event data.

[0026] The temporal correlation analysis module receives structured log events, performs temporal adjacency statistics and semantic correlation calculations, and constructs an undirected temporal correlation network.

[0027] The causal inference module performs conditional independence checks based on the temporal correlation network, eliminates false correlation edges, completes edge orientation, and outputs a preliminary causal graph.

[0028] The knowledge base module stores predefined standard causal patterns and known fault mode features, supports node semantic matching and causal edge calibration, and can also receive statistically validated new causal relationships to complete autonomous updates.

[0029] The graph fusion calibration module performs graph matching, node semantic alignment, and causal edge consistency verification between the initial causal graph and the standard causal pattern of the knowledge base. It corrects erroneous edges, supplements missing nodes and edges, and outputs a knowledge-enhanced fault causal graph.

[0030] The probabilistic reasoning module receives the observation evidence nodes generated by the fault alarm, calls the approximate reasoning algorithm to perform two-way belief propagation, completes the a priori probability calculation, root cause score ranking, and fault propagation path extraction, and outputs the root cause candidate list and fault propagation link.

[0031] The report generation module calculates the evidence support based on the root cause of the failure, the reasoning path, and log evidence, and generates a structured diagnostic report that includes the root cause conclusion, confidence level, evidence chain, and original data references.

[0032] The monitoring closed-loop control module continuously monitors the fault alarm status. After root cause repair, it judges the alarm clearance status based on the monitoring time window. When the alarm is not cleared, it dynamically issues a collection strategy adjustment command, triggering a new round of complete fault analysis process to achieve closed-loop iteration of fault diagnosis.

[0033] Figure 2 This is a flowchart illustrating a fault location method for a baseboard management controller provided in an embodiment of this application. Figure 2 As shown, an embodiment of this application provides a fault location method for a baseboard management controller, the method comprising the following steps: S201. Obtain multi-source heterogeneous logs and extract key events from them.

[0034] In acquiring multi-source heterogeneous logs, raw log data can be collected from multiple data sources such as the BMC, operating system, and application services to obtain multi-source heterogeneous logs. Specifically, a lightweight agent deployed on the target server collects raw log entries in real time from the standard output or dedicated log files of heterogeneous data sources such as the hardware baseboard management controller, operating system kernel (e.g., ` / var / log / messages`, `syslog`), and application services (e.g., databases, web servers) in a streaming or polling manner. , where superscript Indicates the data source identifier. The original timestamp of the local record in the data source.

[0035] This application simultaneously covers three types of heterogeneous data sources: hardware layer, system layer, and business layer. It overcomes the data dimension limitations of traditional technologies that rely solely on a single BMC hardware log for fault analysis, enabling the acquisition of raw observation data across hardware firmware, operating system, and upper-layer application services. firmware system This provides a complete data foundation for cross-module fault diagnosis, effectively avoiding misjudgment of fault causes when upper-layer business anomalies indirectly trigger BMC alarms. The lightweight agent program is deployed on the target server side, directly connecting to various data source native interfaces and log files in a streaming or polling manner. It can complete the real-time collection of raw log entries without modifying the original server hardware, operating system, or application services. It retains the data source identifier and local original timestamp of the raw logs, completely preserving the original information of the raw logs. This provides reliable raw input for subsequent multi-source log timestamp synchronization correction, heterogeneous log structured parsing, and cross-event correlation analysis, overcoming the shortcomings of related technologies in integrating multi-level log data and facilitating subsequent hardware restoration. firmware system The entire fault propagation chain of the business lays the foundation for data.

[0036] In some embodiments, after acquiring multi-source heterogeneous logs, this application preprocesses these multi-source heterogeneous logs to address the issues of asynchronous clocks and heterogeneous log formats among the various data sources. The preprocessing includes timestamp synchronization and formatting to ensure that all events are recorded on a unified time base.

[0037] In the process of timestamp synchronization of multi-source heterogeneous logs, the original timestamps carried by the multi-source heterogeneous logs and the source identifiers of the corresponding data sources are extracted. The reception time of the multi-source heterogeneous logs is determined, and the standardized timestamps of the multi-source heterogeneous logs are calculated based on the original timestamps, reception time, and latency offsets. The latency offset is the time deviation of a single data source's local clock relative to a unified reference clock. This latency offset comprehensively includes the clock drift deviation of the data source's local clock itself, as well as the network transmission latency generated during the transmission of log data from the data source to the acquisition agent. A corresponding latency offset is maintained for each data source. By periodically synchronizing the clock with the reference time server and measuring the network transmission latency, the latency offset is dynamically updated and estimated. This is used to compensate and correct the original timestamps carried by the logs, thereby eliminating clock inconsistencies between different data sources.

[0038] Specifically, for each log entry, extract its original timestamp. and the source identifier of the corresponding data source It also records the system's high-precision clock time when the acquisition agent receives the log. This application constructs and maintains a delay offset model for each data source relative to a unified reference clock. This offset model dynamically estimates the time by periodically synchronizing with a base time server and calculating network transmission latency. Afterward, based on the original timestamp, latency offset, and reception time, the normalized timestamp of the log is calculated according to the formula: ,in It is a smoothing factor with a value between 0 and 1, used to balance the accuracy of the original timestamp and the reliability of the received timestamp, thereby effectively correcting the time deviation caused by source system clock drift or the delay between log generation and recording, aligning all log events to a unified millisecond-level time base, and providing a foundation for subsequent accurate event sequence correlation analysis.

[0039] After time synchronization is completed, during the formatting process of multi-source log data, the text character encoding of the heterogeneous logs is converted to a preset format, and illegal characters in the logs are filtered out. Regular expressions are used to parse the fields of the heterogeneous log content, converting the unstructured log text into intermediate representation data with a unified field structure. The intermediate representation data includes standardized timestamp identifiers, log levels, component identifiers, and log messages.

[0040] Specifically, standard formatting is performed on multi-source heterogeneous logs. First, the character encoding of the log text is uniformly converted to a preset encoding format, such as 8-bit (Unicode Transformation Format, UTF). 8) Encoding format; and filtering out illegal characters in the log text. Then, regular expressions are used to parse the log content, transforming the original unstructured text log lines into structured data, outputting intermediate representation data with a unified field structure: .

[0041] This intermediate data represents data containing standardized timestamps. The source identifier 's', the log level 'severity', the component identifier 'component', and the log message field 'message' are also included.

[0042] This application addresses the encoding normalization and illegal character cleaning of multi-source heterogeneous logs with different sources and disordered formats, eliminating data parsing anomalies caused by inconsistent character encoding and illegal interference characters. It also achieves structured conversion of unstructured log text into data represented by fixed fields through regular expressions, mapping heterogeneous logs with incompatible formats output by BMC, operating system, and application services to the same data paradigm, thus solving the technical shortcomings of traditional technologies in unifying and integrating multi-source heterogeneous logs.

[0043] In some embodiments, during the extraction of key events from multi-source heterogeneous logs, based on the aforementioned pre-implementation embodiments, log messages are extracted from the intermediate representation data and converted into corresponding feature vectors. These feature vectors are then input into a lightweight real-time classification model to calculate the classification score of each event category. Next, based on this classification score, the key events corresponding to the multi-source heterogeneous logs are determined.

[0044] The lightweight real-time classification model is a forward classification model built on text feature vectorization for streaming log data. It can perform low-latency online inference on the input log stream to identify unknown and emerging log patterns that cannot be covered by predefined production rules, and output classification scores for each event type, so that the event type corresponding to the highest score can be selected as the predicted event type of the log.

[0045] Specifically, for each log, the intermediate representation data The log message fields are processed using the bag-of-words model or word frequency. The inverse document frequency algorithm converts the data into the corresponding feature vector x. This application maintains a set of event types. The event type set contains several event categories, each corresponding to a hardware error alarm. e Different business events, such as system state change (c2) and performance threshold exceeding limits (c3), are considered. The feature vector x is input into a lightweight real-time classification model, and the softmax function is used to calculate the real-time classification score of the feature vector belonging to each event category. Where K is the total number of event categories in the event type set; j is the traversal index for the summation operation, traversing all K event categories; k is the index of the target event type for which the classification score is to be calculated, k∈{1,2,…,K}; the calculation of the classification score depends on the model weight parameters w obtained from pre-training. k With bias parameter b k Critical events carry information such as standardized timestamps, data source identifiers, event types, and attribute parameters.

[0046] The above embodiments abandon the event recognition method that relies solely on manually predefined rules. Instead, they achieve intelligent classification of log events through text feature vectorization combined with a lightweight real-time classification model. This effectively compensates for the technical shortcomings of manual rules, such as limited coverage and inability to adapt to new and unknown fault modes, significantly improving the generalization ability and scenario adaptability of fault event recognition. Simultaneously, by quantitatively calculating the classification scores of each event type, refined and quantitative determination of log event types is achieved. Unlike traditional time-series correlation techniques that can only capture surface-level time-series correlations, this approach can deeply mine the implicit abnormal semantic features of log text, avoiding event recognition bias caused by single-time-series analysis. Furthermore, this lightweight model has low inference overhead and strong real-time performance, adaptable to the online parsing and classification needs of massive streaming multi-source logs. It eliminates the need for continuous manual iteration and maintenance of fault rules, significantly reducing the labor costs of cluster operation and maintenance.

[0047] In some embodiments, this application can extract key events based on predefined rules and pattern recognition technology. The predefined rules are production rules, expressed as pattern-mapped event types, where the pattern acts on the log message field representing the intermediate data. The pattern is implemented using regular expressions or keyword combinations. For example... ,in It is defined in The regular expression or keyword combination on the message field of the log message.

[0048] Specifically, based on the log message fields of the intermediate representation data, precise event matching is achieved using predefined production rules composed of regular expressions and keyword combinations; for new or unknown log patterns not covered by the rules, a bag-of-words model or word frequency is used. The Inverse Document Frequency (IVF) algorithm converts log messages into feature vectors. It then uses a pre-trained lightweight softmax classification model with weight and bias parameters to calculate classification scores for each event type. Prioritizing rule matching results, the algorithm determines the target event type corresponding to the log message. If no matching predefined rule exists, the algorithm selects the event type with the highest real-time classification score from the model's output as the predicted target event type. , where arg max(·) is the operator that takes the variable from which the objective function obtains its maximum value; c k It is the k-th candidate event category in the event type set C, where c k ∈C, k is the event category index; s k It is the output of a lightweight real-time classification model, where log feature vectors are assigned to candidate event category c. k Real-time classification scores.

[0049] The above hybrid strategy for extracting key events based on predefined rules and pattern recognition technology combines the high accuracy of predefined rules with the generalization ability of the model, and can reliably extract key events such as "CPU temperature exceeds the threshold", "disk health status error" and "service changes from running to stopped" from massive logs.

[0050] This application can also generate structured records for each key event. ,in s is a standardized timestamp, representing the moment the critical event occurred; s is the source identifier, used to mark the data source to which the critical event belongs; It is the predicted target event type; It is a collection of key-value pairs that stores specific parameters extracted from logs (such as error codes, device IDs, performance metric values, etc.), thus providing a high-quality, structured data foundation for subsequent cross-layer and cross-component event correlation analysis.

[0051] S202. Conduct time-series correlation analysis on key events and construct a time-series correlation network between events.

[0052] Temporal correlation analysis includes temporal adjacency analysis and logical correlation analysis.

[0053] In some embodiments, during step S202, a temporal proximity analysis is first performed on key events to determine their temporal proximity frequency, and a logical relevance analysis is performed to determine the logical relevance score. Then, the temporal proximity frequency and the logical relevance score are weighted and fused to obtain a comprehensive association strength. Finally, a temporal association network is constructed based on association edges whose comprehensive association strength is greater than a preset threshold.

[0054] Specifically, suppose T key events are extracted from multi-source heterogeneous logs, and the event sequence is... Each event Includes its event type identifier and event timestamp .

[0055] During the temporal contiguity analysis of critical events, the event sequence is traversed, and the pairwise co-occurrence frequency of any two critical event types A and B in all critical events is calculated. When critical event e i The event type is first event type A, and the key event is e. j When the time type is the second event type B, and the timestamp difference between two key events is greater than zero and less than or equal to a preset time window Δt, it is recorded as a time-series adjacent event pair from A to B. All time-series adjacent event pairs that satisfy the time constraints are statistically analyzed to generate an adjacent frequency matrix F. The matrix elements F in the adjacent frequency matrix F are... ABThis represents the number of times the second event type B occurs within a preset time window Δt after the first event type A has occurred.

[0056] In the process of performing logical relevance analysis on key events, semantic information such as keywords, service names, and error codes are extracted from log messages. Logical relevance scores between different event types are calculated based on word vectors, and the logical relevance scores are represented by the cosine similarity between word vectors. , where v A It is the word vector corresponding to key event A, v B It is the word vector corresponding to the key event B.

[0057] In the process of weightedly fusing the temporal adjacent frequency and logical relevance scores to obtain the comprehensive association strength, a weight parameter α is introduced. This α is used to perform a weighted fusion calculation on the elements of the normalized adjacent frequency matrix and the logical relevance scores to obtain the comprehensive association strength between the corresponding two event types. Where, normalize(·) is the normalization function; F AB S represents the number of times the second event type B occurs within a preset time window Δt after the first event type A occurs; AB It is the logical relevance score between the first event type A and the second event type B.

[0058] Next, the overall correlation strength W will be determined. AB By comparing with a preset threshold θ, the associated edges with a comprehensive association strength greater than the preset threshold are retained, thereby initially constructing an undirected temporal association network G=(V,E) of event sequence and logical association. This temporal association network G consists of a set of nodes and a set of edges. The nodes V in the node set correspond to event types, and the associated edges E in the edge set represent significant temporal and logical associations between corresponding event types.

[0059] The above embodiments overcome the single-dimensional limitation of traditional temporal association machine learning schemes that rely solely on the co-occurrence of events in a temporal sequence for association analysis. By introducing quantitative evaluation of log semantic logical relevance on top of temporal adjacency statistics, it takes into account both the temporal regularity of event occurrence and business semantic association characteristics. This effectively solves the technical shortcomings of related technologies that rely solely on temporal features, easily confusing accidental temporal overlaps with genuine associations and resulting in low association identification accuracy. Through a weighted fusion mechanism, a balanced adaptation of temporal and semantic features is achieved. Quantitative comprehensive association strength is used as the basis for association edge selection, replacing the traditional single and coarse association determination method. This accurately retains event pairs with genuine association significance, eliminates invalid pseudo-associations, and constructs a temporal association network that more closely reflects the actual occurrence and propagation patterns of server faults. This provides a high-precision and high-reliability network topology foundation for subsequent causal inference, false association elimination, and fault propagation link reconstruction, fundamentally reducing the bias in determining the root cause of subsequent faults and significantly improving the accuracy and completeness of cross-level fault association analysis across hardware, systems, and business layers.

[0060] S203. Identify causal relationships from the temporal correlation network and generate a preliminary causal graph reflecting the fault propagation path.

[0061] In some embodiments, when performing step S203, a conditional independence test is performed on each pair of nodes that have a connection relationship in the temporal association network; based on the conditional independence test results, false association edges in the temporal association network caused by common causes or accidental temporal overlap are deleted; for the remaining valid association edges in the temporal association network, the causal direction of the edges is determined according to the time order principle and the separation set information, and a preliminary causal graph is generated.

[0062] This application, based on the temporal correlation network, employs a PC causal inference algorithm based on conditional independence testing to identify causal relationships. Each event type is abstracted as a random variable, and a binary sequence of whether events occurred within multiple time intervals is used as observation data. Based on the Fisher-Z test of Gaussian distribution and partial correlation coefficients, conditional independence tests are performed on network node pairs. The condition set is progressively expanded to iteratively eliminate spurious correlation edges caused by common causes and accidental temporal overlaps. Combining the temporal principle of event sequence with separation set information, the causal direction of the remaining valid correlation edges is oriented. Finally, all statistically significant directed causal edges are integrated to generate a preliminary causal graph model that can characterize potential fault propagation paths, thus avoiding misjudgment problems caused by simple temporal correlation judgments.

[0063] Specifically, based on the already constructed temporal correlation network, a causal inference algorithm with conditional independence testing is used to perform directed causal identification processing.

[0064] In the temporal correlation network, event types are abstracted as random variables, and the observation data for each random variable is a binary observation sequence consisting of whether the event type has occurred within multiple time intervals. For each pair of nodes X and Y with a connection relationship in the temporal correlation network, an unconditional independence test is performed. The independence test uses a Fisher's law that follows a Gaussian distribution. The Z-test method is used to calculate the partial correlation coefficient based on the condition set Z. Calculate the corresponding test statistic based on the partial correlation coefficient. The input for calculating the test statistic includes the sample size n, the number of elements in the condition set Z, and the estimated partial correlation coefficient. The absolute value of the test statistic |Z| is compared with the quantile threshold of the standard normal cumulative distribution function Φ. The comparison is performed, and this quantile threshold is determined by the significance level β. When the absolute value of the test statistic is less than this quantile threshold, If the condition is given, then node X and node Y are conditionally independent under the given condition set Z.

[0065] The causal inference algorithm starts with a completely undirected graph and repeatedly performs conditional independence tests by gradually expanding the size of the condition set Z. It deletes network edges that satisfy the conditional independence judgment. This process can eliminate false associations caused by common causes or accidental temporal overlaps and retain statistically significant associations.

[0066] For network edges after edge deletion, based on the time sequence principle that the occurrence of the "cause" event must be earlier than that of the "effect" event, and the information of the separation set, the orientation of the edges is completed. The causal direction of the edges is initially inferred to be either node X pointing to node Y (X→Y) or node Y pointing to node X (Y→X).

[0067] Integrate all directed edges after causal inference verification to generate a preliminary causal graph G. p =(V,E p Preliminary cause-effect graph G p It consists of a set of nodes and a set of directed edges, where each directed edge E in the set of directed edges... p "X points to Y" (i.e., X→Y∈E) p The occurrence of event type X significantly increases the probability of subsequent occurrence of event type Y at the statistical level, and is used to characterize potential fault propagation paths.

[0068] Through statistical testing, the preliminary temporal correlation was upgraded to a causal explanatory model, providing a more reliable and essential topological foundation for subsequent root cause localization, and avoiding false alarms caused by relying solely on temporal sequence and co-occurrence frequency.

[0069] S204. Align and merge the preliminary cause-effect graph with the predefined fault knowledge base to obtain the knowledge-enhanced fault cause-effect graph.

[0070] Among them, the predefined fault knowledge base stores the standard causal pattern execution graph. A standard causal pattern consists of a set of nodes and a set of directed edges.

[0071] In some embodiments, when performing step S204, the preliminary causal graph is first structurally matched with the standard causal patterns of the predefined fault knowledge base, and the semantic similarity between the corresponding nodes is calculated; then, based on the semantic similarity, the nodes of the preliminary causal graph are aligned with the nodes of the knowledge base; consistency verification is performed on the outgoing edge set of the aligned nodes, and the consistency score of the causal edges is calculated; based on the consistency score, the erroneous edges in the preliminary causal graph are corrected, the missing key causal nodes are filled in, and the fault causal graph is generated.

[0072] This application targets all node pairs in the preliminary causal graph and the causal pattern in the knowledge base. It integrates the cosine similarity of node text embedding vectors with the matching degree of node attributes such as fault type and severity level. A weighted calculation formula is used to obtain the semantic similarity of nodes. Based on this node similarity, a bipartite graph matching model is constructed, and the Hungarian algorithm is used to solve for the maximum weighted matching result, achieving accurate semantic alignment of nodes across the knowledge base and eliminating mismatches caused by naming differences. After node alignment, consistency verification is performed on the outgoing edge set of paired aligned nodes. A confidence model is constructed by combining the alignment status of the target node, the time delay difference of causal edges, and the attenuation coefficient, and the consistency score of the causal edges is calculated. Suspected erroneous edges in the preliminary causal graph whose consistency scores and statistical confidence do not meet the threshold requirements are corrected or deleted. Simultaneously, based on log events and operational metrics, standard causal edges and corresponding nodes that exist in the knowledge base but are missing in the preliminary causal graph are supplemented. Finally, the high-confidence valid edges of the retained preliminary causal graph, the supplemented edges from the knowledge base, and the calibration and modification edges are integrated, combining the original nodes and the newly added nodes to generate a knowledge-enhanced server fault causal graph with a complete structure and accurate causal relationships.

[0073] Specifically, regarding the preliminary cause-effect graph Each node , compared with the standard causal model Each node Calculate the semantic similarity s between nodes ij This similarity score integrates the cosine similarity of the text embedding vectors corresponding to the node names, as well as the matching degree of node attributes such as fault type and severity level. Semantic similarity is calculated by weighting the cosine similarity of the text embedding vector e with the similarity of the node attribute A using a weight coefficient α. , where e i It is a preliminary cause-effect graph node v i The node name text embedding vector; ej It is a standard causal pattern node v j Node name text embedding vector; sim attr (·) is the node attribute similarity function; A i It is a preliminary cause-effect graph node v i A set of attributes; A j It is a standard causal pattern node v j The set of attributes.

[0074] A bipartite graph matching problem is constructed based on the semantic similarity of all node pairs. The Hungarian algorithm is used to solve the maximum weight matching result, and the optimal mapping relationship between the initial causal graph nodes and the knowledge base nodes is established to achieve semantic alignment between different graph entities and avoid misjudgment caused by inconsistent entity naming.

[0075] After completing the semantic alignment of nodes, edge comparison and calibration of causal relationships are performed; for each pair of aligned nodes... Obtain the set of all outgoing edges of the corresponding nodes in the preliminary causal graph. and the standard outgoing edge set of the corresponding node in the knowledge base. Calculate the edge consistency score between each pair of sides. The edge consistency score is calculated jointly by the target node alignment identifier and the exponential decay confidence model based on the time delay difference and the decay coefficient λ. ,in This is the target node alignment identifier, indicating whether the target node has been aligned, and Δt is the typical time delay corresponding to the causal edge.

[0076] When a causal edge e in the initial causal graph p If a standard edge with a high consistency score greater than the first threshold θ1 cannot be found in the knowledge base, and the statistical confidence of the causal edge itself is lower than the second threshold θ2, the causal edge is marked as a suspected erroneous edge, and correction or deletion is performed on the causal edge. Conversely, if a standard causal edge e exists in the knowledge base... k If there is no matching causal edge between the aligned node and the corresponding target node in the preliminary causal graph, and the validity of the standard causal edge can be verified by other log events or running metrics, then the standard causal edge is added to the preliminary causal graph as a missing key causal edge.

[0077] After the above-mentioned fusion calibration process, the output is a knowledge-enhanced fault cause-effect graph G. e =(V e E e The set of nodes V in the cause-effect graph of the fault. e For the initial causal graph node set V p The necessary node set V obtained by supplementing the knowledge baseadd The union of V e =V p ∪V add The edge set E of the fault cause-effect graph e The causal edges in the initial causal graph that satisfy the second confidence threshold, the causal edges supplemented from the knowledge base, and the causal edges modified by knowledge base pattern calibration together constitute the following: ,in The edges are supplemented from the knowledge base. These are edges that have been modified after calibration according to the standard causal model of the knowledge base.

[0078] The above refined operations based on graph matching and semantic similarity effectively utilize domain prior knowledge, correct for possible spurious correlations or omissions that may arise from simply relying on log statistics, and thus output a more accurate server failure cause-effect graph with more complete nodes, improving the reliability of root cause analysis.

[0079] S205. When a fault alarm event occurs, determine the fault propagation path based on the fault alarm event and the fault cause-effect graph.

[0080] In some embodiments, in determining the fault propagation path based on the fault alarm event and the fault cause graph, the following steps are taken: First, a probabilistic graphical model or graph-based inference algorithm is used to calculate the posterior probability of the evidence node corresponding to the fault alarm event with respect to the states of other nodes in the fault cause graph. Based on the posterior probability, the root cause node that triggered the fault alarm event is located by tracing back from the child node to the parent node and forward from the parent node to the child node along the causal edge starting from the evidence node. Then, the fault propagation path is determined based on the root cause node.

[0081] In determining the fault propagation path based on root cause nodes, a root cause score is calculated for each root cause node. The root cause score is used to characterize the degree to which evidence nodes increase the confidence of the root cause node in the fault. By analyzing the activation paths in the belief propagation process, the optimal explanatory path connecting high-probability root cause nodes and evidence nodes is extracted as the fault propagation path.

[0082] When a new fault alarm event is detected, the alarm event is instantiated as an abnormal state observation evidence node and injected into a pre-constructed causal Bayesian network (i.e., a fault causal graph). A cyclic belief propagation approximate inference algorithm is used to perform global probabilistic inference. Based on the observed evidence, the posterior probability of all nodes in the fault causal graph is calculated. A bidirectional message propagation mechanism completes the global belief update. On one hand, reverse messages from child nodes to parent nodes trace the upstream causes of the fault; on the other hand, forward messages from parent nodes to child nodes assess the downstream impact range of the fault. For potential root cause nodes without parent nodes or with a specified fault source type, a root cause score is obtained by calculating the node fault probability gain before and after evidence observation. A root cause candidate list is generated by sorting the scores in descending order. Simultaneously, activation paths in the belief propagation process are extracted, and the optimal fault propagation link between high-probability root cause nodes and alarm evidence nodes is selected. The posterior probability of each node in the link is recorded. The sorted root cause list and complete fault propagation link can be output through causal path highlighting and node probability heatmap visualization, assisting operations personnel in intuitively determining the root cause and propagation process of the fault.

[0083] Specifically, when a fault alarm event occurs, the fault alarm event is instantiated to obtain an observation evidence node E, and the node state s of the observation evidence node E is... E Assign the observed outlier value, for example, assign the node state s. E Setting it to 1 indicates alarm triggering; the observed evidence node is then injected into a pre-constructed fault causal graph to perform probabilistic inference operations. The fault causal graph is a causal Bayesian network G=(V,A,P), which includes a set of nodes V, a set of directed edges A representing causal relationships, and a joint probability distribution P.

[0084] Update the state beliefs of all nodes within the causal Bayesian network based on the observed evidence node E. That is, given the observed evidence E=e, calculate the state beliefs of any other node X within the causal Bayesian network. i The conditional posterior probability P(X) corresponding to ∈V i |E=e); For large-scale network scenarios, efficient approximate inference algorithms such as cyclic belief propagation are used to solve for probabilities.

[0085] During the belief propagation process, the message is transmitted along the directed causal edges of the causal Bayesian network, and the node X in the causal Bayesian network... i Receive from the parent node set Pa(X) i ) and the set of child nodes Ch(X) i The message is transmitted from child node to parent node and its own marginal beliefs are updated. Starting from the observation evidence node E, the message propagates along two types of paths: one is the reverse tracing path, where the message is transmitted from child node to parent node and used to calculate the posterior probability P(Pa(X) of the parent node set under the observation evidence condition.i The first type is the upstream cause of the fault, where the message is passed from the parent node to the child node, and is used to calculate the posterior probability P(Ch(X) of the child node set under the observed evidence condition. i )|E=e), complete the assessment of the potential impact range of the fault, and ensure the completeness of probabilistic reasoning through a two-way message propagation mechanism.

[0086] To achieve root cause localization, this application configures a root cause scoring function for each potential root cause node R. j The root cause score is obtained by calculating the difference between the posterior probability of the potential root cause node being in a faulty state under observed evidence and the prior probability of the node being in a faulty state. , This indicates that the node is in a faulty state; E=e represents instantiating the newly detected fault alarm event as observational evidence. Potential root cause nodes are generally nodes without parent nodes or fault source nodes of a specified type. This root cause score characterizes the extent to which the observational evidence increases the confidence of the node's fault.

[0087] All candidate root cause nodes can be sorted in descending order based on root cause scores, outputting a root cause candidate list. Simultaneously, the network paths activated during belief propagation are analyzed, extracting the most probable explanatory paths connecting high-probability root cause nodes to observed evidence nodes—the fault propagation links—and recording the posterior probabilities of each node and its corresponding state on these links. Finally, the sorted root cause candidate list and its corresponding fault propagation links are output, visualized through causal path highlighting and node probability heatmaps, assisting operations personnel in quickly identifying fault root causes and understanding the fault propagation process.

[0088] The above embodiments differ from traditional time-series correlation schemes that rely solely on event co-occurrence relationships for root cause determination. Based on causal Bayesian networks, they employ probabilistic inference to quantify the posterior probability of each fault node under alarm evidence conditions. This effectively distinguishes between temporal correlations and true causal relationships between events, avoiding misjudgments of root causes when multiple anomalies are triggered by the same underlying factor. A cyclic belief propagation approximation inference algorithm is used, adaptable to fault causal graphs with a large number of nodes. This reduces computational overhead while ensuring inference accuracy, meeting the requirements for real-time fault location in server clusters. A bidirectional message propagation mechanism simultaneously completes upstream root cause tracing and downstream fault impact assessment, locating both the fault origin and clarifying the fault propagation range. Root cause scores are generated based on probability gain, enabling quantitative ranking of root cause candidates and overcoming the shortcomings of related technologies, such as vague output results and strong subjective dependence. Furthermore, through fault propagation link extraction and visualization, the hardware is fully reconstructed. firmware system The entire business fault propagation process will intuitively output the abstract causal reasoning results, reduce the reliance on the professional experience of operation and maintenance personnel, improve the efficiency of fault diagnosis and the accuracy of root cause location, and adapt to intelligent fault diagnosis scenarios of large-scale server clusters.

[0089] In some embodiments, this application further includes: extracting evidence feature vectors from the evidence source logs of the fault propagation chain; calculating the similarity between the evidence feature vectors and standard fault features in a predefined fault knowledge base to obtain the evidence support degree; and generating a diagnostic report based on the evidence support degree.

[0090] Based on the root cause and propagation chain of the fault obtained by location, this application constructs a multi-evidence confidence fusion model. For each log evidence in the fault propagation chain, the evidence feature vector is extracted and cosine similarity is calculated with the feature vector of the standard fault mode in the knowledge base to quantify the support of a single piece of evidence for the root cause of the fault. Combined with the original log fragments and evidence timestamps, a structured diagnostic report containing the root cause conclusion, confidence level, evidence chain, and original data references is generated according to a unified template.

[0091] Specifically, it can be based on the obtained root cause of the failure c root And the reasoning path, generating a structured diagnostic report. Among them, the reasoning path... For multiple events pointing to the root cause c of the failure root The fault propagation path is constructed. The core of the diagnostic report is built using a confidence fusion model that integrates multiple types of evidence.

[0092] For each piece of evidence e in the reasoning path i For example, specific error log patterns or outlier metrics, obtained from the source log L corresponding to this evidence. j Extracting the evidence feature vector V ei The evidence feature vector V ei Compared with the known fault mode feature vectors stored in the knowledge base Similarity calculations are performed to obtain the individual evidence linking the root cause c of the fault. root Evidence support level s i Evidence support level s i Obtained by cosine similarity calculation between feature vectors The evidence support level (s) for each piece of evidence. i Original log fragments And the evidence timestamp t i Integrate into a unified report template to generate a complete structured diagnostic report that includes root cause conclusions, confidence levels, chains of evidence, and original data citations.

[0093] The above embodiments abandon the traditional method of simply listing log records to generate reports. Instead, they use vector cosine similarity to quantitatively calculate the support of evidence, objectively quantifying the contribution of each log entry. This avoids the drawbacks of relying on the subjective experience of operations personnel to judge the validity of evidence, thus improving the objectivity and interpretability of diagnostic reports. The structured diagnostic report standardizes and integrates root cause conclusions, confidence indicators, evidence chains, and original log reference information, completely preserving the tracing basis of the fault reasoning process, facilitating verification of root cause location results by operations personnel. Simultaneously, the report filters related log evidence based on the fault propagation chain, filtering irrelevant log noise and reducing interference from invalid information. This solves the problem of difficult evidence screening under massive log volumes, intuitively presenting the complete reasoning logic from original log anomalies to the root cause of the fault. This reduces the reliance on the experience of operations experts for fault investigation and improves the efficiency of fault review and problem verification in large-scale server cluster scenarios.

[0094] In some embodiments, this application can also perform validity verification on newly added causal relationships not included in the predefined fault knowledge base; if the validity verification passes, the predefined knowledge base is updated. During the validity verification process, within a historical time window, the actual co-occurrence frequency and expected co-occurrence frequency of the associated events corresponding to the newly added causal relationship are counted; using the chi-square test, the corresponding significance parameter is calculated based on the actual co-occurrence frequency and expected co-occurrence frequency; if the significance parameter is less than a preset significance threshold, the newly added causal relationship is considered to have passed the validity verification.

[0095] While generating the structured diagnostic report, we also analyzed the newly discovered causal relationships that were not previously included in the existing knowledge base, which were uncovered during the fault localization process. Perform validity checks; the newly added causal relationship consists of two sets of related events. The validity was determined using a statistical significance test mechanism. Within a preset historical time window T, two related events were statistically analyzed. and The actual number of co-occurrences N co Number of co-occurrences with the expected value N exp The corresponding significance p-value is calculated using the chi-square test; when the obtained p-value is less than the preset significance threshold... (like ),Right now When determining the newly added causal relationship It possesses statistical validity. Valid new causal relationships that pass verification are written into and updated in the fault knowledge base, along with the confidence level (1-p) calculated from the significance p-value. This enables dynamic iteration and autonomous evolution of the fault knowledge base, allowing the system to continuously adapt to various new and unknown fault modes.

[0096] The above embodiments overcome the technical shortcomings of static knowledge base solutions, such as fixed fault modes and inability to adapt to server hardware upgrades and new fault scenarios. They achieve automated mining, verification, and iterative updates of fault causal knowledge. By using chi-square statistical significance testing to quantitatively verify newly added causal relationships, pseudo-causal associations arising from the co-occurrence of random events can be accurately eliminated, ensuring the statistical reliability and effectiveness of new knowledge and avoiding the problem of decreased diagnostic accuracy caused by invalid or erroneous knowledge being entered into the knowledge base. This solution requires no manual intervention in rule writing and knowledge iteration. It can autonomously discover unknown fault causal patterns based on real operation and maintenance log data, achieving dynamic self-evolution and continuous iteration of the fault knowledge base. This reduces manual operation and maintenance and rule maintenance costs, enabling the system to continuously adapt to new and unknown fault scenarios arising from server hardware upgrades and business iterations, and continuously improving the intelligence level and scenario generalization capability of BMC hardware fault root cause localization.

[0097] In some embodiments, this application further includes detecting alarm status related to the fault propagation link within a preset verification time window; after the root cause of the fault is repaired, if the alarm status is not eliminated after the preset verification time window has expired, a new round of fault location is initiated.

[0098] After completing report generation and knowledge base update, the system enters the continuous monitoring and verification phase, configuring a preset verification time window. (For example, 5 minutes), within the verification time window The system continuously collects and monitors the real-time alarm status corresponding to the initial fault. If the verification time window has expired after the root cause of the fault has been fixed... The fault alarm status continues to be triggered and not cleared (i.e., within the specified range). If the system determines that the root cause localization results are incomplete or that there are unidentified new potential root causes, it will trigger a new round of fine-grained log collection and fault analysis.

[0099] Based on the results of the previous round of fault diagnosis, the log collection strategy is dynamically optimized and adjusted. The focus is on other service components or hardware components that are dependent on the identified root causes of the faults. The complete fault analysis and processing process is executed iteratively from the beginning, and the analysis is continuously looped until the fault alarm status is completely eliminated. ), to complete closed-loop fault verification and iterative diagnosis.

[0100] The above embodiments address the deficiency of related fault location technologies in lacking a post-verification closed-loop mechanism. These technologies terminate the diagnostic process upon outputting the root cause result, failing to identify partial root causes and missing potential fault sources. The above embodiments, through post-repair alarm status monitoring and time window verification mechanisms, objectively verify the actual effect of root cause repair, enabling timely detection of incomplete location results and undiscovered potential root causes. Simultaneously, the collection scope is dynamically adjusted based on existing diagnostic conclusions, implementing fine-grained data collection on related components to avoid the resource overhead of full log collection, targeted reinforcement of missing observation data, and iterative execution of the entire fault analysis process until the alarm is eliminated. This closed-loop iterative mechanism effectively avoids bias in single fault analysis results, improves the completeness of root cause location in complex coupled fault scenarios, reduces the risk of missed fault detection and incomplete handling, reduces repeated fault recurrence, lowers the operational burden, and adapts to the intelligent troubleshooting needs of complex, multi-factor faults in large-scale server clusters.

[0101] Figure 3 A flowchart illustrating another fault location method for a baseboard management controller provided in this application embodiment is shown below. Figure 3 As shown, this method first performs a multi-source raw log collection step, acquiring heterogeneous raw logs from the BMC, operating system, and application services. Next, it proceeds to a unified formatting and timestamp synchronization step, completing the normalization of heterogeneous log formats and time-series alignment of multi-source logs. Then, it extracts and classifies key events, identifying and categorizing various fault-related key events from the structured logs. Next, it analyzes temporal proximity and logical correlation, quantifying the frequency of temporal proximity and logical correlation scores between events to construct a temporal association network. Following this, it performs a causal inference step to generate a preliminary causal graph, eliminating false associations based on the temporal association network and outputting the preliminary causal graph. Finally, it performs an alignment and fusion step with the fault knowledge base, matching and fusing the preliminary causal graph with pre-stored standard fault patterns and causal knowledge in the fault knowledge base, thus forming a knowledge-enhanced fault causal graph. When a fault alarm event is detected, the process involves injecting a new alarm and performing probabilistic reasoning. The new fault alarm event is instantiated as an observation evidence node and input into the fault causal graph for global probabilistic reasoning. Then, root cause candidates and propagation chains are output, resulting in a sorted list of root cause candidates and high-probability fault propagation paths. Next, a diagnostic report is generated and the knowledge base is updated, outputting a structured diagnostic report. Simultaneously, the fault knowledge base is iteratively updated for newly added causal relationships that have passed validity verification. After root cause repair, the process proceeds to a judgment branch where the alarm remains even after root cause repair. If the judgment result is negative, the alarm has been eliminated, and the fault diagnosis process ends. If the judgment result is positive, the alarm persists, and the process jumps back to the multi-source raw log collection step. The log collection strategy is adjusted, and the entire fault analysis process is iteratively executed again, forming a closed-loop iterative mechanism for fault diagnosis.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0103] Figure 4 This is a schematic diagram of the structure of a fault location device for a baseboard management controller provided in an embodiment of this application, as shown below. Figure 4 As shown, the device includes: The acquisition and extraction module 401 is used to acquire multi-source heterogeneous logs and extract key events from the multi-source heterogeneous logs; The temporal correlation analysis module 402 is used to perform temporal correlation analysis on key events and construct a temporal correlation network between events; The causal relationship identification module 403 is used to identify causal relationships from the temporal correlation network and generate a preliminary causal graph reflecting the fault propagation path; Knowledge enhancement module 404 is used to align and merge the initial cause-effect graph with a predefined fault knowledge base to obtain a knowledge-enhanced fault cause-effect graph. The positioning module 405 is used to determine the fault propagation path based on the fault alarm event and the fault cause-effect graph when a fault alarm event occurs.

[0104] As an optional implementation provided in this application, the acquisition and extraction module 401 is used to: acquire multi-source heterogeneous logs; preprocess the multi-source heterogeneous logs, including timestamp synchronization processing and formatting processing; and extract and classify key events from the preprocessed multi-source heterogeneous logs based on a pattern recognition algorithm.

[0105] As an optional implementation method provided in this application embodiment, the acquisition and extraction module 401 is specifically used for: extracting the original timestamp carried by the multi-source heterogeneous log and the source identifier of the corresponding data source during the timestamp synchronization process of the multi-source heterogeneous log; determining the reception time of the multi-source heterogeneous log; and calculating the standardized timestamp of the multi-source heterogeneous log based on the original timestamp, the delay offset, and the reception time.

[0106] As an optional implementation method provided in this application embodiment, the acquisition and extraction module 401, in the process of formatting multi-source heterogeneous logs, is specifically used to: encode the text characters of the multi-source heterogeneous logs into a preset format and filter illegal characters in the logs; use regular expressions to parse the fields of the content of the multi-source heterogeneous logs; and convert unstructured log text into intermediate representation data with a unified field structure.

[0107] As an optional implementation provided in this application embodiment, the acquisition and extraction module 401, in the process of extracting and classifying key events from preprocessed multi-source heterogeneous logs based on a pattern recognition algorithm, is specifically used for: extracting log messages from intermediate representation data and converting the log messages into corresponding feature vectors; inputting the feature vectors into a pre-trained lightweight classification model and calculating the classification score of the feature vectors belonging to each event type; and determining the key events corresponding to the multi-source heterogeneous logs based on the classification scores.

[0108] As an optional implementation provided in this application, the temporal correlation analysis module 402 is used to: perform temporal proximity analysis on key events to determine the temporal proximity frequency; perform logical correlation analysis on key events to determine the logical correlation score; perform weighted fusion of the temporal proximity frequency and the logical correlation score to obtain the comprehensive correlation strength; and construct a temporal correlation network of event temporality and logical correlation based on the correlation edges whose comprehensive correlation strength is greater than a preset threshold.

[0109] As an optional implementation provided in this application embodiment, the causal relationship identification module 403 is used to: perform a conditional independence test on each pair of nodes with a connection relationship in the temporal correlation network; delete false correlation edges in the temporal correlation network caused by common causes or accidental temporal overlap based on the conditional independence test results; and determine the causal direction of the remaining valid correlation edges in the temporal correlation network according to the time order principle and the separation set information, and generate a preliminary causal graph.

[0110] As an optional implementation provided in this application embodiment, the knowledge enhancement module 404 is used to: perform structural matching between the preliminary causal graph and the standard causal pattern of the predefined fault knowledge base, and calculate the semantic similarity between the corresponding nodes; align the nodes of the preliminary causal graph with the nodes of the knowledge base according to the semantic similarity; perform consistency verification on the outgoing edge set of the aligned nodes, and calculate the consistency score of the causal edges; and correct the erroneous edges in the preliminary causal graph, fill in the missing key causal nodes, and generate the fault causal graph according to the consistency score.

[0111] As an optional implementation provided in this application embodiment, the positioning module 405 is used to: calculate the posterior probability of the evidence node corresponding to the fault alarm event to the state of other nodes in the fault causal graph using a probabilistic graphical model or a graph-based inference algorithm; based on the posterior probability, trace back from the evidence node along the causal edge from the child node to the parent node and propagate forward from the parent node to the child node to locate the root cause node that triggered the fault alarm event; and determine the fault propagation link based on the root cause node.

[0112] As an optional implementation provided in this application, the positioning module 405, in the process of determining the fault propagation link based on the root cause node, is specifically used to: calculate the root cause score for each root cause node; the root cause score is used to characterize the degree to which the evidence node increases the confidence of the root cause node in the fault; and extract the optimal explanatory path connecting the high-probability root cause node and the evidence node as the fault propagation link by analyzing the activation path in the belief propagation process.

[0113] As an optional implementation provided in this application, the device further includes a report generation module, used for: extracting evidence feature vectors from the evidence source logs of the fault propagation link; calculating the similarity between the evidence feature vectors and standard fault features in a predefined fault knowledge base to obtain the evidence support degree; and generating a diagnostic report based on the evidence support degree.

[0114] As an optional implementation provided in this application, the device further includes an update module, used for: validating the validity of newly added causal relationships not included in the predefined fault knowledge base; and updating the predefined knowledge base if the validity verification passes.

[0115] As an optional implementation method provided in this application embodiment, the update module, in the process of validating the validity of newly added causal relationships not included in the predefined fault knowledge base, is specifically used to: count the actual co-occurrence frequency and expected co-occurrence frequency of the associated events corresponding to the newly added causal relationship within a historical time window; calculate the corresponding significance parameter based on the actual co-occurrence frequency and expected co-occurrence frequency using a chi-square test; if the significance parameter is less than a preset significance threshold, the newly added causal relationship is considered to have passed the validity verification.

[0116] As an optional implementation method provided in this application embodiment, the monitoring closed-loop control module is used to: detect the alarm status related to the fault propagation link within a preset verification time window; after the root cause of the fault is repaired, if the alarm status is not eliminated after the preset verification time window has expired, a new round of fault location is initiated.

[0117] For a description of the features in the embodiment corresponding to the fault location device of the baseboard management controller, please refer to the relevant description of the embodiment corresponding to the fault location method of the baseboard management controller, which will not be repeated here.

[0118] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program, and the processor 502 is configured to run the computer program to execute the steps in any of the above-described embodiments of the fault location method for a baseboard management controller.

[0119] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the fault location method for a baseboard management controller when it is run.

[0120] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0121] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the fault location method for a baseboard management controller.

[0122] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the fault location method for a baseboard management controller.

[0123] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-chip (SoC), a complex programmable logic device (CPLD), a microcontroller unit (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0124] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] The above provides a detailed description of a fault location method and electronic device for a baseboard management controller provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A fault location method for a baseboard management controller, characterized in that, include: Obtain multi-source heterogeneous logs and extract key events from the multi-source heterogeneous logs; Perform time-series correlation analysis on the key events to construct a time-series correlation network between the events; From the temporal correlation network, causal relationships are identified to generate a preliminary causal graph reflecting the fault propagation path; The preliminary cause-effect graph is aligned and fused with a predefined fault knowledge base to obtain a knowledge-enhanced fault cause-effect graph. When a fault alarm event occurs, the fault propagation path is determined based on the fault alarm event and the fault cause-effect graph.

2. The fault location method according to claim 1, characterized in that, The process of acquiring multi-source heterogeneous logs and extracting and classifying key events from these logs includes: Obtain multi-source heterogeneous logs; The multi-source heterogeneous logs are preprocessed, including timestamp synchronization and formatting. Based on pattern recognition algorithms, the key events are extracted and classified from preprocessed multi-source heterogeneous logs.

3. The fault location method according to claim 2, characterized in that, The timestamp synchronization process for the multi-source heterogeneous logs includes: Extract the original timestamps and source identifiers of the corresponding data sources carried by the multi-source heterogeneous logs; Determine the reception time of the multi-source heterogeneous logs; Based on the original timestamp, delay offset, and reception time, the standardized timestamp of the multi-source heterogeneous log is calculated.

4. The fault location method according to claim 2, characterized in that, The formatting process for the multi-source heterogeneous logs includes: The text characters of the multi-source heterogeneous logs are encoded into a preset format, and illegal characters in the logs are filtered out. Regular expressions were used to parse the fields of the multi-source heterogeneous logs. Convert unstructured log text into intermediate representation data with a unified field structure.

5. The fault location method according to claim 4, characterized in that, The pattern recognition algorithm extracts and classifies key events from preprocessed multi-source heterogeneous logs, including: For the intermediate representation data, extract log messages and convert the log messages into corresponding feature vectors; The feature vector is input into a pre-trained lightweight classification model to calculate the classification score of the feature vector belonging to each event type; Based on the classification scores, the key events corresponding to the multi-source heterogeneous logs are determined.

6. The fault location method according to claim 1, characterized in that, The step of performing time-series correlation analysis on the key events and constructing a time-series correlation network among the events includes: Perform temporal proximity analysis on the key events to determine the temporal proximity frequency; Perform logical relevance analysis on the key events to determine the logical relevance score; The temporal adjacent frequency and the logical relevance score are weighted and fused to obtain the comprehensive association strength; Based on the association edges whose comprehensive association strength is greater than a preset threshold, the temporal association network of event sequence and logical association is constructed.

7. The fault location method according to claim 1, characterized in that, The step of identifying causal relationships from the temporal correlation network and generating a preliminary causal graph reflecting the fault propagation path includes: Perform a conditional independence test on each pair of nodes that have a connection relationship in the temporal correlation network; Based on the conditional independence test results, delete the false association edges in the temporal association network caused by common causes and accidental temporal overlap. For the remaining valid association edges of the temporal association network, the causal direction of the edges is determined according to the time order principle and the separation set information, and the preliminary causal graph is generated.

8. The fault location method according to claim 1, characterized in that, The step of aligning and fusing the preliminary cause-effect graph with a predefined fault knowledge base to obtain a knowledge-enhanced fault cause-effect graph includes: The preliminary causal graph is structurally matched with the standard causal patterns of the predefined fault knowledge base, and the semantic similarity between corresponding nodes is calculated. Based on the semantic similarity, align the nodes of the initial causal graph with the nodes of the knowledge base; Perform consistency checks on the outgoing edge set of aligned nodes and calculate the consistency score of causal edges; Based on the consistency score, erroneous edges in the preliminary causal graph are corrected, missing key causal nodes are added, and the fault causal graph is generated.

9. The fault location method according to claim 1, characterized in that, Based on the fault alarm events and the fault cause-effect graph, the fault propagation path is determined, including: Using a probabilistic graphical model or a graph-based inference algorithm, calculate the posterior probability of the evidence node corresponding to the fault alarm event with respect to the states of other nodes in the fault causal graph; Based on the posterior probability, starting from the evidence node, trace back along the causal edge from the child node to the parent node and propagate forward from the parent node to the child node to locate the root cause node that triggered the fault alarm event; The fault propagation path is determined based on the root cause node.

10. The fault location method according to claim 9, characterized in that, Determining the fault propagation link based on the root cause node includes: For each root cause node, a root cause score is calculated; the root cause score is used to characterize the degree to which the evidence node improves the confidence of the root cause node's failure. By analyzing the activation paths in the belief propagation process, the optimal explanatory path connecting high-probability root cause nodes and evidence nodes is extracted as the fault propagation link.

11. The fault location method according to claim 1, characterized in that, The method further includes: Extract evidence feature vectors from the evidence source logs of the aforementioned fault propagation link; The similarity between the evidence feature vector and the standard fault features in the predefined fault knowledge base is calculated to obtain the evidence support degree; A diagnostic report is generated based on the level of evidence support.

12. The fault location method according to claim 1, characterized in that, The method further includes: Validate the validity of newly added causal relationships that are not included in the predefined fault knowledge base; If the validity check passes, update the predefined knowledge base.

13. The fault location method according to claim 12, characterized in that, The validity verification of newly added causal relationships not included in the predefined fault knowledge base includes: Within the historical time window, the actual co-occurrence count and expected co-occurrence count of the associated events corresponding to the newly added causal relationship are counted. Using the chi-square test, the corresponding significance parameter is calculated based on the actual co-occurrence frequency and the expected co-occurrence frequency; If the significance parameter is less than the preset significance threshold, the newly added causal relationship is considered to have passed the validity check.

14. The fault location method according to claim 1, characterized in that, The method further includes: Within a preset verification time window, detect alarm statuses related to the fault propagation link; If the alarm status is not cleared after the root cause of the fault is repaired, a new round of fault location will be initiated.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the fault location method for the baseboard management controller as described in any one of claims 1 to 14.

Images