Hardware fault real-time detection method and system based on CPU and BMC cooperation

Abstract

The application discloses a hardware fault real-time detection method and system based on CPU and BMC cooperation, comprising the following steps: by establishing a communication mechanism between CPU and BMC, high-frequency state data and trend indicators are collected respectively. The processor uses the CUSUM algorithm to analyze the mutation of the periodically collected running state, and generates a mutation event; the management controller uses the EWMA algorithm to model the trend data and extract abnormal changes. The system analyzes the fusion of the mutation event and the trend anomaly to form a fusion anomaly vector. The risk assessment module calculates the risk score based on the preset rules, determines the fault level, locates the target component, and outputs the processing strategy. According to the strategy, the response action is triggered and the execution state is recorded. The fusion data and the response record are input into the adaptive module together, which is used for dynamically adjusting the CUSUM and EWMA parameters, realizing adaptive updating of the algorithm, and finally generating a fault detection signal.

Description

Technical Field

[0001] This invention relates to the field of hardware fault monitoring technology, and in particular to a method and system for real-time hardware fault detection based on CPU and BMC collaboration. Background Technology

[0002] With the continuous expansion of high-density servers, edge computing nodes, and cloud data centers, the stability and maintainability of hardware operation have become crucial foundations for ensuring system service continuity. Currently, mainstream hardware fault detection generally employs a data acquisition and analysis mechanism completed at a single point by the central processing unit, or a dedicated circuit board management controller that periodically monitors hardware operating trends and identifies potential faults through anomaly threshold alarms.

[0003] Existing fault detection methods still have significant limitations in multi-source data fusion, fault feature extraction, and real-time response. On the one hand, traditional methods lack an efficient collaborative mechanism between the processor and the management controller, making it difficult to detect abrupt faults in a timely manner, while trend-based faults suffer from detection lag. On the other hand, anomaly judgment in existing methods typically relies on static thresholds or single-dimensional feature analysis, lacking the ability to model the joint evolutionary relationship between abrupt events and trend shifts, which easily leads to false alarms or missed alarms. Furthermore, a closed-loop feedback loop is not formed between fault analysis results and response strategies; monitoring algorithm parameters such as CUSUM and EWMA fail to adaptively update according to the operating status, causing the detection system to experience a decline in robustness or an imbalance in sensitivity during long-term operation.

[0004] Therefore, how to provide a real-time hardware fault detection method and system based on CPU and BMC collaboration is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a real-time hardware fault detection method and system based on CPU and BMC collaboration. This invention employs the CUSUM algorithm and EWMA algorithm to construct a joint identification mechanism for mutation-type and trend-type anomalies, designs a shared cache communication structure and field mapping table to achieve dual-channel data interaction, and completes the dynamic determination of fault level, hardware component location and strategy distribution through fusion analysis, risk assessment and response feedback mechanism. At the same time, an adaptive engine is introduced to complete the algorithm parameter update, which has the advantages of wide coverage, strong response closed loop and high long-term operation stability.

[0006] The real-time hardware fault detection method and system based on CPU and BMC collaboration according to embodiments of the present invention includes the following steps:

[0007] Establish a communication link and role allocation mechanism between the CPU and BMC. The CPU collects high-frequency status data and performs mutation analysis, while the BMC collects trend indicators and performs trend modeling. The two exchange data through shared cache.

[0008] The CPU periodically collects transient parameters of each hardware component, forms a state data sequence, and writes it into the shared cache. At the same time, it calls the CUSUM algorithm to process the state data sequence and generate a set of mutation events.

[0009] BMC reads the data sequence from the shared cache, calls the EWMA algorithm to calculate the smoothed trend curve, and extracts the trend offset to form a trend anomaly signal;

[0010] The set of mutation events and trend anomaly signals are taken as input and fed into the fusion analysis module. Time alignment is performed and fusion features are calculated. The fusion anomaly vector is output.

[0011] The fused anomaly vector is input into the risk assessment module, which calculates the risk score and determines the fault level based on the numerical thresholds and combination rules in the preset fused feature fields, and outputs the location component and suggested strategy.

[0012] Based on the positioning components and suggested strategies, trigger the corresponding response actions and generate a response status record;

[0013] The fusion of anomaly vectors, fault levels, and response status records is used as sample input to the adaptive engine, and the CUSUM and EWMA parameters are adjusted to generate fault detection signals.

[0014] Optionally, the mechanism for establishing a communication link and role allocation between the CPU and BMC includes:

[0015] During the system initialization phase, the collaborative management firmware is loaded, the CPU and BMC are connected via the LPC bus, and a command channel is established based on the IPMI protocol. At the same time, a shared memory region is configured in the PCIe address space for bidirectional data interaction.

[0016] The shared memory is divided into a first buffer and a second buffer. The CPU writes the acquired data and the BMC writes the processing feedback, respectively. Each buffer is equipped with a flag field, including a data validity bit and a processing completion bit.

[0017] After the CPU completes initialization, it registers the acquisition module and sets the hardware component number, sampling period and write target buffer address to form a high-frequency mutation monitoring task.

[0018] After BMC completes initialization, it sets up trend analysis tasks and registers the reading sources, modeling frequency, and result write-back paths for performing trend data processing.

[0019] The high-frequency mutation monitoring task and trend data processing generate a field mapping table. The field mapping table corresponding to the acquisition party and the processing party for each piece of hardware data is uniformly scheduled by the collaborative control module and dynamically refreshed according to the sampling status.

[0020] BMC periodically polls the first buffer flag bit. When it detects that the valid bit is set, it extracts data and performs analysis. After processing, it writes the data to the second buffer and sets the processing completion bit.

[0021] After the CPU detects the processing completion bit in the next cycle, it updates the sampling status and synchronously maintains the task status according to the mapping table.

[0022] Optionally, the generation of the mutation event set includes:

[0023] The CPU performs periodic sampling operations according to the target hardware component running status parameter field identified as the high-frequency mutation monitoring task in the field mapping table. After each sampling, the current running status sample value is written to the first buffer in the shared memory area and the CPU local cache sequence.

[0024] The mutation detection module calls the CUSUM algorithm after each round of sampling to perform cumulative analysis of the execution deviation of the running status parameter fields of the corresponding target hardware components in the CPU local cache sequence.

[0025] During the task initialization phase, the mutation detection module configures a set of judgment parameters for the running status parameter fields of each target hardware component, including a reference mean calculation window, a drift constant, and a trigger threshold, and binds them to the field numbers for management.

[0026] When the mutation detection module determines that the sampled value of the running status parameter field in the current period relative to the cumulative offset value reaches a preset trigger threshold, it generates a mutation event data item.

[0027] All mutation event data items that meet the mutation conditions within the current sampling period constitute the mutation event set.

[0028] Optionally, the generation of the trend anomaly signal includes:

[0029] Based on the target hardware component running status parameter field identified as the trend modeling task in the field mapping table, BMC reads the current period sampling value from the first buffer in the shared memory area according to the sampling period uniformly set with the CPU, and writes it into the BMC local cache sequence.

[0030] During the task initialization phase, the trend analysis module loads judgment parameters, including short-term smoothing factor, field weight factor, trend correction factor and trend reference interval, into the running status parameter field of each target hardware component according to the trend modeling parameter configuration table, and binds them with the field number.

[0031] The trend analysis module performs trend modeling on the current sampled value in the BMC local cache sequence, compared with the saved trend values ​​of the previous period and historical trend values, and calculates the trend value of the current period.

[0032]

[0033] Among them, V t T represents the sampled value for the current period. t-1 T represents the trend value of the previous period. t-w Let α be the trend value before period w. s As a weighting factor, β i γ is the field weight factor, and γ is the trend correction factor;

[0034] Trend value If the trend value deviates from the trend reference range for multiple consecutive sampling periods and the direction of deviation is consistent, it is determined that the corresponding field has a trend anomaly.

[0035] After determining that a trend anomaly has been established, a trend anomaly signal is constructed. The trend anomaly signal includes a field number, current trend value, trend offset direction, number of consecutive offset cycles, and trend determination timestamp. The trend anomaly signal is written into the second buffer in the shared memory area for the fusion analysis module to read.

[0036] Optionally, the generation of the fusion anomaly vector includes:

[0037] Based on the field numbers registered in the field mapping table, the mutation event data items and trend anomaly signal items are matched, and the mutation determination timestamp and trend determination timestamp corresponding to each data item are extracted.

[0038] The fusion analysis module maps two timestamps to sampling period numbers and performs time alignment operations based on the matching rule that the difference between the sampling period numbers does not exceed one period, thus establishing a fused data pair with consistent field numbers and time matching.

[0039] For each pair of aligned fused data, perform field-level structural merging to generate fused field anomalies. The fused field anomalies include field number, mutation level, trend offset direction, trend offset period number, and sampling period offset.

[0040] The fusion analysis module constructs a fusion anomaly vector from all fusion field anomalies generated within the current sampling period.

[0041] Optionally, the output of the fault level, target hardware component location, and recommended strategy includes:

[0042] The risk assessment module receives the fusion anomaly vector generated by the fusion analysis module in each sampling period, and extracts the field number, mutation level, trend offset direction, trend offset period number and sampling period offset for each fusion field anomaly item in the fusion anomaly vector;

[0043] During the system initialization phase, the risk assessment module loads the fusion feature field rule table. The fusion feature field rule table establishes a mapping according to the field number, records the numerical threshold and logical combination conditions of each field of the fusion field anomaly item corresponding to each field number, and sets the corresponding risk score level range.

[0044] Scoring is performed based on the values ​​of each field in the anomaly item of the fusion field and the scoring parameters. The scoring results are matched with the level range to determine whether the triggering fault level conditions are met. When the triggering conditions are met, the corresponding fault level is assigned to the anomaly item of the fusion field.

[0045] The target hardware component identifier associated with the fusion field anomaly is found in the field mapping table by the field number, and the matching suggested strategy is retrieved based on the combination key of the field number and the fault level.

[0046] Encapsulate the field number, target hardware component identifier, fault level, and recommended strategy entries into a response control structure.

[0047] Optionally, the generation of the response status record includes:

[0048] In each sampling period, receive the response control structure transmitted by the risk assessment module and extract the field number, target hardware component identifier, fault level and recommended strategy;

[0049] Based on the action types defined in the recommended strategy, perform action mapping operations to determine the sequence of response actions to be executed;

[0050] A response instruction set is generated based on the target hardware component identifier, and instructions are issued to the corresponding target hardware component through the management interface and external interface;

[0051] After each response action is completed, record the action type, execution result status code, response delay, and execution timestamp to generate a response record item;

[0052] Encapsulate all response records completed within the current period into a response status record.

[0053] Optionally, the generation of the fault detection signal includes:

[0054] The adaptive engine receives fused anomaly vectors, fault levels, and response status records in each sampling period;

[0055] The adaptive engine establishes a triplet mapping between the fused anomaly vector, fault level, and response status record based on the field number, and continuously records the fault event sequence corresponding to the corresponding field number;

[0056] After reaching the preset parameter refresh cycle, the adaptive engine traverses the fault event sequence under each field number, and counts the response success rate, field abnormality frequency and sampling cycle offset stability under different fault levels, which are used as feedback factor input parameter adjustment logic.

[0057] The parameter adjustment logic generates updated parameter sets for the CUSUM algorithm and the EWMA algorithm based on preset feedback factor weights and adjustment rules. The updated parameters of the CUSUM algorithm include the reference mean window length and drift gain coefficient, while the updated parameters of the EWMA algorithm include the smoothing factor and the abnormal trend delay threshold.

[0058] The parameter set is updated and bound to the field number to generate a fault detection signal. The fault detection signal includes the field number, the target algorithm type, and the parameter update value. It is distributed to the algorithm module in the central processing unit and the baseboard management controller through the internal configuration synchronization interface.

[0059] After receiving the corresponding fault detection signal, the central processing unit and the baseboard management controller load the updated parameters of the corresponding algorithm modules to complete the adaptive adjustment of the CUSUM algorithm and the EWMA algorithm.

[0060] Optional, including:

[0061] The central processing unit module is used to periodically collect the operating status parameters of the target hardware components and write them into the shared memory area and the central processing unit's local cache sequence;

[0062] The mutation detection module, deployed on the central processing unit side, is used to perform bias accumulation analysis by calling the CUSUM algorithm based on the local cache sequence to generate a mutation event set;

[0063] The baseboard management controller module is used to read data from the shared memory area and perform trend analysis tasks;

[0064] The trend detection module, deployed on the baseboard management controller side, is used to call the EWMA algorithm to generate trend anomaly signals;

[0065] The field mapping management module is used to establish the mapping relationship between field numbers and hardware components, sampling tasks and processing tasks during system initialization, and supports subsequent field alignment and instruction lookup;

[0066] The fusion analysis module is used to align the field numbers and sampling period numbers of the mutation event set and the trend anomaly signal, generate fusion field anomaly items, and construct the fusion anomaly vector.

[0067] The risk assessment module is used to perform field-level scoring based on the fused anomaly vector and the preset fused feature field rule table to determine the fault level, target hardware component identification, and recommended strategy;

[0068] The response control module is used to generate a sequence of response actions based on the suggested strategy, issue response instructions, record response results, and build a response status record.

[0069] The adaptive engine module is used to update the CUSUM and EWMA algorithm parameters based on the fused anomaly vector, fault level, and response status records.

[0070] The beneficial effects of this invention are:

[0071] (1) This invention constructs a collaborative monitoring architecture between the central processing unit and the baseboard management controller, and introduces the CUSUM algorithm and EWMA algorithm to jointly identify high-frequency mutation type and slow trend type hardware anomalies, respectively. This effectively solves the problem of insufficient coverage of hardware fault types by traditional single detection algorithms and improves the breadth and accuracy of fault perception.

[0072] (2) This invention establishes a bidirectional data channel through a shared memory structure, uses a field mapping table to drive the coordinated execution of mutation detection tasks and trend modeling tasks, constructs a fused anomaly vector after fusing mutation event sets and trend anomaly signals, and completes risk assessment and fault level judgment based on a field-level rule table, thereby realizing multi-dimensional data-driven refined diagnosis and fault classification.

[0073] (3) The present invention forms a response control structure by combining the fault level, target hardware components and suggested strategies, sends it to the target module and generates a response status record, and uses it as a sample to feed back to the adaptive engine, dynamically adjusts the parameter settings of the CUSUM and EWMA algorithms, and forms an adaptive closed-loop feedback mechanism, which significantly enhances the stability and adaptability of the system in the long-term operating environment. Attached Figure Description

[0074] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0075] Figure 1 This is a system framework diagram of the real-time hardware fault detection system based on CPU and BMC collaboration proposed in this invention.

[0076] Figure 2This is an overall flowchart of the real-time hardware fault detection method based on CPU and BMC collaboration proposed in this invention.

[0077] Figure 3 This is a flowchart of the process by which the central processing unit performs mutation detection and generates a set of mutation events in this invention. Detailed Implementation

[0078] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0079] refer to Figures 1-3 A real-time hardware fault detection method and system based on CPU and BMC collaboration includes the following steps:

[0080] Establish a communication link and role allocation mechanism between the CPU and BMC. The CPU collects high-frequency status data and performs mutation analysis, while the BMC collects trend indicators and performs trend modeling. The two exchange data through shared cache.

[0081] The CPU periodically collects transient parameters of each hardware component, forms a state data sequence, and writes it into the shared cache. At the same time, it calls the CUSUM algorithm to process the state data sequence and generate a set of mutation events.

[0082] BMC reads the data sequence from the shared cache, calls the EWMA algorithm to calculate the smoothed trend curve, and extracts the trend offset to form a trend anomaly signal;

[0083] The set of mutation events and trend anomaly signals are taken as input and fed into the fusion analysis module. Time alignment is performed and fusion features are calculated. The fusion anomaly vector is output.

[0084] The fused anomaly vector is input into the risk assessment module, which calculates the risk score and determines the fault level based on the numerical thresholds and combination rules in the preset fused feature fields, and outputs the location component and suggested strategy.

[0085] Based on the positioning components and suggested strategies, trigger the corresponding response actions and generate a response status record;

[0086] The fusion of anomaly vectors, fault levels, and response status records is used as sample input to the adaptive engine, and the CUSUM and EWMA parameters are adjusted to generate fault detection signals.

[0087] In this embodiment, the establishment of the communication link and role allocation mechanism between the CPU and BMC includes:

[0088] During the system initialization phase, the collaborative management firmware is loaded, the CPU and BMC are connected via the LPC bus, and a command channel is established based on the IPMI protocol. At the same time, a shared memory region is configured in the PCIe address space for bidirectional data interaction.

[0089] The shared memory is divided into a first buffer and a second buffer. The CPU writes the acquired data and the BMC writes the processing feedback, respectively. Each buffer is equipped with a flag field, including a data validity bit and a processing completion bit.

[0090] After the CPU completes initialization, it registers the acquisition module and sets the hardware component number, sampling period and write target buffer address to form a high-frequency mutation monitoring task.

[0091] After BMC completes initialization, it sets up trend analysis tasks and registers the reading sources, modeling frequency, and result write-back paths for performing trend data processing.

[0092] The high-frequency mutation monitoring task and trend data processing generate a field mapping table. The field mapping table corresponding to the acquisition party and the processing party for each piece of hardware data is uniformly scheduled by the collaborative control module and dynamically refreshed according to the sampling status.

[0093] BMC periodically polls the first buffer flag bit. When it detects that the valid bit is set, it extracts data and performs analysis. After processing, it writes the data to the second buffer and sets the processing completion bit.

[0094] After the CPU detects the processing completion bit in the next cycle, it updates the sampling status and synchronously maintains the task status according to the mapping table.

[0095] In this embodiment, the generation of the mutation event set includes:

[0096] The CPU performs periodic sampling operations according to the target hardware component running status parameter field identified as the high-frequency mutation monitoring task in the field mapping table. After each sampling, the current running status sample value is written to the first buffer in the shared memory area and the CPU local cache sequence.

[0097] The mutation detection module calls the CUSUM algorithm after each round of sampling to perform cumulative analysis of the execution deviation of the running status parameter fields of the corresponding target hardware components in the CPU local cache sequence.

[0098] During the task initialization phase, the mutation detection module configures a set of judgment parameters for the running status parameter fields of each target hardware component, including a reference mean calculation window, a drift constant, and a trigger threshold, and binds them to the field numbers for management.

[0099] When the mutation detection module determines that the sampled value of the running status parameter field in the current period reaches the preset trigger threshold relative to the cumulative offset value, it generates a mutation event data item. The mutation event data item includes the field number in the field mapping table, the current sampled value, the historical reference average, the mutation determination timestamp, and the anomaly level.

[0100] All mutation event data items that meet the mutation conditions within the current sampling period constitute a mutation event set;

[0101] The field number corresponding to the mutation event data item is marked as "mutation pending" in the task status maintenance table. The mutation detection module records the historical trigger frequency of this field number and performs adaptive updates to its judgment parameters based on this frequency.

[0102] The periodic sampling operation is initiated by the CPU after initialization and loading of the field mapping table. The scheduling task is triggered at regular intervals based on the sampling period time parameter set by the system. The sampling operation is performed sequentially according to the target hardware component running status parameter field marked as high frequency mutation monitoring task in the field mapping table. In each round of sampling operation, the current running status value of the corresponding target hardware component is obtained through the hardware abstraction layer interface, and the sampled value is written into the field number corresponding position of the first buffer in the shared memory area, and simultaneously appended to the CPU local cache sequence.

[0103] The deviation accumulation analysis is performed by the mutation detection module based on the continuous sampling values ​​of the target hardware component running status parameter field recorded in the CPU local cache sequence. The analysis process uses the historical mean calculation window of the corresponding field configured in the task initialization phase as a benchmark to calculate the offset of the current sampling value relative to the corresponding mean. In each round of analysis, the mutation detection module incrementally iterates the current offset with the cumulative deviation state of the previous round to form the real-time deviation accumulation value of the field. The deviation accumulation value is compared with the mutation judgment threshold bound to the task scheduling table. When the deviation accumulation value reaches the threshold, it is determined that the current field has mutated, and the judgment result is used as the triggering basis for the mutation event data item.

[0104] In this embodiment, the generation of the trend anomaly signal includes:

[0105] Based on the target hardware component running status parameter field identified as the trend modeling task in the field mapping table, BMC reads the current period sampling value from the first buffer in the shared memory area according to the sampling period uniformly set with the CPU, and writes it into the BMC local cache sequence.

[0106] During the task initialization phase, the trend analysis module loads judgment parameters, including short-term smoothing factor, field weight factor, trend correction factor and trend reference interval, into the running status parameter field of each target hardware component according to the trend modeling parameter configuration table, and binds them with the field number.

[0107] The trend analysis module performs trend modeling on the current sampled value in the BMC local cache sequence, compared with the saved trend values ​​of the previous period and historical trend values, and calculates the trend value of the current period.

[0108]

[0109] Among them, V t T represents the sampled value for the current period. t-1 T represents the trend value of the previous period. t-w Let α be the trend value before period w. s As a weighting factor, β i γ is the field weight factor, and γ is the trend correction factor;

[0110] This formula is derived from the EWMA model in mathematical statistics, and its original form is T. t =α·V t +(1-α)·T t-1 T t V represents trend estimation. t This represents the current observation value, where α is a smoothing factor. To adapt to the diverse dynamic characteristics of the state parameters of key server components, a field difference weighting factor β is first introduced. i This is used to reflect the degree of influence of different fields on trend changes, realizing field-level weighting. Secondly, a new trend offset feedback term γ·(T) is added. t-1 -T t-w This is used to enhance the response capability to slowly evolving fault trends. Its structure is based on quantitative modeling of the rate of change of trend values ​​between adjacent weeks. All parameters in the formula are loaded from the trend modeling parameter configuration table and bound to the field numbers during the system initialization phase.

[0111] Trend value If the trend value deviates from the trend reference range for multiple consecutive sampling periods and the direction of deviation is consistent, it is determined that the corresponding field has a trend anomaly.

[0112] After determining that a trend anomaly has been established, a trend anomaly signal is constructed. The trend anomaly signal includes a field number, current trend value, trend offset direction, number of consecutive offset cycles, and trend determination timestamp. The trend anomaly signal is written into the second buffer in the shared memory area for the fusion analysis module to read.

[0113] In this embodiment, the generation of the fusion anomaly vector includes:

[0114] Based on the field numbers registered in the field mapping table, the mutation event data items and trend anomaly signal items are matched, and the mutation determination timestamp and trend determination timestamp corresponding to each data item are extracted.

[0115] The fusion analysis module maps two timestamps to sampling period numbers and performs time alignment operations based on the matching rule that the difference between the sampling period numbers does not exceed one period, thus establishing a fused data pair with consistent field numbers and time matching.

[0116] For each pair of aligned fused data, perform field-level structural merging to generate fused field anomalies. The fused field anomalies include field number, mutation level, trend offset direction, trend offset period number, and sampling period offset.

[0117] The fusion analysis module constructs a fusion anomaly vector from all fusion field anomalies generated within the current sampling period;

[0118] The time alignment operation first calls the field number information recorded in the field mapping table, filters the records with the same field number in the mutation event data item and the trend anomaly signal item, extracts the mutation determination timestamp in the mutation event data item and the trend determination timestamp in the trend anomaly signal item, and maps the above timestamps to the corresponding unified sampling period number. The sampling period number is numbered based on a unified set sampling period parameter. Then, the fusion analysis module calculates the difference between the two period numbers. If the difference does not exceed the preset time offset tolerance threshold (usually set to ±1 period), the data item corresponding to the field number is considered to have the fusion condition in the time dimension, and the time alignment is completed. All matching items that meet the conditions of consistent field number and satisfactory difference in period number are selected as candidate fusion pairs.

[0119] In this embodiment, the output of the fault level, target hardware component location, and recommended strategy includes:

[0120] The risk assessment module receives the fusion anomaly vector generated by the fusion analysis module in each sampling period, and extracts the field number, mutation level, trend offset direction, trend offset period number and sampling period offset for each fusion field anomaly item in the fusion anomaly vector;

[0121] During the system initialization phase, the risk assessment module loads the fusion feature field rule table. The fusion feature field rule table establishes a mapping according to the field number, records the numerical threshold and logical combination conditions of each field of the fusion field anomaly item corresponding to each field number, and sets the corresponding risk score level range.

[0122] Scoring is performed based on the values ​​of each field in the anomaly item of the fusion field and the scoring parameters. The scoring results are matched with the level range to determine whether the triggering fault level conditions are met. When the triggering conditions are met, the corresponding fault level is assigned to the anomaly item of the fusion field.

[0123] The target hardware component identifier associated with the fusion field anomaly is found in the field mapping table by the field number, and the matching suggested strategy is retrieved based on the combination key of the field number and the fault level.

[0124] Encapsulate the field number, target hardware component identifier, fault level, and recommended strategy entries into a response control structure.

[0125] In this embodiment, the generation of the response status record includes:

[0126] In each sampling period, receive the response control structure transmitted by the risk assessment module and extract the field number, target hardware component identifier, fault level and recommended strategy;

[0127] Based on the action types defined in the recommended strategy, an action mapping operation is performed to determine the sequence of response actions to be performed. The sequence of response actions includes interruption, frequency reduction, restart, marking, or reporting operations of the target component.

[0128] A response instruction set is generated based on the target hardware component identifier, and instructions are issued to the corresponding target hardware component through the management interface and external interface;

[0129] After each response action is completed, record the action type, execution result status code, response delay, and execution timestamp to generate a response record item;

[0130] Encapsulate all response record items completed within the current period into a response status record;

[0131] The action mapping operation refers to determining the matching sequence of response actions based on the field numbers and fault levels contained in the response control structure by querying a preset policy mapping rule table. The policy mapping rule table is loaded during system initialization and indexed according to the combination key of field number and fault level. Each combination key corresponds to one or more response action entries. Each response action entry records the action type, target, and execution parameters. The policy mapping mechanism completes the conversion path from field numbers to control commands, used to construct a hardware-level response instruction set.

[0132] In this embodiment, the generation of the fault detection signal includes:

[0133] The adaptive engine receives fused anomaly vectors, fault levels, and response status records in each sampling period;

[0134] The adaptive engine establishes a triplet mapping between the fused anomaly vector, fault level, and response status record based on the field number, and continuously records the fault event sequence corresponding to the corresponding field number;

[0135] After reaching the preset parameter refresh cycle, the adaptive engine traverses the fault event sequence under each field number, and counts the response success rate, field abnormality frequency and sampling cycle offset stability under different fault levels, which are used as feedback factor input parameter adjustment logic.

[0136] The parameter adjustment logic generates updated parameter sets for the CUSUM algorithm and the EWMA algorithm based on preset feedback factor weights and adjustment rules. The updated parameters of the CUSUM algorithm include the reference mean window length and drift gain coefficient, while the updated parameters of the EWMA algorithm include the smoothing factor and the abnormal trend delay threshold.

[0137] The parameter set is updated and bound to the field number to generate a fault detection signal. The fault detection signal includes the field number, the target algorithm type, and the parameter update value. It is distributed to the algorithm module in the central processing unit and the baseboard management controller through the internal configuration synchronization interface.

[0138] After receiving the corresponding fault detection signal, the central processing unit and the baseboard management controller respectively load the updated parameters of the corresponding algorithm module to complete the adaptive adjustment of the CUSUM algorithm and the EWMA algorithm.

[0139] During the real-time hardware fault detection process, the adaptive engine performs periodic parameter reconfiguration operations based on the parameter refresh cycle set by the sampling scheduling module. The parameter refresh cycle is a fixed-length cycle set during the system initialization phase based on the field number, hardware component type and sampling task density registered in the field mapping table, with the unit being sampling rounds. Whenever a parameter refresh cycle is completed, the adaptive engine aggregates and statistically analyzes the fusion anomaly vector, fault level label and response status record generated within the corresponding cycle to form a fault event sequence indexed by field number.

[0140] The feedback factors include the mutation event judgment deviation rate, the trend anomaly trigger rate, and the consistency ratio of the response execution results, which are used to measure the mutation detection accuracy, trend analysis sensitivity, and response chain reliability, respectively. For each feedback factor, the adaptive engine defines a set of normalized weight parameters based on the field number. The weight values ​​range from 0 to 1, and the sum of the weights is 1. After each parameter refresh cycle, the weight combination function is executed based on the cumulative statistical values ​​of each feedback factor within the cycle. It performs a difference analysis with the baseline value of the previous period, and the difference result is used to determine whether the automatic adjustment of the drift constant, reference window or smoothing coefficient of the CUSUM and EWMA algorithm parameters is triggered.

[0141] This embodiment includes:

[0142] The central processing unit module is used to periodically collect the operating status parameters of the target hardware components and write them into the shared memory area and the central processing unit's local cache sequence;

[0143] The mutation detection module, deployed on the central processing unit side, is used to perform bias accumulation analysis by calling the CUSUM algorithm based on the local cache sequence to generate a mutation event set;

[0144] The baseboard management controller module is used to read data from the shared memory area and perform trend analysis tasks;

[0145] The trend detection module, deployed on the baseboard management controller side, is used to call the EWMA algorithm to generate trend anomaly signals;

[0146] The field mapping management module is used to establish the mapping relationship between field numbers and hardware components, sampling tasks and processing tasks during system initialization, and supports subsequent field alignment and instruction lookup;

[0147] The fusion analysis module is used to align the field numbers and sampling period numbers of the mutation event set and the trend anomaly signal, generate fusion field anomaly items, and construct the fusion anomaly vector.

[0148] The risk assessment module is used to perform field-level scoring based on the fused anomaly vector and the preset fused feature field rule table to determine the fault level, target hardware component identification, and recommended strategy;

[0149] The response control module is used to generate a sequence of response actions based on the suggested strategy, issue response instructions, record response results, and build a response status record.

[0150] The adaptive engine module is used to update the CUSUM and EWMA algorithm parameters based on the fused anomaly vector, fault level, and response status records.

[0151] Example 1:

[0152] To verify the feasibility of this invention in practice, a comparative experiment was conducted in a server cluster environment of a large data center. The system evaluation test lasted for two months. The server environment included a total of 1,200 rack servers from two mainstream manufacturers. The servers were distributed in four physical partitions: a general business area, a high-performance computing area, a database area, and an edge computing node area.

[0153] In traditional systems, this data center primarily uses the Base Controller (BMC) to periodically collect operational data from key components and upload it to the cluster controller. The controller then uses offline algorithms to analyze historical trends and assess potential risks, with response measures including restarting, isolation, and frequency reduction. This approach has the following significant drawbacks: the BMC is limited by resources and cannot achieve high-frequency sampling, leading to system response delays; after a fault is triggered, there is a lack of integrated identification of sudden changes and trends, resulting in some slow-evolving faults not being detected in a timely manner; and the triggering of response actions mainly relies on static threshold rules, which lack accuracy and are prone to false alarms or response delays.

[0154] To address the aforementioned issues, this comparative experiment selected 600 servers as the experimental group, deploying the detection method described in this invention, while the remaining 600 servers served as the control group, continuing to use the original system's detection strategy. The experimental group system established a collaborative communication mechanism between the CPU and BMC via the LPC bus and IPMI protocol. The CPU side frequently sampled the operating status parameters of the target hardware components (such as voltage, current, core temperature, etc.) and performed cumulative deviation analysis on the data based on the CUSUM algorithm to detect abrupt events in real time. Simultaneously, the BMC side performed the EWMA algorithm to model trends on the sampled data read from the shared memory region and identified trend anomalies. Abrupt events and trend anomalies were time-aligned and field-paired through a field mapping mechanism, constructing a fused anomaly vector in the fusion analysis module. The risk assessment module scored the anomaly vector according to the fusion feature field rule table and output the fault level, target hardware component identifier, and suggested strategy. The response control module issued control commands such as interrupts, frequency reduction, or restarts according to the suggested strategy and recorded the response status. All data is fed back to the adaptive engine, which adjusts the drift factor, smoothing coefficient, and window parameters of the CUSUM and EWMA algorithms to achieve adaptive optimization of the detection strategy.

[0155] During the experimental period, the system processed approximately 24 million sampling cycles of data per day, and the detection response performance of the two systems was quantitatively evaluated. Relevant data are shown in the table below:

[0156] Table 1 Comparison of Fault Detection Performance between Experimental Group and Control Group

[0157] Indicator Items Experimental group (this invention) Control group (traditional system) Average fault detection delay (seconds) 5.3 28.7 Average number of faults identified per server per month 2.8 1.6 Fault location accuracy rate (%) 94.5 81.3 False alarm rate (%) 2.1 9.6 Response action execution success rate (%) 97.6 89.2 Average system availability improved (%) +7.8 — Average execution latency of response action (seconds) 2.4 12.6 Trend-based fault identification rate (%) 91.2 57.4

[0158] As shown in Table 1, the experimental data demonstrates that this invention significantly outperforms traditional systems in several key performance indicators. First, the average fault detection latency decreased from 28.7 seconds to 5.3 seconds, indicating that the system effectively combines high-frequency sampling with a rapid judgment mechanism, significantly improving the real-time performance of fault identification. Second, the fault location accuracy increased from 81.3% to 94.5%, reflecting greater precision in locating target components through the joint analysis of abrupt events and trend anomalies. The false alarm rate significantly decreased to 2.1%, indicating that the fusion feature field rules and dynamic scoring mechanism used in the risk assessment module effectively suppressed misjudgments. The trend-based fault identification rate reached 91.2%, far exceeding the 57.4% of traditional systems, validating the value of trend modeling and trend offset extraction at the BMC end in slow-evolution fault identification. The success rate of response actions also increased from 89.2% to 97.6%, with the average execution latency shortened to 2.4 seconds, reflecting the system's integrity and efficiency in automated control links and command responses. In addition, the overall system availability improved by 7.8%, and the average number of fault identifications per server per month increased to 2.8 times. This indicates that the system can not only identify faults faster and more accurately, but also discover more hidden problems that traditional systems have failed to capture, significantly enhancing the stability and fault recovery capabilities of data center operations.

[0159] In summary, this embodiment fully demonstrates the practical application effect of the present invention in complex data center environments, proving that the invention has significant advantages in terms of fault identification accuracy, response processing real-time performance, and continuous optimization of system adaptability. It is suitable for real-time fault monitoring and adaptive risk control in various server environments. The system possesses high scalability and versatility, and can be widely applied to various scenarios such as cloud computing platforms, edge computing nodes, and industrial control equipment.

[0160] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A hardware fault real-time detection method based on CPU and BMC cooperation, characterized in that, include: Establish a communication link and role allocation mechanism between the CPU and BMC. The CPU collects high-frequency status data and performs mutation analysis, while the BMC collects trend indicators and performs trend modeling. The two exchange data through shared cache. The CPU periodically collects transient parameters of each hardware component, forms a state data sequence, and writes it into the shared cache. At the same time, it calls the CUSUM algorithm to process the state data sequence and generate a set of mutation events. BMC reads the data sequence from the shared cache, calls the EWMA algorithm to calculate the smoothed trend curve, and extracts the trend offset to form a trend anomaly signal; The set of mutation events and trend anomaly signals are taken as input and fed into the fusion analysis module. Time alignment is performed and fusion features are calculated. The fusion anomaly vector is output. The generation of the fusion anomaly vector includes: Based on the field numbers registered in the field mapping table, the mutation event data items and trend anomaly signal items are matched, and the mutation determination timestamp and trend determination timestamp corresponding to each data item are extracted. The fusion analysis module maps two timestamps to sampling period numbers and performs time alignment operations based on the matching rule that the difference between the sampling period numbers does not exceed one period, thus establishing a fused data pair with consistent field numbers and time matching. For each pair of aligned fused data, perform field-level structural merging to generate fused field anomalies. The fused field anomalies include field number, mutation level, trend offset direction, trend offset period number, and sampling period offset. The fusion analysis module constructs a fusion anomaly vector from all fusion field anomalies generated within the current sampling period; The fused anomaly vector is input into the risk assessment module, which calculates the risk score and determines the fault level based on the numerical thresholds and combination rules in the preset fused feature fields, and outputs the location component and suggested strategy. Based on the positioning components and suggested strategies, trigger the corresponding response actions and generate a response status record; The fusion of anomaly vectors, fault levels, and response status records is used as sample input to the adaptive engine, and the CUSUM and EWMA parameters are adjusted to generate fault detection signals.

2. The CPU and BMC cooperative based hardware fault real-time detection method according to claim 1, characterized in that, The mechanism for establishing a communication link and role allocation between the CPU and BMC includes: During the system initialization phase, the collaborative management firmware is loaded, the CPU and BMC are connected via the LPC bus, and a command channel is established based on the IPMI protocol. At the same time, a shared memory region is configured in the PCIe address space for bidirectional data interaction. The shared memory is divided into a first buffer and a second buffer. The CPU writes the acquired data and the BMC writes the processing feedback, respectively. Each buffer is equipped with a flag field, including a data validity bit and a processing completion bit. After the CPU completes initialization, it registers the acquisition module and sets the hardware component number, sampling period and write target buffer address to form a high-frequency mutation monitoring task. After BMC completes initialization, it sets up trend analysis tasks and registers the reading sources, modeling frequency, and result write-back paths for performing trend data processing. The high-frequency mutation monitoring task and trend data processing generate a field mapping table. The field mapping table corresponding to the acquisition party and the processing party for each piece of hardware data is uniformly scheduled by the collaborative control module and dynamically refreshed according to the sampling status. BMC periodically polls the first buffer flag bit. When it detects that the valid bit is set, it extracts data and performs analysis. After processing, it writes the data to the second buffer and sets the processing completion bit. After the CPU detects the processing completion bit in the next cycle, it updates the sampling status and synchronously maintains the task status according to the mapping table.

3. The CPU and BMC cooperative based hardware fault real-time detection method according to claim 1, characterized in that, The generation of the mutation event set includes: The CPU performs periodic sampling operations according to the target hardware component running status parameter field identified as the high-frequency mutation monitoring task in the field mapping table. After each sampling, the current running status sample value is written to the first buffer in the shared memory area and the CPU local cache sequence. After each round of sampling, the mutation detection module calls the CUSUM algorithm to perform cumulative analysis of the execution deviation of the running status parameter fields of the corresponding target hardware components in the CPU local cache sequence. During the task initialization phase, the mutation detection module configures a set of judgment parameters for the running status parameter fields of each target hardware component, including a reference mean calculation window, a drift constant, and a trigger threshold, and binds them to the field numbers for management. When the mutation detection module determines that the sampled value of the running status parameter field in the current period relative to the cumulative offset value reaches a preset trigger threshold, it generates a mutation event data item. All mutation event data items that meet the mutation conditions within the current sampling period constitute the mutation event set.

4. The real-time hardware fault detection method based on CPU and BMC collaboration according to claim 1, characterized in that, The generation of the trend anomaly signal includes: Based on the target hardware component running status parameter field identified as the trend modeling task in the field mapping table, BMC reads the current period sampling value from the first buffer in the shared memory area according to the sampling period uniformly set with the CPU, and writes it into the BMC local cache sequence. During the task initialization phase, the trend analysis module loads judgment parameters, including short-term smoothing factor, field weight factor, trend correction factor and trend reference interval, into the running status parameter field of each target hardware component according to the trend modeling parameter configuration table, and binds them with the field number. The trend analysis module performs trend modeling on the current sampled value in the BMC local cache sequence and the saved trend value of the previous period and historical trend value, and calculates the trend value of the current period. The trend value is compared with the trend reference interval. If the trend values of continuous multiple sampling periods deviate from the trend reference interval and the deviation direction is consistent, it is determined that a trend anomaly occurs in the corresponding field. After determining that a trend anomaly has been established, a trend anomaly signal is constructed. The trend anomaly signal includes a field number, current trend value, trend offset direction, number of consecutive offset cycles, and trend determination timestamp. The trend anomaly signal is written into the second buffer in the shared memory area for the fusion analysis module to read.

5. The CPU and BMC cooperative based hardware fault real-time detection method according to claim 1, characterized in that, The output of the fault level, target hardware component location, and recommended strategy includes: The risk assessment module receives the fusion anomaly vector generated by the fusion analysis module in each sampling period, and extracts the field number, mutation level, trend offset direction, trend offset period number and sampling period offset for each fusion field anomaly item in the fusion anomaly vector; During the system initialization phase, the risk assessment module loads the fusion feature field rule table. The fusion feature field rule table establishes a mapping according to the field number, records the numerical threshold and logical combination conditions of each field of the fusion field anomaly item corresponding to each field number, and sets the corresponding risk score level range. The scoring process is performed based on the values ​​of each field in the anomaly item of the fusion field and the scoring parameters. The scoring results are matched with the level range to determine whether the triggering fault level conditions are met. When the triggering conditions are met, the corresponding fault level is assigned to the anomaly item of the fusion field. The target hardware component identifier associated with the fusion field anomaly is found in the field mapping table by the field number, and the matching suggested strategy is retrieved based on the combination key of the field number and the fault level. Encapsulate the field number, target hardware component identifier, fault level, and recommended strategy entries into a response control structure.

6. The CPU and BMC cooperative based hardware fault real-time detection method according to claim 1, characterized in that, The generation of the response status record includes: In each sampling period, the response control structure transmitted by the risk assessment module is received, and the field number, target hardware component identifier, fault level and recommended strategy are extracted. Based on the action types defined in the recommended strategy, perform action mapping operations to determine the sequence of response actions to be executed; A response instruction set is generated based on the target hardware component identifier, and instructions are issued to the corresponding target hardware component through the management interface and external interface; After each response action is completed, record the action type, execution result status code, response delay, and execution timestamp to generate a response record item; Encapsulate all response records completed within the current period into a response status record.

7. The CPU and BMC cooperative based hardware fault real-time detection method according to claim 1, wherein, The generation of the fault detection signal includes: The adaptive engine receives fused anomaly vectors, fault levels, and response status records in each sampling period; The adaptive engine establishes a triplet mapping between the fused anomaly vector, fault level, and response status record based on the field number, and continuously records the fault event sequence corresponding to the corresponding field number; After reaching the preset parameter refresh cycle, the adaptive engine traverses the fault event sequence under each field number, and counts the response success rate, field abnormality frequency and sampling cycle offset stability under different fault levels, which are used as feedback factor input parameter adjustment logic. The parameter adjustment logic generates updated parameter sets for the CUSUM algorithm and the EWMA algorithm based on preset feedback factor weights and adjustment rules. The updated parameters of the CUSUM algorithm include the reference mean window length and drift gain coefficient, while the updated parameters of the EWMA algorithm include the smoothing factor and the abnormal trend delay threshold. The parameter set is updated and bound to the field number to generate a fault detection signal. The fault detection signal includes the field number, the target algorithm type, and the parameter update value. It is distributed to the algorithm module in the central processing unit and the baseboard management controller through the internal configuration synchronization interface. After receiving the corresponding fault detection signal, the central processing unit and the baseboard management controller load the updated parameters of the corresponding algorithm modules to complete the adaptive adjustment of the CUSUM algorithm and the EWMA algorithm.

8. A hardware fault real-time detection system based on CPU and BMC cooperation, characterized in that, include: The central processing unit module is used to periodically collect the operating status parameters of the target hardware components and write them into the shared memory area and the central processing unit's local cache sequence; The mutation detection module, deployed on the central processing unit side, is used to perform deviation accumulation analysis by calling the CUSUM algorithm based on the local cache sequence to generate a mutation event set; The baseboard management controller module is used to read data from the shared memory area and perform trend analysis tasks; The trend detection module, deployed on the baseboard management controller side, is used to call the EWMA algorithm to generate trend anomaly signals; The field mapping management module is used to establish the mapping relationship between field numbers and hardware components, sampling tasks and processing tasks during system initialization, and supports subsequent field alignment and instruction lookup; The fusion analysis module is used to align the field numbers and sampling period numbers of the mutation event set and the trend anomaly signal, generate fusion field anomaly items, and construct the fusion anomaly vector. The generation of the fusion anomaly vector includes: Based on the field numbers registered in the field mapping table, the mutation event data items and trend anomaly signal items are matched, and the mutation determination timestamp and trend determination timestamp corresponding to each data item are extracted. The fusion analysis module maps two timestamps to sampling period numbers and performs time alignment operations based on the matching rule that the difference between the sampling period numbers does not exceed one period, thus establishing a fused data pair with consistent field numbers and time matching. For each pair of aligned fused data, perform field-level structural merging to generate fused field anomalies. The fused field anomalies include field number, mutation level, trend offset direction, trend offset period number, and sampling period offset. The fusion analysis module constructs a fusion anomaly vector from all fusion field anomalies generated within the current sampling period; The risk assessment module is used to perform field-level scoring based on the fused anomaly vector and the preset fused feature field rule table to determine the fault level, target hardware component identification, and recommended strategy; The response control module is used to generate a sequence of response actions based on the suggested strategy, issue response instructions, record response results, and build a response status record. The adaptive engine module is used to update the CUSUM and EWMA algorithm parameters based on the fused anomaly vector, fault level, and response status records.

Images