Fault processing method, fault processing system and medium

Abstract

The invention discloses a fault processing method, a fault processing system and a medium, and relates to the technical field of information processing, and the method comprises the steps: obtaining the historical fault information of a hardware component; obtaining current system load information, and obtaining a current storm judgment condition corresponding to each hardware component according to the historical fault information and the current system load information; according to the current sliding time window, window accumulated fault information of the hardware component is obtained, when the window accumulated fault information of the hardware component meets the corresponding current storm judgment condition, a storm trigger flag bit of the hardware component is set, and the hardware component is recorded as a storm trigger component; and receiving fault information of the storm triggering components, and setting the suppression mark position of the at least one storm triggering component when the fault information of the at least one storm triggering component meets any preset suppression triggering condition. The technical problem of low current fault processing accuracy can be solved, and the technical effects of improving the fault processing accuracy and quick response are achieved.

Description

Technical Field

[0001] This application relates to the field of information processing technologies, and in particular, to a fault handling method, a fault handling system, and a medium. Background Art

[0002] In current data centers and computing platforms, to ensure the stable operation of the system, a hardware error monitoring mechanism is usually equipped. For example, through a Reliability, Availability, Serviceability (RAS) architecture, various hardware fault errors generated during the operation of the system are detected, reported, and response-controlled.

[0003] In related technologies, most of the hardware error handling mechanisms rely on statically set fixed thresholds. Once the threshold trigger condition is met, the operating system is reported or a preset policy is executed. Since the threshold is statically configured and cannot be dynamically adjusted according to the operating environment status or fault trend in different operating stages, some fault events may be over-responded, increasing the probability of misjudgment; while other abnormal behaviors may be ignored because they do not reach the fixed threshold, resulting in missed judgments. Based on this, there is an urgent need for a fault handling method to solve the technical problem of low accuracy of current fault handling. Summary of the Invention

[0004] This application provides a fault handling method, a fault handling system, and a medium to at least solve the problem of low accuracy of fault handling in related technologies.

[0005] In a first aspect, this application provides a fault handling method, including:

[0006] Receiving fault information of at least one hardware component sent by the Basic Input / Output System (BIOS), and obtaining historical fault information of at least one hardware component according to the stored fault information;

[0007] wherein the fault information is obtained by the BIOS when a hardware component fails and triggers an interrupt;

[0008] Obtaining current system load information, and obtaining current storm determination conditions corresponding to each hardware component according to the historical fault information and the current system load information;

[0009] According to the current sliding time window, obtaining window cumulative fault information of at least one hardware component, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, setting the storm trigger flag bit of the hardware component and marking the hardware component as a storm trigger component;

[0010] Receive the fault information of at least one storm trigger component, and set the suppression flag bit of at least one storm trigger component when the fault information of at least one storm trigger component meets any preset suppression trigger condition.

[0011] In a second aspect, the present application provides a fault handling method, including:

[0012] Trigger an interrupt when at least one hardware component fails, and obtain the fault information of at least one hardware component;

[0013] Send the fault information of at least one hardware component to the baseboard control manager, so that the baseboard control manager receives the fault information of at least one hardware component, and according to the stored fault information, obtain the historical fault information of at least one hardware component; obtain the current system load information, and according to the historical fault information and the current system load information, obtain the current storm determination conditions corresponding to each hardware component; according to the current sliding time window, obtain the window cumulative fault information of at least one hardware component, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, set the storm trigger flag bit of the hardware component, and record the hardware component as a storm trigger component; receive the fault information of at least one storm trigger component, and set the suppression flag bit of at least one storm trigger component when the fault information of at least one storm trigger component meets any preset suppression trigger condition.

[0014] In a third aspect, a fault handling method of the present application includes:

[0015] The basic input / output system triggers an interrupt when at least one hardware component fails, and obtains the fault information of at least one hardware component;

[0016] The basic input / output system sends the fault information of at least one hardware component;

[0017] The baseboard management controller receives the fault information of at least one hardware component, and according to the stored fault information, obtains the historical fault information of at least one hardware component;

[0018] The baseboard management controller obtains the current system load information, and according to the historical fault information and the current system load information, obtains the current storm determination conditions corresponding to each hardware component;

[0019] The baseboard management controller obtains the window cumulative fault information of at least one hardware component according to the current sliding time window, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, sets the storm trigger flag bit of the hardware component, and records the hardware component as a storm trigger component;

[0020] The Basic Input / Output System (BIOS) monitors the storm flag bit in real time. After reading that the storm flag bit of the storm trigger component is set, it masks the fault information of the storm trigger component from being transmitted upward to the upper layer.

[0021] The Baseboard Management Controller (BMC) receives the fault information of at least one storm trigger component. When the fault information of at least one storm trigger component meets any preset suppression trigger condition, it sets the suppression flag bit of at least one storm trigger component.

[0022] The Basic Input / Output System (BIOS) monitors the suppression flag bit in real time. After reading that the suppression flag bit is set, it executes the suppression policy corresponding to the suppression flag bit.

[0023] In a fourth aspect, the present application also provides a Baseboard Management Controller (BMC), including:

[0024] A data receiving module, configured to receive the fault information of at least one hardware component sent by the Basic Input / Output System (BIOS), and obtain the historical fault information of at least one hardware component according to the stored fault information; wherein, the fault information is obtained by the Basic Input / Output System (BIOS) when a hardware component fails and triggers an interrupt.

[0025] A condition generating module, configured to obtain the current system load information, and obtain the current storm determination condition corresponding to each hardware component according to the historical fault information and the current system load information.

[0026] A storm judgment module, configured to obtain the window cumulative fault information of at least one hardware component according to the current sliding time window, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, set the storm trigger flag bit of the hardware component, and record the hardware component as a storm trigger component.

[0027] A suppression monitoring module, configured to receive the fault information of at least one storm trigger component, and when the fault information of at least one storm trigger component meets any preset suppression trigger condition, set the suppression flag bit of at least one storm trigger component.

[0028] In a fifth aspect, the present application also provides a Basic Input / Output System (BIOS), including:

[0029] An information acquisition module, configured to trigger an interrupt when at least one hardware component fails, and acquire the fault information of at least one hardware component.

[0030] An information sending module, configured to send the fault information of at least one hardware component to the baseboard control manager, so that the baseboard control manager receives the fault information of at least one hardware component, and obtains the historical fault information of at least one hardware component according to the stored fault information; obtains the current system load information, and obtains the current storm determination condition corresponding to each hardware component according to the historical fault information and the current system load information; obtains the window cumulative fault information of at least one hardware component according to the current sliding time window, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, sets the storm trigger flag bit of the hardware component, and records the hardware component as a storm trigger component; receives the fault information of at least one storm trigger component, and when the fault information of at least one storm trigger component meets any preset suppression trigger condition, sets the suppression flag bit of at least one storm trigger component.

[0031] In a sixth aspect, the present application further provides a fault handling system, including: a baseboard management controller, a basic input / output system, and a processor;

[0032] The baseboard management controller includes a microprocessor and a management control memory. The management control memory is used to store a management control program, and the microprocessor is used to execute the management control program stored in the management control memory to perform the steps of the fault handling method according to the first aspect;

[0033] The basic input / output system is used to store a basic input / output program;

[0034] The processor is used to execute the basic input / output program stored in the basic input / output system to perform the steps of the fault handling method according to the second aspect.

[0035] In a seventh aspect, the present application further provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any one of the above fault handling methods when executing the computer program.

[0036] In an eighth aspect, the present application further provides a computer-readable storage medium, in which a computer program is stored, and wherein the computer program, when executed by a processor, implements the steps of any one of the above fault handling methods.

[0037] In a ninth aspect, the present application further provides a computer program product, including a computer program, and the computer program, when executed by a processor, implements the steps of any one of the above fault handling methods.

[0038] This application constructs an adaptive error storm dynamic perception and determination mechanism by obtaining the current system load information and combining it with historical fault information to obtain the storm determination conditions for hardware components. It can dynamically adjust the determination threshold according to load fluctuations and historical fault trends, reduce misjudgment or missed judgment caused by unreasonable threshold setting, and ensure that the storm determination result is more targeted and accurate. Then, by statistically analyzing the window cumulative fault information of hardware components within the current sliding time window and determining whether it meets the corresponding storm determination conditions, the rapid identification of error storms is achieved, improving the real-time performance and accuracy of fault detection. After identifying the storm-triggering component, continue to receive its fault information. When the fault information of the storm-triggering component meets any preset condition, set its suppression flag bit for subsequent execution of targeted response strategies, ensuring that in the case of continuous intensive occurrence of faults, corresponding measures can be quickly responded to and implemented. Therefore, this application can solve the technical problem of low accuracy of current fault handling and achieve the technical effects of improving fault identification accuracy and rapid response. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] To more clearly illustrate the embodiments of this application, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 Flow diagram of the fault handling method provided by the embodiment of this application Figure 1 ;

[0041] Figure 2 Flow diagram of the method for determining the current sliding time window provided by the embodiment of this application;

[0042] Figure 3 Flow diagram of the fault handling method provided by the embodiment of this application Figure 2 ;

[0043] Figure 4 Flow diagram of the fault handling method provided by the embodiment of this application Figure 3 ;

[0044] Figure 5 Timing diagram of the fault handling method provided by the embodiment of this application;

[0045] Figure 6 Structural diagram of the baseboard management controller provided by the embodiment of this application;

[0046] Figure 7 Structural diagram of the basic input / output system provided by the embodiment of this application;

[0047] Figure 8 This is a schematic structural diagram of the electronic device provided for this application. Detailed implementation manners

[0048] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0049] It should be noted that in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0050] In the related art, the processing logic for hardware errors generally adopts a fixed threshold strategy, which appears to be insufficiently flexible when facing dynamically changing load conditions, different hardware life cycle states, and error burst patterns. For example, in a high-load or complex application environment, a large number of transient errors may occur. These errors trigger the response strategy without being fully analyzed, easily leading to misjudgment and overprocessing.

[0051] Based on the above technical problems and requirements, the inventive concept of this application lies in proposing a fault handling method. By combining historical fault information with the current system load state, a storm determination mechanism that can be adaptively adjusted is constructed, and hierarchical response control is achieved through hierarchical suppression.

[0052] To enable those skilled in the art of this technology to better understand the solution of this application, the following further detailed description of this application will be given in conjunction with the accompanying drawings and specific implementation manners.

[0053] Figure 1 This is a flowchart of the fault handling method provided for the embodiments of this application Figure 1 , as Figure 1 shown, and includes:

[0054] S11. Receive the fault information of at least one hardware component sent by the basic input / output system, and obtain the historical fault information of at least one hardware component according to the stored fault information.

[0055] In this embodiment, first, fault information of at least one hardware component sent by the Basic Input Output System (BIOS for short) is received to sense the abnormal state at the current hardware level. The Basic Input Output System can monitor various hardware components before the operating system is loaded and generate interrupt events. Once an abnormality occurs in a certain hardware component during operation, the BIOS will obtain and send the fault information according to the preset interrupt response mechanism. After receiving the fault information, historical fault information corresponding to the hardware component will be retrieved in a local or other storage structure. By obtaining the historical faults and retrospectively modeling the historical behavior patterns of the hardware components, it provides a data basis for subsequent steps such as error trend analysis, storm identification, and dynamic threshold calculation.

[0056] S12, obtain the current system load information, and based on the historical fault information and the current system load information, obtain the current storm determination conditions corresponding to each hardware component.

[0057] In this embodiment, by obtaining the current system load information and combining it with the historical fault information, the storm determination conditions of each hardware component in the current operating state are determined, aiming to provide a dynamic threshold basis for subsequent fault storm identification. Among them, the load information refers to the comprehensive state data reflecting the operating pressure of the entire system at a certain moment, usually from the operation state monitoring of multiple subsystems such as the processor, memory, storage, and network. The way to obtain the system load information can be based on the resource monitoring interface provided by the operating system, the underlying hardware sensors, or the embedded management module, which can be determined according to the actual deployment environment. The historical fault information reflects the fault characteristics exposed by each hardware component during past operation, such as error frequency, type, timing pattern, etc., and is an important basic data for analyzing the stability of the hardware component.

[0058] The historical fault information is combined with the current load state to construct a strategy for dynamic determination, so as to be able to adjust the tolerance for fault behaviors according to the current load level of the system, making it more sensitive to the same type of errors in a high-load environment, thereby improving the timeliness and accuracy of fault response. On the contrary, when the load is light, the tolerance threshold can be appropriately increased to reduce overreaction to minor occasional errors. Finally, based on the above analysis results, a current storm determination condition is generated for each hardware component as the determination benchmark for identifying abnormal fault behaviors in subsequent steps. This mechanism realizes real-time update and personalized setting of the determination thresholds for different hardware components, solves the problem of insufficient accuracy caused by the inability of traditional static thresholds to adapt to changes in the operating state, and improves the adaptability and robustness of the entire fault handling.

[0059] S13. Obtain the window cumulative failure information of at least one hardware component according to the current sliding time window, and when the window cumulative failure information of the hardware component meets its corresponding current storm determination condition, set the storm trigger flag bit of the hardware component and record the hardware component as a storm trigger component.

[0060] In this embodiment, first determine the current sliding time window to define the observation interval for failure statistics. The sliding time window is a continuously updated time period used to track and analyze the distribution of failure events during recent operation. Its setting ensures the timeliness and continuity of failure statistics. After the time window is determined, collect the cumulative failure information of at least one hardware component within this time window. This cumulative failure information is the aggregated statistical result of historical failure data within the sliding time window, usually recorded separately by failure type, such as the occurrence frequencies of correctable errors and uncorrectable errors, etc. This information can reflect the operating stability of the hardware component during this time period.

[0061] Subsequently, compare the current window cumulative failure information according to the previously obtained storm determination conditions. The storm determination conditions are used to define what degree of failure density should be recognized as an abnormal storm state. If the cumulative failure information of a certain hardware component within the sliding time window meets its corresponding storm determination condition, that is, set the storm trigger flag bit of this hardware component, indicating that this hardware component has entered a state that may trigger a failure storm. The hardware component with the set flag bit will be marked as a storm trigger component for subsequent implementation of targeted failure suppression and isolation strategies. This embodiment effectively identifies the components with intensive failure outbreaks by introducing the sliding time window statistical mechanism, realizes the early detection and response to error storms in failure handling, and improves the automation and accuracy of failure handling.

[0062] S14. Receive the failure information of at least one storm trigger component, and when the failure information of at least one storm trigger component meets any preset suppression trigger condition, set the suppression flag bit of at least one storm trigger component.

[0063] In this embodiment, by continuously monitoring the fault behavior of the storm trigger component and further determining its status based on the preset suppression trigger conditions, dynamic control and active intervention of the influence range of the fault storm are achieved. After identifying the storm trigger component, continuously receive the subsequent fault information generated by the storm trigger component to characterize whether the current status of the storm trigger component tends to deteriorate or has a trend of continuously disturbing operation. After receiving the fault information of at least one storm trigger component, compare and judge the current fault status according to the preset suppression trigger conditions. The suppression trigger conditions are used to reflect different levels of risk scenarios and may include multiple dimensions such as the fault frequency threshold, error density, outbreak trend within a short period of time, and whether there is linkage of key components. The diversity and hierarchical setting of the preset suppression trigger conditions enable more detailed and accurate response strategies to be made in the face of complex fault scenarios. When the fault information of a certain storm trigger component meets any suppression trigger condition, the suppression flag bit of the storm trigger component is set. The setting of the suppression flag bit is not only a mark of the current fault status of the storm trigger component but also represents that the storm trigger component has triggered a specific prevention and control mechanism. In this embodiment, through continuous tracking and analysis of the subsequent fault behavior of the storm trigger component, dynamic scenario stratification and accurate intervention after storm identification are realized, ensuring that the fault storm is effectively contained before it spreads widely, and reducing the occurrence of performance degradation or irrecoverable fault problems caused by persistent faults.

[0064] Figure 2 It is a schematic flowchart of the method for determining the current sliding time window provided by the embodiment of the present application. On the basis of the above embodiment, it includes:

[0065] S201, obtain the time slot width corresponding to the previous sliding time window, and obtain the window cumulative fault information corresponding to the previous sliding time window;

[0066] S202, calculate and obtain the error density corresponding to the previous sliding time window according to the window cumulative fault information;

[0067] S203, obtain the time slot width corresponding to the current sliding time window according to the error density, the time slot width, and the preset time slot width sequence;

[0068] S204, based on the time slot width corresponding to the current sliding time window, determine the start time and end time of the current sliding time window, and complete the determination of the current sliding time window.

[0069] In this embodiment, to ensure the dynamic adaptability of the storm trigger condition, the time slot width of the sliding time window is adjusted to match the occurrence density of fault events in the current system. First, obtain the time slot width corresponding to the previous sliding time window. The time slot width is the unit time length that divides the entire time window into several consecutive time periods and is used for subsequent time attribution and classification statistics of fault events. At the same time, also obtain the window cumulative fault information corresponding to the previous sliding time window, that is, the data of various fault events collected and summarized within this time window. Based on the fault information, further calculate the error density. The error density refers to the number of fault events occurring per unit time and reflects the current activity level of faults. The process of obtaining the error density usually involves the statistics of all fault events within the time window and normalization in combination with the total length of the time window to obtain a density value with time correlation.

[0070] Subsequently, based on the obtained error density, time slot width, and the preset time slot width sequence, determine whether it is necessary to adjust the time slot width corresponding to the current sliding time window. The preset time slot width sequence is a set of time lengths with multiple different granularities and is used to achieve dynamic granularity control. When the error density is high, a smaller time slot granularity can be selected to improve the event response accuracy; when the error density is low, a larger time slot granularity can be selected to reduce the system load. Finally, based on the time slot width corresponding to the current sliding time window, determine the start time and end time of the current sliding time window. This process needs to refer to the end time of the last time window and calculate the new time window boundary in combination with the new slot width to complete the update of the sliding time window. Through this embodiment, the dynamic adjustment of the monitoring range of fault events in the time dimension is realized, providing a more accurate time window basis for subsequent storm determination and suppression mechanisms.

[0071] In a specific embodiment, a specific implementation manner is provided for step S23 above. On the basis of the above embodiment, according to the error density, time slot width, and the preset time slot width sequence, obtaining the time slot width corresponding to the current sliding time window includes:

[0072] S2301, obtain the preset time slot width sequence; where the time slot width sequence includes multiple slot width granularities arranged in descending order.

[0073] S2302, if the error density is greater than or equal to the preset first density threshold and the time slot width is not the smallest slot width granularity in the time slot width sequence, then select the next slot width granularity corresponding to the time slot width from the time slot width sequence and record the next slot width granularity as the time slot width corresponding to the current sliding time window.

[0074] S2303, if the error density is greater than a preset second density threshold and less than a preset first density threshold, set the time slot width to the time slot width corresponding to the current sliding time window;

[0075] S2304, if the error density is less than or equal to the preset second density threshold and the time slot width is not the largest slot width granularity in the time slot width sequence, select the previous slot width granularity corresponding to the time slot width from the time slot width sequence and record the previous slot width granularity as the time slot width corresponding to the current sliding time window.

[0076] In this embodiment, in order to achieve dynamic adjustment of the time slot granularity under different fault density levels to improve the accuracy of fault detection and the adaptability of response, the time slot width adopted by the current sliding time window is flexibly configured through a preset time slot width sequence. First, obtain the preset time slot width sequence, which is a set composed of multiple slot width granularities with different time lengths and is arranged in descending order. This sequence is designed to select an appropriate time granularity for window division when the fault event density changes, ensuring both detection sensitivity and overall computational load. On this basis, compare the error density corresponding to the current time window with two preset density thresholds.

[0077] If the current error density is greater than or equal to the first density threshold and the currently adopted time slot width is not the smallest slot width granularity listed in the time slot width sequence, it is considered that the current fault activity is relatively high, and it is necessary to further refine the monitoring granularity to improve the timeliness of error event location and response. Therefore, advance the current time slot width one position backward in the sequence and select the corresponding next slot width granularity as the new time slot width that the current sliding time window should adopt.

[0078] If the current error density is between the first density threshold and the second density threshold, it is judged that this density level is in an intermediate state, neither an anomaly occurs nor it belongs to the range of low density that can be ignored. Therefore, keep the current time slot width unchanged to maintain the stability and continuity of the overall operation. In this case, the time slot granularity will not be reduced or enlarged, and the granularity determined by the previous sliding time window is still adopted.

[0079] If the current error density is less than or equal to the second density threshold and the current time slot width is not the largest slot width granularity listed in the time slot width sequence, it is judged that the fault density is relatively low, and it is necessary to reduce the system statistical burden and the possibility of over-response to events by expanding the time granularity. At this time, select the previous granularity of the current slot width, that is, a larger granularity slot width, from the time slot width sequence as the new time slot width that the current sliding time window should adopt.

[0080] In this embodiment, through the determination and adjustment for different fault density levels, an adaptive update mechanism for the time window granularity is realized, providing a more reasonable time division basis for subsequent window fault statistics and storm judgment. This mechanism enhances the ability to capture the fault outbreak trend and reduces unnecessary resource occupation during the inactive period.

[0081] Further, in another specific embodiment, determining the current sliding time window further includes:

[0082] S211, obtaining the time slot width corresponding to the previous sliding time window, and receiving the fault information of the hardware components in real time, and according to the end time of the previous sliding time window, counting the real-time error density of the hardware components after the end time;

[0083] S212, if the real-time error density is greater than or equal to the preset third density threshold, and the time slot width is not the smallest slot width granularity in the time slot width sequence, then select the next slot width granularity corresponding to the time slot width from the time slot width sequence, and record the next slot width granularity as the time slot width corresponding to the current sliding time window;

[0084] S213, if the real-time error density is greater than the preset fourth density threshold and less than the preset third density threshold, then set the time slot width as the time slot width corresponding to the current sliding time window;

[0085] S214, if the real-time error density is less than or equal to the preset fourth density threshold, and the time slot width is not the largest slot width granularity in the time slot width sequence, then select the previous slot width granularity corresponding to the time slot width from the time slot width sequence, and record the previous slot width granularity as the time slot width corresponding to the current sliding time window;

[0086] S215, based on the time slot width corresponding to the current sliding time window, determine the start time and end time of the current sliding time window, and complete the determination of the current sliding time window.

[0087] In this embodiment, the same as the previous embodiment, obtain the time slot width corresponding to the previous sliding time window. Subsequently, continuously receive the fault information reported by each hardware component, and based on these real-time reported fault events, count the real-time error density from the end time of the previous sliding time window to the current time. This real-time error density can be calculated by dividing the number of errors per unit time by the corresponding time interval length, and is used to reflect the current fault activity level.

[0088] If the statistically obtained real-time error density is greater than or equal to a preset third density threshold, and the currently adopted time slot width is not the smallest slot width granularity in the time slot width sequence (i.e., it can still be further refined), then according to the order of the time slot width sequence, select the next slot width granularity corresponding to the current slot width from it, and set it as the time slot width corresponding to the current sliding time window. By reducing the time slot granularity, sudden changes in errors can be captured on a shorter time scale, thereby enhancing the early recognition ability of storm errors.

[0089] If the real-time error density is between the preset fourth density threshold and the third density threshold, the current time slot width remains unchanged. The existing time granularity can already adapt well to the current fault frequency, and there is no need to adjust the granularity setting of the sliding time window.

[0090] If the real-time error density is less than or equal to the fourth density threshold, and the currently used time slot width is not the largest slot width granularity in the time slot width sequence (i.e., it can still be further relaxed), then according to the time slot width sequence, select the previous slot width granularity corresponding to the current slot width from it, and set it as the time slot width of the current sliding time window. By relaxing the time slot granularity, unnecessary fine-grained processing can be reduced in low error activity scenarios, thereby reducing resource occupancy and improving monitoring efficiency.

[0091] Finally, based on the currently selected time slot width, combined with the latest error density change situation, determine the start time and end time of the current sliding time window, and then complete the boundary definition of the current sliding time window. This embodiment realizes the adaptive adjustment of the sliding time window to the real-time error density of hardware components, effectively improving the sensitivity and stability of fault handling.

[0092] In one embodiment, for obtaining the historical fault information of at least one hardware component in the above step S11, a specific implementation manner is provided as follows:

[0093] S111, receive the fault metadata of at least one hardware component sent by the basic input / output system;

[0094] S112, receive and parse the fault metadata, and store the parsed fault metadata into the system event log; wherein, the system event log stores the historical fault information corresponding to at least one fault metadata;

[0095] S113, based on the historical fault information stored in the system event log, map the historical fault information to a preset fault monitoring buffer according to the preset time length of the sliding time window;

[0096] S114. Filter and extract the historical fault data of each hardware component from a preset fault monitoring buffer according to the start time and end time of the current sliding time window. The fault monitoring buffer is indexed by timestamps and records the fault events that occur at different time points for each hardware component and the corresponding fault types of the fault events.

[0097] In this embodiment, fault metadata of at least one hardware component transmitted by the basic input / output system is received. The fault metadata is usually automatically triggered and reported by the underlying firmware after an abnormality occurs in the hardware component and includes fields such as the timestamp of the fault occurrence, the fault type identifier, and the fault source component identifier. After receiving the fault metadata, a parsing operation is performed to extract fields and normalize the format of the original fault information, and the normalized fault metadata structure is stored in the system event log. The system event log, as a unified error event storage medium, can continuously record the fault information that has occurred in multiple hardware components, ensuring that subsequent analysis can be performed based on a complete historical track.

[0098] Subsequently, according to the preset time length of the sliding time window, the historical fault information recorded in the system event log is mapped by timestamp and written into the preset fault monitoring buffer. The buffer is divided into multiple storage units with a fixed time span, each unit corresponding to a time slot, and based on the fault timestamp as the index, it classifies and records the fault events of different hardware components at each time point. This mapping process ensures that the fault data can be accurately classified into the corresponding time slots by comparing the event timestamp with the time boundaries of each slot in the sliding time window, facilitating subsequent time window statistical processing. On this basis, according to the start time and end time of the current sliding time window, the data content in the corresponding time period in the fault monitoring buffer is retrieved, and the fault information that occurs to each hardware component during this time period is filtered and extracted to form the historical fault data set under the current window. By constructing a buffer structure with a time index and a sliding window mapping mechanism, the orderly organization and fast access of the hardware component fault history are realized, providing a controllable and traceable error data support basis for the fault handling method, and improving the accuracy and efficiency in the fault analysis process.

[0099] Exemplarily, access the system event log. In a log query, obtain the 26th event record according to a preset program execution command. The content of this record includes detailed fields such as the event number, record type, timestamp, event source device, sensor type, event type, and event direction. For example, the sensor type of this event is A and the event description is B, indicating that this event corresponds to a correctable bus-level hardware error.

[0100] In a specific embodiment, an implementation manner is provided for the above step S12. On the basis of the above embodiment, according to historical fault information and current system load information, obtain the current storm determination conditions corresponding to each hardware component, including:

[0101] S121, collect and obtain the current system load information, and normalize the current system load information to obtain the normalized value Load corresponding to the current load information;

[0102] S122, according to the current sliding event window, obtain the historical correctable error density of the hardware component; wherein, the historical correctable error density is the correctable error density of the previous sliding event window; and / or, obtain the historical uncorrectable error density of the hardware component; wherein, the historical uncorrectable error density is the uncorrectable error density of the previous sliding event window;

[0103] S123, use the formula:

[0104]

[0105] Calculate and obtain the first trigger density Threshold1, and according to the first trigger density and the preset time window length, obtain the correctable error storm trigger value of the hardware component; wherein, Base1 is the preset correctable error base threshold, α1 is the preset first load adjustment coefficient, β1 is the preset first error density adjustment coefficient, D1 is the historical correctable error density, and k1 is the preset first historical influence sensitivity; and / or

[0106] Calculate and obtain the second trigger density Threshold2; and according to the second trigger density and the preset time window length, obtain the uncorrectable error storm trigger value of the hardware component; wherein, Base2 is the preset uncorrectable error base threshold, α2 is the preset second load adjustment coefficient, β2 is the preset second error density adjustment coefficient, D2 is the historical correctable error density, and k2 is the preset second historical influence sensitivity.

[0107] In this embodiment, the current system load information is collected. The system load information generally includes, but is not limited to, parameters such as the current processor utilization rate, memory occupancy rate, I / O bandwidth occupancy, task queue length, etc. For the convenience of use in subsequent algorithms, the original system load information needs to be processed to generate normalized values with a unified dimension. Exemplarily, any load information (such as the processor utilization rate) can be selected, and the corresponding normalized value can be calculated as the normalized value corresponding to the current load information; or, multiple load information (such as the processor utilization rate and memory occupancy rate) can be selected, the normalized value of each load information is calculated respectively, and then, according to the weight corresponding to each load information, the weighted sum is obtained as the normalized value corresponding to the current load information. It should be noted here that normalization can be carried out by using existing technical means, and no specific limitation is made here. On the basis of obtaining the normalized current load value, further according to the current sliding time window, the historical correctable error density and / or historical uncorrectable error density of each hardware component are obtained. The historical error density represents the frequency of cumulative errors occurring per unit time and is a key indicator for measuring the frequency of error occurrence. The correctable error density reflects the perturbation intensity of non-fatal faults in the system, while the uncorrectable error density focuses more on the level of system reliability.

[0108] On the basis of obtaining the system load information and historical error density, the set non-linear combination function is used to calculate the correctable error storm trigger value and uncorrectable error storm trigger value corresponding to each hardware component respectively. Among them, the formula contains three important variables: one is the basic threshold term, which represents the trigger standard under ideal load and historical clean state (the set range for the corresponding correctable error is usually 3 - 10 times per second); the second is the load adjustment term, which is used to dynamically reflect the effect of the current load on raising the trigger threshold; the third is the historical influence term, which introduces the error density and adjusts the response intensity through the historical influence sensitivity parameter. This combination function takes into account the current load level and historical error behavior, enabling the storm determination threshold to have an adaptive adjustment ability when the system state changes, and reducing misjudgment or missed reports caused by static thresholds. In this embodiment, by introducing two dynamic factors of normalized system load and historical error density, combined with the preset adjustment coefficient and non-linear adjustment formula, a storm trigger determination standard that can automatically adjust with the system operation state is constructed, realizing a sensitive response to abnormal fluctuations of hardware components and effectively enhancing the adaptability and accuracy of the error storm determination mechanism.

[0109] In a specific embodiment, a specific implementation manner is provided for obtaining the window cumulative fault information of the hardware component in step S13 above. On the basis of the above embodiment, it includes:

[0110] S131. Obtain the start time and end time of the current sliding time window, and the time slot width corresponding to the current sliding time window; wherein, the time slot width is used to divide the current sliding time window into multiple consecutive and non-overlapping time slots;

[0111] S132. Extract the historical fault information of one or more hardware components from a preset fault monitoring buffer according to the start time and end time of the current sliding time window;

[0112] S133. Map the historical fault information to each time slot within the current sliding time window according to the occurrence timestamp corresponding to the historical fault information, and map the collected real-time fault information to the corresponding time slot;

[0113] S134. Classify and count the fault information of each hardware component based on the fault type of the historical fault information, and obtain the cumulative number of correctable errors and / or the cumulative number of uncorrectable errors of at least one hardware component within the current sliding time window.

[0114] In this embodiment, the start time and end time of the current sliding time window are obtained, and the time slot width corresponding to the current sliding time window is determined. The sliding time window represents the current time range for analyzing the hardware fault behavior, and its start time and end time jointly define the time span of the window. By setting the time slot width, granular time positioning and aggregation statistics of fault events can be performed. Subsequently, according to the start and end times of the current sliding time window, the historical fault information of the target hardware component is extracted from a preset fault monitoring buffer. The fault monitoring buffer is used to cache and store the fault records reported by multiple hardware components during operation in timestamp order. Each piece of historical fault information includes fields such as the timestamp of the fault occurrence, the corresponding hardware component identifier, and the fault type identifier. After the historical fault information is extracted, it is necessary to determine whether it is within the range of the current sliding time window based on the occurrence timestamp corresponding to each piece of information, and map the historical fault information that meets the conditions to the corresponding time slots within the current time window. At the same time, to ensure the real-time nature of the statistics, the newly collected real-time fault information in the current period also needs to be mapped to the corresponding time slots, so as to ensure the integrity and accuracy of the fault data within the time window.

[0115] After completing the mapping of fault information at the time slot level, it is necessary to further classify and count the fault information of each hardware component in each time slot based on the fault type identifier. Common fault types may include correctable errors (such as memory error correction code check errors) and uncorrectable errors (such as fatal central processing unit failures). The results of the classification and counting will be respectively summarized as the cumulative number of correctable errors and the cumulative number of uncorrectable errors of each hardware component within the current sliding time window, which are used for subsequent storm determination. In this embodiment, by introducing an error mapping and classification counting mechanism at the time slot granularity, the structured management and statistical aggregation of hardware fault events are realized, providing accurate data support for error density analysis and storm trigger judgment within the sliding window, thereby improving the response efficiency and judgment reliability in the face of high-frequency error events.

[0116] Further, in a specific embodiment, for the window cumulative fault information of the hardware component in step S13 above to meet its corresponding current storm determination condition, a specific judgment form is provided here. Based on the above embodiment, it includes:

[0117] S135, according to the window cumulative fault information of the hardware component, obtain the cumulative number of correctable errors corresponding to the hardware component, and obtain the correctable error storm trigger value corresponding to the hardware component;

[0118] S136, if the cumulative number of correctable errors is greater than or equal to the correctable error storm trigger value, it is determined that the window cumulative fault information of the hardware component meets its corresponding current storm determination condition;

[0119] S137, obtain the cumulative number of uncorrectable errors corresponding to the hardware component, and obtain the uncorrectable error storm trigger value corresponding to the hardware component;

[0120] S138, if the cumulative number of uncorrectable errors is greater than or equal to the uncorrectable error storm trigger value, it is determined that the window cumulative fault information of the hardware component meets its corresponding current storm determination condition.

[0121] In this embodiment, based on the previously statistically obtained window cumulative fault information, the cumulative number of correctable errors and the cumulative number of uncorrectable errors of each hardware component within the current sliding time window are respectively obtained. The window cumulative fault information is the result of aggregating and statistically analyzing the historical fault data collected and classified within the current sliding time window. Correctable errors usually refer to non-fatal errors that occur at the hardware level but can be restored through an automatic repair mechanism, which do not immediately affect system stability, but their frequent occurrence may indicate potential hardware degradation. Uncorrectable errors usually indicate relatively serious faults that occur during device operation and cannot be repaired through self-recovery means.

[0122] Next, obtain the correctable error storm trigger value and the uncorrectable error storm trigger value for each current hardware component. Compare the cumulative number of correctable errors recorded in the window cumulative fault information with its corresponding correctable error storm trigger value. If the former is greater than or equal to the latter, it indicates that the occurrence frequency of correctable errors of this hardware component within the current sliding time window has reached the storm determination threshold, and it is determined that the current storm determination condition is met. Similarly, if the cumulative number of uncorrectable errors of this component within the current sliding time window is greater than or equal to its corresponding uncorrectable error storm trigger value, it can also be determined that the storm determination condition is met. In this embodiment, by setting a storm trigger threshold based on dynamic calculation and comparing the fault aggregation statistical results within the sliding time window, the judgment process of whether the error behavior of the hardware component reaches the error storm determination standard is realized. It not only considers the severity of the error type, but also incorporates the dynamic adjustment of the system operation state and historical characteristics, improving the accuracy and scenario adaptability of the fault storm determination, thereby effectively supporting the triggering and execution of suppression strategies in subsequent fault handling.

[0123] In one embodiment, the fault handling method further includes:

[0124] S15, if the cumulative number of correctable errors of the storm-triggered component is less than or equal to a preset number threshold within a preset sliding waiting time window, cancel the setting of the storm trigger flag bit of the storm-triggered component.

[0125] In this embodiment, a preset sliding waiting time window is set. Within this time window, continuously monitor the error behavior of each hardware component with the storm trigger flag bit set. During the sliding waiting time window, the cumulative number of correctable errors of the storm-triggered component is updated again at a set time interval, and these values are compared with the preset number threshold. This number threshold reflects the tolerance range for minor error fluctuations in the stable operation state. If the cumulative number of correctable errors of the storm-triggered component is less than or equal to this number threshold throughout the sliding waiting time window, it indicates that the error behavior of this hardware component has not continued to deteriorate or has tended to be stable.

[0126] When the above judgment condition is satisfied, it is determined that the hardware component no longer has the risk of continuously triggering a storm. Therefore, the previously set storm trigger state will be revoked. Specifically, the setting of the storm trigger flag corresponding to the hardware component will be cancelled, so that the reporting situation of the hardware component returns to the normal state and it will no longer be the target component in the subsequent suppression strategy. The revocation of this flag reduces unnecessary suppression measures for this component. In this embodiment, by introducing a sliding waiting time window, a dynamic exit mechanism during error behavior mitigation is realized, improving the stability of the entire error storm determination and suppression. In addition, this mechanism also improves the adaptability to the natural fluctuations of correctable errors, ensuring sufficient fault tolerance and time flexibility during the storm identification process.

[0127] In a specific embodiment, for step S14 above, a specific trigger embodiment of the flag setting situation is provided here, that is, multiple suppression trigger conditions are preset, and different levels of error flags are triggered according to different conditions. On the basis of the above embodiment, it includes:

[0128] S141, within the first preset time range, if the cumulative number of correctable errors of the storm trigger component is greater than or equal to the first threshold, and the corresponding correctable error density is less than the first density threshold, then set the observation-level flag of the storm trigger component.

[0129] In this embodiment, the cumulative situation of correctable errors of all storm trigger components is continuously monitored, such as single-bit memory errors, transmission error retries, etc. The first preset time range is set to limit the observation period for error behavior analysis. For example, this time range can be set from a few minutes to ten minutes. Within this time range, the correctable error data of the target storm trigger component is obtained, the cumulative number of correctable errors within this time range is statistically obtained, and the error density within this time period is calculated. If the cumulative number of correctable errors of the storm trigger component is greater than or equal to the set first threshold, it means that the number of faults of this component within the observation period has reached a certain level. At the same time, it is detected that the correctable error density of this component is still lower than the first density threshold, indicating that although the cumulative number is high, the error behavior distribution is relatively uniform and has not shown typical high-density error storm characteristics, and the overall operation has not been severely disturbed. For example, the cumulative number of correctable errors of the same storm trigger component is greater than or equal to 100 times within 10 minutes, and its corresponding correctable error density is less than 0.5 times / second. When the above judgment conditions are all met, the observation-level flag of the storm trigger component is set. This observation-level flag is the lowest-level flag in hierarchical suppression, indicating that this component is currently in a suspicious state and does not trigger actual suppression measures. By setting up the observation-level flag, the early identification and classification management of potential problems are realized, which helps to carry out trend warning before the storm breaks out and improves the accuracy of fault prediction.

[0130] S142. If the cumulative number of correctable errors of the storm trigger component is greater than or equal to the second threshold within the second preset time range, set the warning level flag bit of the storm trigger component.

[0131] In this embodiment, continuously track the correctable error behaviors of each storm trigger component and set the second preset time range. Different from the observation level, the warning level mainly focuses on the absolute increase in the number of errors rather than whether the error density changes drastically. Within the second preset time range, obtain the correctable error data of the target component and count the total number of correctable errors that occurred in this component during this time period. If the cumulative number is greater than or equal to the second threshold preset by the system, it is determined that this component is in a state of frequent errors, indicating that its hardware performance or electrical characteristics may be continuously deteriorating, and there is a problem of functional failure in the short to medium term. After confirming that the cumulative number of errors reaches the second threshold, set the warning level flag bit of this storm trigger component. For example, the cumulative number of correctable errors of the same storm trigger component is greater than or equal to 50 times within 1 minute. Compared with the observation level, the suppression strategy corresponding to the warning level flag bit is usually more stringent. For example, it may trigger local performance restrictions, load transfer suggestions, or enter the detailed diagnosis stage of hardware failures. This embodiment enhances the response speed and handling accuracy for fault trends, reduces the potential chain risks brought by hardware failures while ensuring system availability.

[0132] S143. When receiving the uncorrectable error information of the storm trigger component, set the warning level flag bit of the storm trigger component.

[0133] In this embodiment, a response strategy with a higher priority is implemented for uncorrectable errors to ensure that the fault source can be identified and controlled in a timely manner when a serious hardware failure signal appears. Uncorrectable errors usually indicate that some functions of the hardware component have been damaged, or the error cannot be restored by software means. If not responded to in a timely manner, it may cause system crashes, data loss, or hardware damage. Therefore, after receiving the uncorrectable error information of the storm trigger component, without waiting for cumulative analysis, immediately set the warning level flag bit of this storm trigger component. By setting the warning level flag bit, this hardware component can be incorporated into the risk management mechanism in a timely manner. For example, once this flag bit is triggered, it can synchronously notify the operating system to execute emergency measures such as device isolation, driver uninstallation, or service sinking, and at the same time generate high-priority alarm entries in the system event log for subsequent review by operation and maintenance personnel. This embodiment realizes the fast positioning and intervention ability for serious errors in the fault storm handling system by strengthening the response ability to uncorrectable errors, effectively reducing the spread of hardware damage or service interruption caused by delayed response, thus ensuring continuous availability and fault tolerance in a complex load environment.

[0134] S144. If the cumulative number of uncorrectable errors of the storm trigger component is greater than or equal to the third threshold within the third preset time range, set the isolation level flag of the storm trigger component.

[0135] In this embodiment, within the preset third preset time range, continuous monitoring and statistics of uncorrectable error events are performed on each piece of hardware marked as a storm trigger component. If the cumulative number of its uncorrectable errors is greater than or equal to the set third threshold, immediately set the isolation level flag of the hardware component. For example, the cumulative number of uncorrectable errors of the same storm trigger component is greater than or equal to 3 times within 10 seconds. Uncorrectable errors usually include faults such as bus communication interruptions and device self - test failures, which cannot be automatically repaired at the software level. Their repeated occurrence often indicates that the component has reached a serious degradation or is about to fail. By setting the third preset time range, it is ensured that the monitoring window has sufficient granularity to capture high - frequency error bursts, and at the same time, it will not misjudge short - term abnormal fluctuations as serious faults due to short - term abnormal fluctuations. Once the cumulative number exceeds the set threshold, the operating state of the hardware component will be elevated to the isolation level through the flag mechanism.

[0136] S145. Record the number of storm trigger components whose received correctable error count is greater than or equal to the fourth threshold within the fourth preset time range. If the number is greater than or equal to the first quantity threshold, set the isolation level flag corresponding to one or more storm trigger components.

[0137] In this embodiment, horizontal correlation analysis is performed based on group - type fault behaviors to identify potential local fault outbreaks or common - cause fault propagations. Within the fourth preset time range, continuously monitor and record all hardware components in the storm trigger state, focusing on screening storm trigger components whose cumulative correctable error count is greater than or equal to the fourth threshold, and count their number. If this number reaches or exceeds the first quantity threshold, it is determined that there may be multiple hardware components in the current system that simultaneously exhibit an abnormal active state within a similar time period, which may cause problems such as resource contention, interrupt storms, or system stability degradation.

[0138] Exemplarily, the statistical process adopts a sliding time window mechanism and unifies the analysis of the abnormal behaviors of multiple components in the time dimension through a preset event aggregation strategy, ensuring the identification of not only the components with individual abnormal activities but also the trend of abnormal activities at the overall level. The fourth threshold is used to limit the minimum error quantity threshold required for each component to ensure that the components being statistically analyzed are indeed in a continuous abnormal state; the first quantity threshold defines the lower limit at which the system considers the number of abnormal components to reach the level requiring systematic intervention. For example, the cumulative number of correctable errors (i.e., the number of received correctable errors) of three storm-triggering components within 1 minute is greater than or equal to 100 times. Once the above-mentioned aggregation abnormal triggering conditions are met, a unified suppression action is performed on these storm-triggering components that have met the error quantity threshold by setting the isolation level flag bit of the corresponding component to activate the corresponding isolation mechanism, such as closing the corresponding channel, pausing the task scheduling of the component, restricting the access priority, or stripping from the running state resource pool. Through the dual judgment method of the component quantity and the error accumulation degree, this embodiment not only enhances the identification ability of group error storms but also improves the coordination of fault response and the overall fault tolerance ability.

[0139] S146. Within the fifth preset time range, record the number of storm-triggering components that receive uncorrectable errors. If the number is greater than or equal to the second quantity threshold, set the severity level flag bits of multiple storm-triggering components.

[0140] In this embodiment, within the fifth preset time range, continuously monitor the fault reporting information from each storm-triggering component. Meanwhile, count the number of storm-triggering components that have cumulatively occurred uncorrectable errors within the current time range. If this number reaches or exceeds the second quantity threshold, it means that multiple key components have simultaneously experienced serious fault events within a short period. For example, more than or equal to 3 storm-triggering components trigger uncorrectable errors within 30 seconds. This fault mode often has systematic risks, which may lead to partial service interruption, service quality degradation, or affect the execution of the high availability strategy. Therefore, based on the suppression strategy framework, set the severity level flag bits of all these storm-triggering components to trigger the corresponding level of suppression action. By using uncorrectable errors as the judgment basis and the number of faulty components as the measurement scale, a unified identification and marking mechanism for multiple-point serious hardware faults is constructed, realizing the rapid response and control of the system's serious risk state and enhancing the accuracy and comprehensiveness of fault handling.

[0141] S147. If the joint density of correctable errors and uncorrectable errors of multiple storm-triggering components within a unit time is greater than the preset joint density threshold, and there is an associated relationship in the physical topology structure of multiple storm-triggering components, set the severity level flag bits corresponding to the multiple storm-triggering components.

[0142] In this embodiment, a determination mechanism for the fault aggregation trend of a local hardware area is established by setting the joint density of correctable errors and uncorrectable errors as a composite index and combining the physical topology structure among multiple storm trigger components. Specifically, first, within a unit time window, the occurrence frequencies of correctable errors and uncorrectable errors of each storm trigger component are statistically counted in real time, and the total occurrence amounts of the two types of errors are normalized into a unified error density value. To determine whether there is a local fault area with a risk of collaborative failure, the error density values of multiple storm trigger components are accumulated to calculate their joint density. The joint density is the normalized value of the total frequency of correctable errors and uncorrectable errors reported by all storm trigger components in this area per unit time, and it is a parameter reflecting the intensity of local error outbreaks. If the joint density is greater than a preset joint density threshold, it indicates that the area is currently in a high-intensity error outbreak state.

[0143] On this basis, further analyze the deployment relationship of these storm trigger components in the physical space to determine whether there is a direct or indirect association in the physical topology. For example, these components may be concentrated in the same cabinet, the same motherboard, or the same backplane connection link, and have a common power supply or signal path. The above topological relationship is judged by reading the component topology information, the hardware layout map, or the association table. If multiple storm trigger components not only have a joint error density exceeding the threshold but also have a structural association in the physical topology, the severity flag positions of all relevant storm trigger components are set. For example, the joint density of correctable errors and uncorrectable errors is greater than 5 times / second and meets the topological relevance. This embodiment realizes the effective perception and timely response to the outbreak situation of high-risk regional faults through the dual criteria of joint error density and topological relevance, and strengthens the active defense ability of fault handling in a complex operating environment.

[0144] Figure 3 Schematic diagram of the fault handling method provided by the embodiment of the present application Figure 2 As Figure 3 shown, it includes:

[0145] S31, trigger an interrupt when at least one hardware component fails, and obtain the fault information of at least one hardware component.

[0146] In this embodiment, when at least one hardware component fails, an interrupt is triggered. The interrupt mechanism is an asynchronous signal actively triggered by hardware, used to respond to critical events in a timely manner, especially in reliability, availability, and serviceability architectures. Through the interrupt-triggered method, the exception handling process can be started immediately when a hardware failure occurs, thereby improving the real-time performance and accuracy of fault response. After the interrupt is triggered, the parameter information corresponding to the fault event is collected. The fault information usually includes the timestamp of the fault occurrence, the fault type (such as correctable error, uncorrectable error), the hardware identifier of the fault source, the fault level, the sensor number, and the original data of the error event, etc. The obtained fault information will be used as the basic input for subsequent fault analysis and judgment processing, providing a data basis for further determining whether an error storm is formed. By adopting the interrupt-based fault trigger mechanism in this embodiment, the active reporting of hardware component fault information is realized, reducing the delay and resource waste problems existing in the traditional polling mechanism. The interrupt response can complete the perception of the fault state with extremely low delay and provide a data basis for the subsequent multi-level judgment mechanism based on historical information and system load status, thus ensuring that the overall system has an efficient and stable fault response ability.

[0147] S32. Send the fault information of at least one hardware component to the baseboard control manager.

[0148] So that the baseboard control manager receives the fault information of at least one hardware component, and based on the stored fault information, obtains the historical fault information of at least one hardware component; obtains the current system load information, and based on the historical fault information and the current system load information, obtains the current storm determination conditions corresponding to each hardware component; according to the current sliding time window, obtains the window cumulative fault information of at least one hardware component, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, sets the storm trigger flag bit of the hardware component and marks the hardware component as a storm trigger component; receives the fault information of at least one storm trigger component, and when the fault information of at least one storm trigger component meets any preset suppression trigger condition, sets the suppression flag bit of at least one storm trigger component.

[0149] In this embodiment, the fault information of at least one hardware component is sent to the baseboard control manager (Baseboard Management Controller, abbreviated as BMC). As a controller in the server platform used to monitor, manage, and control the underlying hardware status, the baseboard control manager has the ability to perform logical processing on the hardware status and can still operate normally when the operating system fails or the main processor is abnormal. By reporting the fault information to the baseboard control manager, it can ensure that the continuous tracking and analysis of the hardware component fault status can still be completed in the case of system-level anomalies.

[0150] After receiving the fault information of at least one hardware component, the substrate control manager determines whether the hardware component meets the storm trigger condition and the suppression trigger condition. The corresponding description has been made in the foregoing embodiments and will not be elaborated here one by one.

[0151] Figure 4 Schematic flow of the fault handling method provided by the embodiments of the present application Figure 3 . On the basis of the above embodiments, as Figure 4 shown, it further includes:

[0152] S41, monitor the storm flag bit in real time, and after reading that the storm flag bit of the storm trigger component is set, block the upward transmission of the fault information of the storm trigger component.

[0153] In this embodiment, by monitoring the status of the storm flag bit in real time, it continuously senses whether each hardware component is in the storm trigger state. The storm flag bit is a logical identification bit set for each hardware component, and is usually dynamically updated by the management control module (such as the substrate control manager) according to the fault statistics and determination logic. When it is detected that the fault behavior of a certain hardware component meets its storm determination condition, this flag bit is set to the effective state, indicating that this component is in the fault storm state. To avoid a large number of error messages continuously generated in the storm state from interfering with the system, after detecting that the storm flag bit is set, the fault information of this storm trigger component is immediately blocked from being transmitted upward. Specifically, the upward transmission path of the fault information will be logically interrupted or marked as ignored, and the blocking behavior can be achieved by modifying the RAS-related registers, etc., to ensure that the subsequent errors generated by this component are not uploaded to the upper layer within a certain period. Through the above blocking operation, a series of chain reactions such as load soar caused by error storms, redundant recording of fault events, and mis-triggering of error responses are effectively reduced, thereby improving the overall operation stability and the reliability of exception handling. At the same time, the blocking mechanism does not affect the BMC's continuous monitoring of the status of this component, and the fault information of this component can still be obtained in the management channel during the blocking period. This embodiment realizes the rapid suppression of fault storm behavior and interference isolation by reading the storm flag bit in real time and dynamically controlling the flow of error information.

[0154] S42, monitor the suppression flag bit in real time, and after reading that the suppression flag bit of the storm trigger component is set, execute the suppression strategy corresponding to the suppression flag bit.

[0155] In this embodiment, the inhibition flag bits of each hardware component are continuously monitored in real time. The inhibition flag bit is a status identifier set for the corresponding hardware component after determining the component's fault behavior and the preset inhibition trigger condition, and is used to indicate whether the component is currently in a state where inhibition measures need to be taken. During the monitoring process, if the inhibition flag bit of any hardware component is detected to be set, it indicates that the current fault behavior of the storm trigger component has met the risk conditions of the corresponding level, and the preset inhibition strategy corresponding to the level of this inhibition flag bit will be executed. Different levels of inhibition flag bits will trigger inhibition actions of different intensities and scopes to ensure the pertinence of the inhibition strategy. Through the real-time reading and response of the inhibition flag bits, this embodiment realizes rapid intervention in faults, improves the overall stability and the automation level of error handling.

[0156] In a specific embodiment, for the above step S42, a specific inhibition strategy is provided here. On the basis of the above embodiment, it includes:

[0157] S421, when the observation-level flag bit of the storm trigger component is read as set, record the fault information of the storm trigger component in the system event log and output a health report;

[0158] S422, when the warning-level flag bit of the storm trigger component is read as set, reduce the execution frequency of the subsystem corresponding to the storm trigger component and output an alarm notification carrying the information of the storm trigger component;

[0159] S423, when the isolation-level flag bit of the storm trigger component is read as set, isolate the storm trigger component; or,

[0160] S424, when the critical-level flag bit of the storm trigger component is read as set, perform a system-level operation mode degradation and output a health check instruction to the baseboard management controller.

[0161] In this embodiment, by real-time reading the flag bits of different levels of each storm trigger component, fine-grained perception and response control of the component fault status are realized. When the observation-level, warning-level, isolation-level, or critical-level flag bit of a certain storm trigger component is read as set, according to the preset strategy corresponding to this level, matching fault tolerance or inhibition operations are taken.

[0162] Specifically, when it is detected that the observation-level flag bit of the storm trigger component is set, it indicates that the component has had a certain frequency of correctable errors. At this time, the relevant fault information of the component (including error type, timestamp, component identifier, error density, etc.) will be recorded in the system event log for fault analysis and trend tracking. At the same time, a health report is generated for reference by the system administrator or the fault management service.

[0163] When the warning - level flag bit of the storm - trigger component is read as set, it indicates that the error behavior of this component has been significantly abnormal and may further deteriorate into an irreparable error or trigger a chain effect. At this time, a targeted processing mechanism will be triggered. For example, actively reduce the execution frequency of the subsystem to which the storm - trigger component belongs (such as the access rate of the memory subsystem, link bandwidth, etc.) to inhibit the further spread of the fault. In addition, an alarm notification carrying the component identifier, fault level, and recommended handling measures will be output to notify for manual intervention or remote diagnosis.

[0164] When the isolation - level flag bit of the storm - trigger component is read as set, it indicates that this component may no longer be able to maintain normal functions or is continuously generating serious errors. To reduce the spread of system - level impacts, a physical or logical isolation operation will be immediately performed on this storm - trigger component. Such operations may include disconnecting the component from the main data path, closing the corresponding device port, stopping service allocation, or dynamically redirecting tasks to redundant components, etc., to ensure fault isolation with minimal losses.

[0165] When the critical - level flag bit of the storm - trigger component is detected as set, it means that multiple components in the system are simultaneously in a high - density fault state and have characteristics of correlation or synchronous errors. At this time, the system - level operating mode will be degraded, such as switching to a safe operating mode, restricting the number of concurrent tasks, or pausing some services, to quickly stabilize the system operating environment. At the same time, a health check instruction will be output to the baseboard management controller, instructing the baseboard management controller to initiate a full - system - level component status inspection, temperature and power consumption verification, or remote diagnosis task, providing basic information for subsequent fault tracing and repair decision - making.

[0166] In this embodiment, through the hierarchical response to different - level flag bits, a multi - level and fine - grained error handling is constructed, realizing the full - process linkage control from primary perception to high - level intervention, effectively improving the controllability and stability in the face of complex fault scenarios.

[0167] Figure 5 This is a timing diagram of the fault - handling method provided by the embodiment of this application. As Figure 5 shown, it includes:

[0168] S51, The basic input - output system triggers an interrupt when at least one hardware component has a fault and obtains the fault information of at least one hardware component;

[0169] S52, The basic input - output system sends the fault information of at least one hardware component;

[0170] S53, The baseboard management controller receives the fault information of at least one hardware component and obtains the historical fault information of at least one hardware component according to the stored fault information;

[0171] S54, the baseboard management controller obtains the current system load information, and based on the historical fault information and the current system load information, obtains the current storm determination conditions corresponding to each hardware component;

[0172] S55, the baseboard management controller obtains the window cumulative fault information of at least one hardware component according to the current sliding time window, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, sets the storm trigger flag bit of the hardware component, and records the hardware component as a storm trigger component;

[0173] S56, the basic input / output system monitors the storm flag bit in real time, and after reading that the storm flag bit of the storm trigger component is set, shields the upward transmission of the fault information of the storm trigger component;

[0174] S57, the baseboard management controller receives the fault information of at least one storm trigger component, and when the fault information of at least one storm trigger component meets any preset suppression trigger condition, sets the suppression flag bit of at least one storm trigger component;

[0175] S58, the basic input / output system monitors the suppression flag bit in real time, and after reading that the suppression flag bit is set, executes the suppression strategy corresponding to the suppression flag bit.

[0176] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0177] Figure 6 It is a schematic structural diagram of the baseboard management controller provided by the embodiment of the present application. As Figure 6 shown, the baseboard management controller 6 includes:

[0178] A data receiving module 61, configured to receive the fault information of at least one hardware component sent by the basic input / output system, and obtain the historical fault information of at least one hardware component according to the stored fault information; wherein, the fault information is obtained by the basic input / output system when a hardware component fails to trigger an interrupt;

[0179] A condition generating module 62, configured to obtain the current system load information, and based on the historical fault information and the current system load information, obtain the current storm determination conditions corresponding to each hardware component;

[0180] A storm determination module 63, configured to obtain window cumulative fault information of at least one hardware component according to a current sliding time window, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, set a storm trigger flag bit of the hardware component and mark the hardware component as a storm trigger component;

[0181] A suppression monitoring module 64, configured to receive fault information of at least one storm trigger component, and when the fault information of at least one storm trigger component meets any preset suppression trigger condition, set a suppression flag bit of at least one storm trigger component.

[0182] For the description of the features in the corresponding embodiments of the baseboard management controller, reference may be made to the relevant descriptions in the corresponding embodiments of the above-mentioned fault handling method, which will not be elaborated here one by one.

[0183] Figure 7 This is a schematic structural diagram of the basic input / output system provided by the embodiments of the present application. As Figure 6 shown, the basic input / output system 7 includes:

[0184] An information acquisition module 71, configured to trigger an interrupt when at least one hardware component fails and obtain fault information of at least one hardware component;

[0185] An information sending module 72, configured to send fault information of at least one hardware component to a baseboard control manager, so that the baseboard control manager receives the fault information of at least one hardware component, and according to the stored fault information, obtain historical fault information of at least one hardware component; obtain current system load information, and according to the historical fault information and the current system load information, obtain current storm determination conditions corresponding to each hardware component; according to the current sliding time window, obtain window cumulative fault information of at least one hardware component, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, set a storm trigger flag bit of the hardware component and mark the hardware component as a storm trigger component; receive fault information of at least one storm trigger component, and when the fault information of at least one storm trigger component meets any preset suppression trigger condition, set a suppression flag bit of at least one storm trigger component.

[0186] For the description of the features in the corresponding embodiments of the basic input / output system, reference may be made to the relevant descriptions in the corresponding embodiments of the above-mentioned fault handling method, which will not be elaborated here one by one.

[0187] The present application further provides a fault handling system, including: a baseboard management controller, a basic input / output system, and a processor;

[0188] The baseboard management controller includes a microprocessor and a management control memory. The management control memory is used to store a management control program, and the microprocessor is used to execute the management control program stored in the management control memory to perform the steps according to the corresponding fault handling method described above.

[0189] The basic input / output system is used to store a basic input / output program.

[0190] The processor is used to execute the basic input / output program stored in the basic input / output system to perform the steps according to the corresponding fault handling method described above.

[0191] Figure 8 This is a schematic structural diagram of the electronic device provided by this application. As Figure 8 shown, the electronic device 8 provided in this embodiment includes: at least one processor 81 and a memory 82. Optionally, the electronic device 8 further includes a communication component 83. Among them, the processor 81, the memory 82, and the communication component 83 are connected through a bus 84.

[0192] In the specific implementation process, at least one processor 81 executes the computer-executable instructions stored in the memory 82, so that at least one processor 81 executes the above-mentioned embodiment of the fault handling method.

[0193] For the specific implementation process of the processor 81, reference can be made to the above-mentioned method embodiment. The implementation principle and technical effect are similar, and will not be elaborated here in this embodiment.

[0194] In the above embodiment, it should be understood that the processor may be a central processing unit (Central Processing Unit, abbreviated as: CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, abbreviated as: DSP), application specific integrated circuits (Application Specific Integrated Circuit, abbreviated as: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0195] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0196] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0197] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program is configured to execute the steps in any of the above-mentioned embodiments of the fault handling method when running.

[0198] In an exemplary embodiment, the above-mentioned computer-readable storage medium may include but is not limited to: various media that can store computer programs such as a USB flash drive, a read-only memory (ROM for short), a random access memory (RAM for short), a mobile hard disk, a magnetic disk, or an optical disc.

[0199] An embodiment of the present application also provides a computer program product. The above-mentioned computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the fault handling method are implemented.

[0200] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the fault handling method are implemented.

[0201] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0202] The above has introduced in detail a fault handling method provided by this application. Specific examples are used in this text to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art of this technology, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A fault handling method, characterized in that, Including: Receiving fault information of at least one hardware component sent by the basic input / output system, and obtaining historical fault information of at least one hardware component according to the stored fault information; Wherein, the fault information is obtained by the basic input / output system when a fault occurs in the hardware component and triggers an interrupt; Obtaining current system load information, and obtaining current storm determination conditions corresponding to each of the hardware components according to the historical fault information and the current system load information; According to the current sliding time window, obtaining window cumulative fault information of at least one of the hardware components, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, setting the storm trigger flag bit of the hardware component, and recording the hardware component as a storm trigger component; Receiving fault information of at least one of the storm trigger components, and when the fault information of at least one storm trigger component meets any preset suppression trigger condition, setting the suppression flag bit of the at least one storm trigger component.

2. The fault handling method according to claim 1, wherein, The determination of the current sliding time window includes: Obtaining the time slot width corresponding to the previous sliding time window, and obtaining the window cumulative fault information corresponding to the previous sliding time window; Calculating and obtaining the error density corresponding to the previous sliding time window according to the window cumulative fault information; Obtaining the time slot width corresponding to the current sliding time window according to the error density, the time slot width, and a preset time slot width sequence; Based on the time slot width corresponding to the current sliding time window, determining the start time and end time of the current sliding time window, and completing the determination of the current sliding time window.

3. The fault handling method according to claim 2, characterized in that, The obtaining the time slot width corresponding to the current sliding time window according to the error density, the time slot width, and a preset time slot width sequence includes: Obtaining a preset time slot width sequence; wherein the time slot width sequence includes a plurality of slot width granularities arranged in descending order; If the error density is greater than or equal to a preset first density threshold, and the time slot width is not the smallest slot width granularity in the time slot width sequence, then selecting the next slot width granularity corresponding to the time slot width from the time slot width sequence, and recording the next slot width granularity as the time slot width corresponding to the current sliding time window; or, If the error density is greater than a preset second density threshold and less than a preset first density threshold, then setting the time slot width as the time slot width corresponding to the current sliding time window; or, If the error density is less than or equal to a preset second density threshold, and the time slot width is not the largest slot width granularity in the time slot width sequence, then selecting the previous slot width granularity corresponding to the time slot width from the time slot width sequence, and recording the previous slot width granularity as the time slot width corresponding to the current sliding time window.

4. The fault handling method according to claim 1, characterized in that The receiving fault information of at least one hardware component sent by the basic input / output system, and obtaining historical fault information of at least one hardware component according to the stored fault information includes: Receiving fault metadata of at least one of the hardware components sent by the basic input / output system; Receive and parse the fault metadata, and store the parsed fault metadata in the system event log; wherein, the system event log stores historical fault information corresponding to at least one piece of fault metadata; Based on the historical fault information stored in the system event log, map the historical fault information to a preset fault monitoring buffer according to the preset time length of the sliding time window; According to the start time and end time of the current sliding time window, screen and extract the historical fault data of each hardware component from the preset fault monitoring buffer; Wherein, the fault monitoring buffer is indexed by time stamps, and records the fault events occurring at different time points of each hardware component and the fault types corresponding to the fault events.

5. The fault handling method according to claim 1, characterized in that, The obtaining of the current storm determination conditions corresponding to each hardware component according to the historical fault information and the current system load information includes: Collect and obtain the current system load information, and normalize the current system load information to obtain the normalized value Load corresponding to the current load information; According to the current sliding event window, obtain the historical correctable error density of the hardware component; wherein, the historical correctable error density is the correctable error density of the previous sliding event window; and / or, obtain the historical uncorrectable error density of the hardware component; wherein, the historical uncorrectable error density is the uncorrectable error density of the previous sliding event window; Adopt the formula: Calculate to obtain the first trigger density Threshold1; and according to the first trigger density and the preset time window length, obtain the correctable error storm trigger value of the hardware component; wherein, Base1 is a preset correctable error base threshold, α1 is a preset first load adjustment coefficient, β1 is a preset first error density adjustment coefficient, D1 is the historical correctable error density, and k1 is a preset first historical influence sensitivity; and / or Calculate to obtain the second trigger density Threshold2; and according to the second trigger density and the preset time window length, obtain the uncorrectable error storm trigger value of the hardware component; wherein, Base2 is a preset uncorrectable error base threshold, α2 is a preset second load adjustment coefficient, β2 is a preset second error density adjustment coefficient, D2 is the historical correctable error density, and k2 is a preset second historical influence sensitivity.

6. The fault handling method according to claim 1, characterized in that The obtaining of the window cumulative fault information of at least one hardware component according to the current sliding time window includes: Obtain the start time and end time of the current sliding time window, and the time slot width corresponding to the current sliding time window; wherein, the time slot width is used to divide the current sliding time window into multiple continuous and non-overlapping time slots; According to the start time and end time of the current sliding time window, extract the historical fault information of one or more hardware components from the preset fault monitoring buffer; According to the time stamps corresponding to the historical fault information, map the historical fault information to each time slot within the current sliding time window, and map the collected real-time fault information to the corresponding time slots; Classify and count the fault information of each of the hardware components according to the fault type based on the historical fault information, and obtain the cumulative number of correctable errors and / or the cumulative number of uncorrectable errors of at least one of the hardware components within the current sliding time window.

7. The fault handling method according to claim 6, wherein The window cumulative fault information of the hardware component satisfies its corresponding current storm determination condition, including: According to the window cumulative fault information of the hardware component, obtain the cumulative number of correctable errors corresponding to the hardware component, and obtain the correctable error storm trigger value corresponding to the hardware component; If the cumulative number of correctable errors is greater than or equal to the correctable error storm trigger value, determine that the window cumulative fault information of the hardware component satisfies its corresponding current storm determination condition; or, Obtain the cumulative number of uncorrectable errors corresponding to the hardware component, and obtain the uncorrectable error storm trigger value corresponding to the hardware component; If the cumulative number of uncorrectable errors is greater than or equal to the uncorrectable error storm trigger value, determine that the window cumulative fault information of the hardware component satisfies its corresponding current storm determination condition.

8. The fault handling method according to claim 1, further comprising: If within a preset sliding waiting time window, the cumulative number of correctable errors of the storm trigger component is less than or equal to a preset number threshold, cancel the setting of the storm trigger flag bit of the storm trigger component.

9. The method according to claim 1, wherein The setting the suppression flag bit of the at least one storm trigger component when the fault information of the at least one storm trigger component satisfies any preset suppression trigger condition includes: Within a first preset time range, if the cumulative number of correctable errors of the storm trigger component is greater than or equal to a first threshold and the corresponding correctable error density is less than a first density threshold, set the observation-level flag bit of the storm trigger component; or, Within a second preset time range, if the cumulative number of correctable errors of the storm trigger component is greater than or equal to a second threshold, set the warning-level flag bit of the storm trigger component; or, When receiving the uncorrectable error information of the storm trigger component, set the warning-level flag bit of the storm trigger component; or, Within a third preset time range, if the cumulative number of uncorrectable errors of the storm trigger component is greater than or equal to a third threshold, set the isolation-level flag bit of the storm trigger component; or, Within a fourth preset time range, record the number of storm trigger components whose received number of correctable errors is greater than or equal to a fourth threshold. If the number is greater than or equal to a first quantity threshold, set the isolation-level flag bits corresponding to one or more of the storm trigger components; or, Within a fifth preset time range, record the number of storm trigger components that receive uncorrectable errors. If the number is greater than or equal to a second quantity threshold, set the severe-level flag bits of multiple of the storm trigger components; If the combined density of correctable errors and uncorrectable errors of multiple storm trigger components within a unit time is greater than a preset combined density threshold, and there is an associated relationship in the physical topology structure of the multiple storm trigger components, then set the severity flag bits corresponding to the multiple storm trigger components.

10. A fault handling method, characterized in that, Including: Trigger an interrupt when at least one hardware component fails, and obtain the fault information of the at least one hardware component; Send the fault information of the at least one hardware component to the baseboard control manager, so that the baseboard control manager receives the fault information of the at least one hardware component, and according to the stored fault information, obtain the historical fault information of the at least one hardware component; obtain the current system load information, and according to the historical fault information and the current system load information, obtain the current storm determination conditions corresponding to each of the hardware components; according to the current sliding time window, obtain the window cumulative fault information of at least one of the hardware components, and when the window cumulative fault information of the hardware component meets its corresponding current storm determination condition, set the storm trigger flag bit of the hardware component, and record the hardware component as a storm trigger component; Receive the fault information of at least one of the storm trigger components, and when the fault information of at least one storm trigger component meets any preset suppression trigger condition, set the suppression flag bit of the at least one storm trigger component.

11. The fault handling method according to claim 10, characterized in that, Further including: Monitor the storm flag bit in real time, and after reading that the storm flag bit of the storm trigger component is set, block the upward transmission of the fault information of the storm trigger component; And Monitor the suppression flag bit in real time, and after reading that the suppression flag bit is set, execute the suppression policy corresponding to the suppression flag bit.

12. The fault handling method according to claim 11, wherein The monitoring the suppression flag bit in real time, and after reading that the suppression flag bit is set, execute the suppression policy corresponding to the suppression flag bit includes: When reading that the observation-level flag bit of the storm trigger component is set, record the fault information of the storm trigger component in the system event log and output a health report; or, When reading that the warning-level flag bit of the storm trigger component is set, reduce the execution frequency of the subsystem corresponding to the storm trigger component and output an alarm notification carrying the information of the storm trigger component; or, When reading that the isolation-level flag bit of the storm trigger component is set, isolate the storm trigger component; or, When reading that the severity-level flag bit of the storm trigger component is set, perform a system-level operating mode downgrade and output a health check instruction to the baseboard management controller.

13. A fault handling method, characterized in that, Including: The basic input / output system triggers an interrupt when at least one hardware component fails, and obtains the fault information of the at least one hardware component; The basic input / output system sends the fault information of at least one hardware component; The baseboard management controller receives the fault information of at least one hardware component, and according to the stored fault information, obtains the historical fault information of at least one hardware component; The baseboard management controller obtains the current system load information, and according to the historical fault information and the current system load information, obtains the current storm determination conditions corresponding to each of the hardware components; The baseboard management controller obtains the window cumulative failure information of at least one of the hardware components according to the current sliding time window, and when the window cumulative failure information of the hardware component meets its corresponding current storm determination condition, sets the storm trigger flag bit of the hardware component and marks the hardware component as a storm trigger component; The basic input / output system monitors the storm flag bit in real time, and after reading that the storm flag bit of the storm trigger component is set, shields the upward transmission of the failure information of the storm trigger component; The baseboard management controller receives the failure information of at least one of the storm trigger components, and when the failure information of at least one storm trigger component meets any preset suppression trigger condition, sets the suppression flag bit of the at least one storm trigger component; The basic input / output system monitors the suppression flag bit in real time, and after reading that the suppression flag bit is set, executes the suppression strategy corresponding to the suppression flag bit.

14. A fault handling system, characterized in that, Including: A baseboard management controller, a basic input / output system, and a processor; The baseboard management controller includes a microprocessor and a management control memory, the management control memory is used to store a management control program, and the microprocessor is used to execute the management control program stored in the management control memory to perform the steps of the fault handling method according to any one of claims 1 to 9; The basic input / output system is used to store a basic input / output program; The processor is used to execute the basic input / output program stored in the basic input / output system to perform the steps of the fault handling method according to any one of claims 10 - 12.

15. A computer-readable storage medium stores a computer program therein, where When the computer program is executed by a processor, it implements the steps of the fault handling method according to any one of claims 1 to 9 or claims 10 - 12.

Images