Early warning method, device, medium and system for server hardware fault

Abstract

The invention provides a server hardware fault early warning method, device, medium and system. The method comprises the following steps: acquiring operation data of a server; processing the operation data by adopting an analysis engine to obtain a rule matching result; under the condition that the rule matching result represents that matching is successful, first-level early warning information is generated to prompt that server hardware has a fault risk; under the condition that the rule matching result represents that matching fails, the first neural network model is adopted to process the hardware sensor data so as to obtain a result whether early warning information is generated or not, and the early warning information represents that the hardware sensor data is abnormal. The server hardware fault early warning method and device solve the problem that the existing server hardware fault early warning technology is difficult to identify novel hidden hardware faults in time and accurately position specific hardware components, so that the risk of pre-business interruption is large.

Description

Technical Field

[0001] This application relates to the field of server operation and maintenance management technology, and in particular to a method for early warning of server hardware failure, a device for early warning of server hardware failure, a computer-readable storage medium, and a system for early warning of server hardware failure. Background Technology

[0002] As server clusters continue to expand, server hardware inevitably experiences performance degradation or potential malfunctions during long-term operation, such as abnormal changes in critical components like hard drives, memory, and power supplies. Failure to effectively identify and address these malfunctions before they occur can easily lead to server downtime, thereby impacting business continuity.

[0003] Existing server hardware fault early warning technologies are unable to identify latent faults and new fault modes that gradually develop during hardware operation in a timely manner. Furthermore, they are difficult to accurately locate specific hardware components before a fault occurs, thereby increasing the difficulty of fault handling and the risk of business interruption. Summary of the Invention

[0004] This application provides a method, device, computer-readable storage medium, and system for early warning of server hardware failures, in order to at least solve the problem that existing server hardware failure early warning technologies are unable to promptly identify new hidden hardware failures and accurately locate specific hardware components, leading to a high risk of pre-service interruption.

[0005] This application provides a method for early warning of server hardware failures, comprising: acquiring server operating data, including hardware sensor data, hardware performance counter data, and system log features, wherein the system log features are the characteristics of log information generated during server operation, the hardware sensor data includes the vibration amplitude of the hard disk, and the hardware performance counter data includes memory bandwidth utilization; processing the operating data using an analysis engine to obtain rule matching results, wherein the rule matching results indicate whether the operating data matches the failure feature rules in a preset database, and the failure feature rules indicate the correlation between the operating data and server hardware failures; generating a first-level early warning message when the rule matching results indicate a successful match, to indicate that there is a risk of server hardware failure; and processing the hardware sensor data using a first neural network model when the rule matching results indicate a failure, wherein the first neural network model is a model trained based on historical hardware sensor data from historical server hardware failure samples, and the early warning message indicates abnormal hardware sensor data.

[0006] This application also provides a server hardware failure early warning device, comprising: an acquisition unit for acquiring server operating data, the operating data including hardware sensor data, hardware performance counter data, and system log features, wherein the system log features are the features of log information generated during server operation, the hardware sensor data includes the vibration amplitude of the hard disk, and the hardware performance counter data includes memory bandwidth utilization; a first processing unit for processing the operating data using an analysis engine to obtain rule matching results, wherein the rule matching results indicate whether the operating data matches the failure feature rules in a preset database, and the failure feature rules indicate the correlation between the operating data and server hardware failure; a generation unit for generating first-level early warning information when the rule matching results indicate a successful match, to indicate that there is a risk of server hardware failure; and a second processing unit for processing the hardware sensor data using a first neural network model when the rule matching results indicate a failed match, to obtain a result indicating whether to generate early warning information, wherein the first neural network model is a model trained based on historical hardware sensor data from historical server hardware failure samples, and the early warning information indicates abnormal hardware sensor data.

[0007] This application also provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the above-described methods.

[0008] This application also provides a server hardware failure early warning system, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for performing any one of them.

[0009] This application utilizes server hardware sensor data, hardware performance counter data, and system log features. By combining rule matching with a first neural network model to analyze the operational data, this solution can further uncover abnormal changes in hardware sensor data when existing fault feature rules cannot cover the full picture. This enables the early identification of novel, latent hardware faults. Compared to early warning methods that rely solely on rules or single monitoring data, this solution improves the timeliness of server hardware fault warnings while reducing the risk of business interruption due to unidentified latent hardware faults. It provides a more reliable early warning basis for subsequent fault location and handling. This solves the problem in existing server hardware fault warning technologies that struggle to promptly identify novel, latent hardware faults and accurately locate specific hardware components, leading to a high risk of pre-business interruption. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a method for early warning of server hardware failures according to an embodiment of this application is shown.

[0012] Figure 2 A flowchart illustrating another method for early warning of server hardware failures according to an embodiment of this application is shown.

[0013] Figure 3 A schematic diagram of the structure of a server hardware fault early warning system according to an embodiment of this application is shown;

[0014] Figure 4 A structural block diagram of a server hardware failure early warning device provided according to an embodiment of this application is shown. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0016] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0017] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] The embodiments of this application provide a method for early warning of server hardware failures. The method is described in detail below in conjunction with the execution flow of the method for early warning of server hardware failures.

[0019] Figure 1This is a flowchart of a server hardware fault early warning method according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0020] Step S101: Obtain server operating data, which includes hardware sensor data, hardware performance counter data, and system log features. System log features are the characteristics of log information generated during server operation. Hardware sensor data includes the vibration amplitude of the hard drive, and hardware performance counter data includes memory bandwidth utilization.

[0021] Specifically, hardware sensor data characterizes the physical operating status of server hardware and can be collected through the server's management and control unit or sensors deployed on the hardware. Hardware sensor data includes: hard drive vibration amplitude, reflecting the mechanical vibration experienced by the hard drive during operation; processor temperature, reflecting changes in the processor's thermal state during operation; and power supply ripple, reflecting changes in the stability of the power output during server power supply. Hardware performance counter data characterizes the performance status and load of server hardware during operation and can be obtained through performance probes or system performance counter interfaces. Hardware performance counter data includes: memory bandwidth utilization, reflecting the usage of server memory resources during operation; and PCIe (Peripheral Component Interconnect Express) bit error rate, reflecting the stability and reliability of the server's internal high-speed bus during data transmission.

[0022] Step S102: The analysis engine is used to process the running data to obtain the rule matching result. The rule matching result indicates whether the running data matches the fault feature rules in the preset database. The fault feature rules indicate the correlation between the running data and the server hardware fault.

[0023] Specifically, the rule matching result indicates whether the running data is consistent with the fault feature rules in the preset database. The fault feature rules represent the correlation between running data and server hardware faults. For example, they are used to describe the judgment relationship of the corresponding hardware fault risk when log characteristics, performance counter changes, or sensor indicators meet specific conditions within a specific time window.

[0024] Step S103: If the rule matching result indicates a successful match, generate a first-level warning message to indicate that there is a risk of server hardware failure.

[0025] Specifically, since the fault feature rules are derived from known fault modes in a preset database, a successful match means that the current running data matches the known fault risk features. Therefore, it can quickly trigger an early warning with low computational overhead, thereby enabling timely alerts for known types of hardware fault risks.

[0026] In step S104, if the rule matching result indicates that the matching has failed, the first neural network model is used to process the hardware sensor data to obtain the result of whether to generate a warning message. The first neural network model is a model trained based on historical hardware sensor data in historical server hardware failure samples. The warning message indicates that the hardware sensor data is abnormal.

[0027] Specifically, since the fault feature rules in the preset database may not cover the precursor features of new or latent hardware faults, when a rule is not matched, the first neural network model is further invoked to perform time-series analysis or anomaly detection on the hardware sensor data to identify abnormal change patterns that are difficult for the rules to cover. When the model analysis results meet preset conditions, a warning message is generated to indicate that there is an anomaly in the hardware sensor data, thereby achieving supplementary identification of potential hardware fault risks. The first neural network model is trained based on historical hardware sensor data from historical server hardware fault samples, and therefore can extract fault-related feature patterns from the time-series change trends of hardware sensor data to identify abnormal situations that are difficult for the rule base to cover. By introducing the first neural network model, automatic identification of hardware sensor data anomalies is achieved, thereby improving the timeliness and accuracy of server hardware fault warnings and reducing the risk of missed reports due to missing rules.

[0028] Through the above steps, by acquiring server hardware sensor data, hardware performance counter data, and system log features, and analyzing the operational data using a combination of rule matching and a first neural network model, this solution can further uncover abnormal changes in hardware sensor data when existing fault feature rules cannot cover them, thereby achieving early identification of new hidden hardware faults. Compared to early warning methods that rely solely on rules or single monitoring data, this solution improves the timeliness of server hardware fault early warning while reducing the risk of business interruption due to unidentified hidden hardware faults, providing a more reliable early warning basis for subsequent fault location and handling. It solves the problem in existing server hardware fault early warning technologies that struggle to promptly identify new hidden hardware faults and accurately locate specific hardware components, leading to a high risk of pre-business interruption.

[0029] In one optional scheme, a first neural network model is used to process the hardware sensor data to determine whether to generate an early warning message. This includes: processing the hardware sensor data using the first neural network model to obtain an anomaly confidence level of the hardware sensor data, whereby the anomaly confidence level characterizes the degree of credibility of the hardware sensor data anomaly; generating a second-level early warning message characterizing the hardware sensor data anomaly when the anomaly confidence level is greater than or equal to a preset confidence level, wherein the handling priority of the second-level early warning message is higher than that of the first-level early warning message; and determining that no second-level early warning message needs to be generated when the anomaly confidence level is less than the preset confidence level.

[0030] In the above embodiments, by using a first neural network model to analyze hardware sensor data and output anomaly confidence levels, this solution can quantify and determine potential abnormal changes in hardware sensor data even when rules are not matched, thereby identifying new or hidden hardware fault risks that are difficult to cover by traditional rules. Furthermore, by comparing the anomaly confidence level with a preset confidence level, a second-level warning message with higher priority is generated only when the confidence conditions are met, and no second-level warning message is generated when the confidence conditions are not met. This improves the timeliness and sensitivity of warnings while reducing the probability of false alarms and avoiding unnecessary high-priority actions triggered by occasional fluctuations.

[0031] Specifically, when the anomaly confidence level is greater than or equal to a preset confidence level, it indicates that there are significant anomalous features in the hardware sensor data, and these anomalous features reach a level of confidence sufficient to trigger an early warning. In this case, a second-level early warning message characterizing the hardware sensor data anomaly is generated. Compared to the first-level early warning message generated based on rule matching, the second-level early warning message has a higher priority for handling, so as to trigger more proactive response measures in a timely manner in the event of potential hidden hardware failure risks. When the anomaly confidence level is less than the preset confidence level, it indicates that the anomalous features in the current hardware sensor data are insufficient to support the generation of an early warning message. In this case, it is determined that there is no need to generate a second-level early warning message, thereby avoiding unnecessary early warning triggering due to occasional fluctuations or noisy data.

[0032] In another alternative approach, the method further includes: upon generating first-level and / or second-level warning information, acquiring fault characteristics of operational data, including time-series characteristics of sensor data; processing the fault characteristics using a second neural network model to determine target hardware, where the target hardware represents server hardware at risk of failure, and the second neural network model is a model trained based on historical fault characteristic data; determining the physical location information of the target hardware according to a preset server hardware topology, where the preset server hardware topology represents the connection relationships of each server hardware; determining the processing strategy for the target hardware based on the physical location information and the level corresponding to the warning information, and generating prompt information including the physical location information of the target hardware and the processing strategy to prompt the execution of the processing strategy for the target hardware.

[0033] Specifically, the generation of Level 1 and / or Level 2 warning messages indicates that the server's operational status has exhibited abnormal risks. At this point, fault characteristics are further extracted from the operational data for subsequent fault location and handling analysis. Fault characteristics include time-series characteristics of hardware sensor data, log characteristics corresponding to system log features, and / or performance inflection point characteristics corresponding to hardware performance counter data. Time-series characteristics reflect the changing trends and temporal relationships of hardware sensor data within a preset time range, thus obtaining the characteristics of hardware state evolution over time. Subsequently, a second neural network model is used to process the fault characteristics to analyze the correlation between the fault characteristics and various hardware components in the server, thereby identifying the target hardware. The target hardware is the server hardware component that is determined in the analysis results to have a high correlation with the fault characteristics and poses a fault risk, thus achieving the location of potential fault sources. After identifying the target hardware, the physical location information of the target hardware is determined according to the preset server hardware topology. Server hardware topology is used to characterize the connections between various hardware components within a server. By mapping target hardware to the topology, its corresponding physical location can be determined, such as its node, chassis, or connection channel, thus providing a location basis for subsequent maintenance operations. After obtaining the physical location information of the target hardware, a handling strategy is determined based on the level of the warning information. The handling strategy indicates the appropriate actions to be taken for the target hardware, and generates prompts based on the physical location information and the handling strategy to prompt maintenance personnel or automated systems to perform corresponding processing operations on the target hardware.

[0034] In the above embodiments, after generating first-level and / or second-level warning information, fault characteristics in the operational data are further obtained, and the fault characteristics are analyzed using a second neural network model. The second neural network model can learn the correlation between fault characteristics and server hardware from historical fault characteristic data, thereby inferring the correlation between fault characteristics and outputting the target hardware at risk of failure. This reduces the risk of mislocation due to human experience judgment and enables precise location of the target hardware at risk of failure based on warning triggering. Simultaneously, by combining the preset server hardware topology to determine the physical location information of the target hardware, the fault location result can be directly mapped to the specific hardware location, thereby reducing the complexity of operation and maintenance troubleshooting. Furthermore, by comprehensively considering the physical location information of the target hardware and the level corresponding to the warning information to determine the processing strategy and generate prompt information, the orderly execution of subsequent handling operations can be guided, improving the targeting and efficiency of server hardware fault response and further reducing the risk of hardware faults evolving into business interruptions.

[0035] In some exemplary embodiments, determining the processing strategy for the target hardware based on the physical location information and the level corresponding to the warning information includes: determining the fault propagation path of the target hardware based on the physical location information of the target hardware, wherein the fault propagation path characterizes the propagation relationship of the fault between server hardware when the target hardware fails; determining the fault risk level of the target hardware based on the fault propagation path; and determining the processing strategy for the target hardware based on the fault risk level.

[0036] In the above embodiments, by determining the fault propagation path based on the physical location information of the target hardware, and further assessing the fault risk level of the target hardware accordingly, this solution can quantitatively analyze the potential impact range and severity of hardware faults after the warning is triggered. Furthermore, it determines a matching processing strategy based on the fault risk level, so that the processing decision no longer depends solely on the warning trigger itself, but comprehensively considers the fault propagation relationship and the degree of potential impact, thereby achieving differentiated handling of different risk scenarios, improving the pertinence and rationality of the processing strategy, and reducing the adverse impact of fault propagation on other hardware and business operations.

[0037] Specifically, physical location information is used to indicate the exact location of the target hardware within the server and its connections to other hardware components. By combining this with a pre-defined server hardware topology, it's possible to infer the path the fault might propagate along hardware connections or dependencies to other hardware components when the target hardware fails, thus characterizing the potential impact range of the target hardware. After determining the fault propagation path, the failure risk of the target hardware is assessed based on this path to determine its failure risk level. The failure risk level characterizes the severity of the potential impact of a target hardware failure. Its assessment comprehensively considers factors such as the number of hardware components involved in the fault propagation path, their criticality, and the propagation hierarchy, thereby distinguishing different levels of failure risk. After obtaining the failure risk level of the target hardware, a handling strategy is determined based on this level. The handling strategy indicates the appropriate actions to be taken regarding the target hardware.

[0038] In other exemplary embodiments, after generating prompts including the physical location information of the target hardware and the processing strategy, the method further includes: processing the fault features using a self-learning engine to obtain fault feature rules, and storing the fault feature rules in a preset database.

[0039] In the above embodiments, after generating a prompt message containing the physical location information of the target hardware and the processing strategy, a self-learning engine is introduced to process the fault features corresponding to this warning and generate fault feature rules. The fault feature rules are then stored in a preset database. This allows the effective features extracted during the warning process to be precipitated into reusable rule knowledge, thereby continuously expanding and updating the fault feature rule library. This enables the analysis engine to use the updated fault feature rules to perform faster rule matching and warning triggering for similar fault patterns in subsequent operations, improving the coverage of new or hidden hardware faults and the timeliness of warnings, while reducing the reliance on manual experience to maintain the rules.

[0040] Specifically, the self-learning engine is a processing module used to achieve adaptive updates of fault feature rules. It is configured to summarize and analyze the fault features corresponding to the current warning after the warning is triggered and the target hardware is located and the handling strategy is prompted, so as to form reusable fault feature rules and write them into a preset database, thereby continuously improving the fault feature rule library.

[0041] In some exemplary embodiments of this application, obtaining server operating data includes: obtaining system logs and extracting preset key fields from the system logs; generating system log features based on the preset key fields.

[0042] In the above embodiments, by acquiring system logs and extracting preset key fields, and then generating system log features based on the preset key fields, unstructured log information can be transformed into structured feature data that can be used for subsequent analysis, thereby improving the parsability and usability of log information. Furthermore, the generated system log features can participate in fault feature rule matching and early warning determination as an important component of the running data, improving the efficiency of identifying hardware event-related anomalies, reducing the reliance on manual log review, and facilitating the automation and timeliness of server hardware fault early warning.

[0043] Specifically, preset key fields are used to extract key information related to the server hardware status from the system logs. These preset key fields include hardware event keywords, event type fields, event object fields, and / or severity level fields. The preset key fields can be set based on historical fault logs and preset fault characteristic rules, or configured according to system log format conventions, and stored in a preset database or configuration file for generating system log characteristics.

[0044] In some further exemplary embodiments of this application, the method further includes: in the event of a server failure, acquiring operational data within a preset time period prior to the server failure; processing the operational data using a self-learning engine to obtain failure features of the operational data; processing the failure features using a third neural network model to obtain multiple correlation values ​​between the failure features and each server hardware and a confidence value corresponding to each correlation value, wherein the third neural network model is a model trained based on the correspondence between historical failure features and server hardware; determining failure features whose correlation values ​​are greater than a preset correlation value and whose corresponding confidence values ​​are greater than a preset confidence value, and determining the operational data corresponding to the failure features, and generating failure feature rules based on the failure features and the operational data corresponding to the failure features; and storing the failure feature rules in a preset database.

[0045] In the above embodiments, after a server failure, operational data within a preset time period prior to the failure is retrieved. A self-learning engine is used to analyze the operational data to extract failure features. Then, a third neural network model is used to quantify the correlation and corresponding confidence levels between the failure features and each server hardware component. The third neural network model is used to infer the correlation between the failure features and server hardware based on the correspondence between historical failure features and server hardware, outputting the correlation and confidence values ​​for each server hardware component to generate the strength and reliability of the correlation between the failure features and each server hardware component. This solution can filter out features highly correlated with hardware failures and with high reliability from real failure samples. Furthermore, the highly correlated and high-confidence failure features and their corresponding operational data are abstracted into failure feature rules and stored in a preset database. This allows the system to automatically precipitate the experience of past failures into rule knowledge for rule matching and judgment in subsequent early warning stages, thereby continuously improving the ability to identify new or latent hardware failures, enhancing the completeness and adaptability of early warning rules, and reducing reliance on manual rule maintenance.

[0046] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the server hardware failure early warning method of this application will be described in detail below with reference to specific embodiments.

[0047] This embodiment relates to a specific method for early warning of server hardware failures, such as... Figure 2 As shown, it includes the following steps:

[0048] Step S1: Collect runtime data;

[0049] The server's operational data is acquired through a multi-source monitoring module. This data includes system logs, hardware sensor data, and hardware performance counter data. Specifically, the log acquisition unit collects system log information in real time, including Syslog (System Log Protocol) and system management and health monitoring information recorded and collected via IPMI (Intelligent Platform Management Interface). Natural language processing technology is used to extract hardware event keywords from the system logs. The sensor acquisition unit acquires physical status data of the server hardware through the server's management control unit (BMC) according to preset acquisition frequency and accuracy requirements. The performance probe unit monitors hardware performance counters and acquires performance data showing hardware performance changes over time.

[0050] Step S2: Perform rule matching on the running data;

[0051] The acquired operational data is input into the intelligent analysis engine, which then calls fault characteristic rules from the dynamic knowledge base to perform rule matching processing on the operational data. When the operational data successfully matches a preset fault characteristic rule, a first-level warning message is generated to indicate a potential risk of server hardware failure.

[0052] Step S3: Advanced early warning determination based on neural network model;

[0053] When the rule matching result in step S2 indicates a failure to match, the deep learning model analysis process is triggered, and a Temporal Convolutional Network (TCN) model is used to analyze the changing trends of the hardware sensor data. The TCN model infers the temporal characteristics of the hardware sensor data to obtain an anomaly confidence level indicating hardware failure. When the anomaly confidence level exceeds a preset confidence level, a second-level warning message is generated to indicate the presence of a hidden hardware failure risk. When the rule matching result in step S2 indicates a successful match, a first-level warning message is generated.

[0054] Step S4: Analysis of the relationship between root cause location and propagation of the fault;

[0055] After generating Level 1 and / or Level 2 warning information, the root cause localization module is invoked. Based on the extracted fault characteristics and the fault propagation graph in the dynamic knowledge base, the fault correlation between server hardware components is analyzed. A hardware component fault dependency model constructed using a graph neural network is used to determine the target hardware corresponding to the fault source and locate the physical location of the target hardware within the server.

[0056] Step S5: Fault Impact Analysis;

[0057] After identifying the target hardware, generate fault impact analysis results related to the target hardware, analyze the business services carried by the target hardware and their impact on business processes and data flow, thereby assessing the degree of business impact that the fault may cause.

[0058] Step S6: Generate processing strategy;

[0059] Based on the physical location information of the target hardware, the level of the warning information, and the SLA requirements of the business services, the migration decision matrix is ​​invoked to determine the corresponding processing strategy. When a high-risk fault is determined and the business SLA requirements are high, a target node that meets the load conditions is selected, and business migration and redundant instance deployment operations are performed. When a medium-risk fault is determined, traffic diversion and alarm notification operations are performed. When a low-risk fault is determined, only log information is recorded for subsequent analysis.

[0060] Step S7: Update the dynamic knowledge base.

[0061] After the fault handling is completed, the self-learning engine is activated to perform backtracking analysis on the operational data corresponding to this fault event, extracting fault features including log patterns, time-series features of hardware sensor data, and inflection point features of hardware performance counter data. The graph neural network unit analyzes the correlation between fault features and server hardware components to identify high-confidence fault features. The rule generation unit then generates new fault feature rules from these high-confidence fault features and stores them in a dynamic knowledge base for use in predicting and judging similar faults in the future, thus achieving system self-learning and continuous optimization.

[0062] This invention also provides a server hardware fault early warning system for implementing a method for early warning of server hardware faults. The system adopts a modular structure design, and through the collaborative work of multiple modules, it achieves monitoring, early warning, location, and response processing of server hardware faults, such as... Figure 3 As shown, it includes: a multi-source monitoring module: used to collect server system logs, hardware sensor data, and performance counters. A log acquisition unit: captures system logs in real time and extracts hardware event keywords through natural language processing (NLP). Hardware event keywords include "ECC error".

[0063] The log acquisition unit performs semantic understanding and sentiment analysis on system logs to determine the severity of the hardware status reflected in the logs. The sensor acquisition unit acquires physical indicators, including CPU (Central Processing Unit) temperature, hard drive vibration, and power ripple, through the Baseboard Management Controller (BMC) according to set acquisition frequency and accuracy requirements. The CPU temperature is acquired once per second with an accuracy of ±0.1℃. The sensor acquisition unit detects whether the hard drive vibration amplitude exceeds a threshold set at 4.5 mm / s. The performance probe unit monitors hardware performance counters, including memory bandwidth utilization and PCIe bit error rate. The performance probe unit monitors the trends of these hardware performance counters and triggers an alert when the utilization suddenly increases to a set threshold. The dynamic knowledge base module stores historical fault characteristics and updates rules through a self-learning engine. The fault feature library stores multi-dimensional characteristics of historical faults, including log patterns, sensor timing, and performance inflection points. The fault feature library periodically optimizes and simplifies existing features to improve the efficiency and accuracy of the knowledge base. Self-learning engine: When a new fault occurs, it automatically extracts data features from the hour preceding the fault (T-1 hour) and generates new rules; it constructs a fault propagation model based on a graph neural network (GNN) to learn component-level fault associations; the self-learning engine includes a fault feature extraction unit, a graph neural network unit, and a rule generation unit. Fault feature extraction unit: Extracts log patterns, sensor time series, and performance inflection points from new fault events. Graph neural network unit: Constructs a hardware component fault dependency model to determine the weights and confidence levels of component-level fault associations. Rule generation unit: Stores high-confidence features in a knowledge base. After generating first-level and / or second-level warning information, the intelligent analysis engine is invoked to perform dual warning determination through rule matching and deep learning model inference; a primary warning is triggered when multiple source data simultaneously match preset rules; a high-level warning is triggered when data does not match rules but the confidence level of inference by a temporal convolutional network (TCN, a deep learning model for processing sequence data) model is greater than 90%; the TCN model is used to analyze sensor data trends and identify latent faults, including capacitor aging. The intelligent analysis engine should generate warning information within seconds when a primary warning is triggered, and complete inference and judgment within minutes when a high-level warning is triggered. The root cause localization module outputs the physical coordinates of the faulty hardware based on the fault propagation map. Combining the server's hardware topology and actual layout, the root cause localization module provides intuitive and detailed fault location information. When generating a fault impact report, it correlates business services, dependent components, business processes, and data flow. Through the fault propagation map, the root cause localization module can pinpoint the faulty component to its specific physical location.The adaptive response module generates migration decisions based on fault levels and Service Level Agreements (SLAs). The migration decision matrix executes different actions based on fault levels, SLAs, and target node load, including but not limited to immediate migration and redundancy startup, traffic and alarm routing, and logging only. The migration decision matrix formulates specific migration strategies and priorities based on different types of service SLAs and fault levels. When executing migration decisions, the adaptive response module considers the resource reservation and scalability of the target node to ensure service stability and reliability after migration. It also calculates the weight of target node resource utilization in relation to the service SLA. If the fault level is high-risk and the target node load is less than 70%, service migration and redundant instance deployment are initiated. The multi-source monitoring module, dynamic knowledge base module, intelligent analysis engine, root cause localization module, and adaptive response module interact and collaborate through data interfaces, achieving full automation and intelligence from data collection to early warning, root cause localization, and response. A data security mechanism is also included to protect the privacy and confidentiality of collected server data and prevent malicious attacks from affecting the early warning system. The system is scalable, allowing for easy addition of new data source types, updates to analytical models, and expansion of response strategies to adapt to the expansion of data center scale and technological advancements. It also includes a user interface with a simple and clear design, enabling operations and maintenance personnel to easily use the system for fault warning and handling. The user interface provides extensive help documentation and technical support, improving user experience and satisfaction.

[0064] This application also provides a specific implementation scenario for generating new fault feature rules. In this scenario, the server has experienced an actual hardware failure during operation, specifically a hard drive suddenly failing and causing the interruption of the business services it carries. After detecting the server failure, the system automatically obtains the server's operating data for the hour prior to the failure. The operating data includes system log features, hardware sensor data, and hardware performance counter data. Analysis reveals that within a preset time period before the failure, low-severity log events related to the hard drive appeared in the system log at a frequency of once every 10 minutes, failing to meet the preset log alarm triggering conditions. The hard drive vibration amplitude gradually increased from 3.2 mm / s to 4.1 mm / s within one hour, consistently remaining below the vibration threshold of 4.5 mm / s. The hard drive read / write throughput cumulatively decreased by approximately 8% within 30 minutes. Fluctuations in storage bandwidth utilization of less than 5% did not trigger threshold-based performance alarms, thus no warning information was generated before the failure occurred. After the hard drive experienced an actual failure, the self-learning engine processed the runtime data, extracted log patterns, hard drive vibration timing features, and performance change inflection points, and used a third neural network model to analyze the correlation between the fault features and various hardware components of the server. This yielded the correlation degree value and corresponding confidence value between the fault features and the target hard drive. When both the correlation degree value and the confidence value were greater than the preset threshold, the fault features and their corresponding runtime data were used to generate new fault feature rules and stored in the preset database. This enabled the system to identify such hidden hardware failures in advance and generate warning information when the server subsequently exhibited the same or similar runtime characteristics, thereby achieving continuous improvement and self-optimization of the server hardware failure warning capability.

[0065] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0066] Embodiments of this application also provide an early warning device for server hardware failures. Figure 4 This is a structural block diagram of a server hardware fault early warning device according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0067] The first acquisition unit 10 is used to acquire the server's operating data, which includes: hardware sensor data, hardware performance counter data, and system log features. The system log features are the characteristics of log information generated during the server's operation. The hardware sensor data includes the vibration amplitude of the hard drive, and the hardware performance counter data includes the memory bandwidth utilization rate.

[0068] The first processing unit 20 is used to process the running data using an analysis engine to obtain rule matching results. The rule matching results indicate whether the running data is successfully matched with the fault feature rules in the preset database. The fault feature rules indicate the correlation between the running data and the server hardware fault.

[0069] The generation unit 30 is used to generate a first-level warning message when the rule matching result indicates a successful match, so as to indicate that there is a risk of server hardware failure.

[0070] The second processing unit 40 is used to process the hardware sensor data using a first neural network model when the rule matching result indicates that the matching has failed, so as to obtain a result of whether to generate a warning information. The first neural network model is a model trained based on historical hardware sensor data in historical server hardware failure samples. The warning information indicates that the hardware sensor data is abnormal.

[0071] By acquiring server hardware sensor data, hardware performance counter data, and system log features, and analyzing operational data using a combination of rule matching and neural network models, this solution can further uncover abnormal changes in hardware sensor data when existing fault feature rules cannot cover them, thereby achieving early identification of new hidden hardware faults. Compared to early warning methods that rely solely on rules or single monitoring data, this solution improves the timeliness of server hardware fault early warning while reducing the risk of business interruption due to unidentified hidden hardware faults, providing a more reliable early warning basis for subsequent fault location and handling. It solves the problem in existing server hardware fault early warning technologies that struggle to promptly identify new hidden hardware faults and accurately locate specific hardware components, leading to a high risk of pre-business interruption.

[0072] In one optional embodiment, the second processing unit includes: a processing module, configured to process the hardware sensor data using a first neural network model to obtain an anomaly confidence level of the hardware sensor data, wherein the anomaly confidence level characterizes the degree of credibility of the hardware sensor data anomaly; a first generation module, configured to generate a second-level warning message characterizing the hardware sensor data anomaly when the anomaly confidence level is greater than or equal to a preset confidence level, wherein the handling priority of the second-level warning message is higher than the handling priority of the first-level warning message; and a first determination module, configured to determine that no second-level warning message needs to be generated when the anomaly confidence level is less than the preset confidence level.

[0073] In another optional embodiment, the device further includes: a second acquisition unit, configured to acquire fault characteristics of operational data, including temporal characteristics of sensor data, when generating first-level and / or second-level warning information; a first determination unit, configured to process the fault characteristics using a second neural network model to determine target hardware, the target hardware representing server hardware at risk of failure, the second neural network model being a model trained based on historical fault characteristic data; a second determination unit, configured to determine the physical location information of the target hardware according to a preset server hardware topology, the preset server hardware topology representing the connection relationship of each server hardware; and a third determination unit, configured to determine the processing strategy for the target hardware based on the physical location information and the level corresponding to the warning information, and generate prompt information including the physical location information of the target hardware and the processing strategy to prompt the execution of the processing strategy for the target hardware.

[0074] In some exemplary embodiments, the third determining unit includes: a second determining module, configured to determine the fault propagation path of the target hardware based on the physical location information of the target hardware, wherein the fault propagation path characterizes the propagation relationship of the fault between server hardware when the target hardware fails; a third determining module, configured to determine the fault risk level of the target hardware based on the fault propagation path; and a fourth determining module, configured to determine the processing strategy of the target hardware based on the fault risk level.

[0075] In some other exemplary embodiments, the apparatus further includes a third processing unit, configured to process the fault features using a self-learning engine to obtain fault feature rules, and store the fault feature rules in a preset database.

[0076] In some exemplary embodiments of this application, the first acquisition unit includes: an acquisition module, configured to acquire system logs and extract preset key fields from the system logs; and a second generation module, configured to generate system log features based on the preset key fields.

[0077] In some further exemplary embodiments of this application, the apparatus further includes: a third acquisition unit, configured to acquire operating data within a preset time period before the server malfunctions in the event of a server malfunction; a fourth processing unit, configured to process the operating data using a self-learning engine to obtain fault features of the operating data; a fifth processing unit, configured to process the fault features using a third neural network model to obtain multiple correlation values ​​between the fault features and each server hardware and a confidence value corresponding to each correlation value, wherein the third neural network model is a model trained based on the correspondence between historical fault features and server hardware; a fourth determination unit, configured to determine fault features whose correlation values ​​are greater than a preset correlation value and whose confidence values ​​corresponding to the correlation values ​​are greater than a preset confidence value, and determine the operating data corresponding to the fault features, and generate fault feature rules based on the fault features and the operating data corresponding to the fault features; and a storage unit, configured to store the fault feature rules in a preset database.

[0078] For a description of the features in the embodiment of the server hardware failure early warning device, please refer to the relevant description of the embodiment of the server hardware failure early warning method, which will not be repeated here.

[0079] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described server hardware failure early warning method embodiments.

[0080] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described server hardware failure early warning method embodiments when running.

[0081] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0082] Embodiments of this application also provide a server hardware failure early warning system, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for performing any of the above-described methods.

[0083] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0084] The foregoing has provided a detailed description of a server hardware failure early warning method, a server hardware failure early warning device, a computer-readable storage medium, and a server hardware failure early warning system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for early warning of server hardware failure, characterized in that, include: The server's operational data is obtained, including hardware sensor data, hardware performance counter data, and system log features. The system log features are the characteristics of log information generated during the server's operation. The hardware sensor data includes the vibration amplitude of the hard drive, and the hardware performance counter data includes memory bandwidth utilization. The analysis engine processes the running data to obtain rule matching results. The rule matching results indicate whether the running data matches the fault feature rules in the preset database. The fault feature rules indicate the correlation between the running data and the server hardware fault. If the rule matching result indicates a successful match, a first-level warning message is generated to indicate that the server hardware has a risk of failure. If the rule matching result indicates a failure to match, a first neural network model is used to process the hardware sensor data to determine whether to generate a warning message. The first neural network model is a model trained based on historical hardware sensor data from historical server hardware failure samples. The warning message indicates that the hardware sensor data is abnormal.

2. The method of claim 1, wherein, The hardware sensor data is processed using a first neural network model to determine whether to generate a warning message, including: The hardware sensor data is processed using the first neural network model to obtain the anomaly confidence level of the hardware sensor data, wherein the anomaly confidence level characterizes the degree of credibility of the anomaly of the hardware sensor data. When the anomaly confidence level is greater than or equal to a preset confidence level, a second-level warning message is generated to characterize the anomaly of the hardware sensor data. The processing priority of the second-level warning message is higher than that of the first-level warning message. If the anomaly confidence level is less than the preset confidence level, it is determined that there is no need to generate the second-level warning information.

3. The method of claim 1, wherein, The method further includes: When generating the first-level warning information and / or the second-level warning information, the fault characteristics of the operating data are obtained, and the fault characteristics include the time-series characteristics of the sensor data. The fault features are processed using a second neural network model to determine the target hardware, which represents the server hardware that is at risk of failure. The second neural network model is a model trained based on historical fault feature data. Based on a preset server hardware topology, the physical location information of the target hardware is determined, wherein the preset server hardware topology represents the connection relationship of each of the server hardware components. Based on the physical location information and the level corresponding to the warning information, a processing strategy for the target hardware is determined, and a prompt message including the physical location information of the target hardware and the processing strategy is generated to prompt the execution of the processing strategy on the target hardware.

4. The method of claim 3, wherein, Based on the physical location information and the level corresponding to the warning information, a processing strategy for the target hardware is determined, including: The fault propagation path of the target hardware is determined based on the physical location information of the target hardware, and the fault propagation path represents the propagation relationship of the fault between the server hardware when the target hardware fails. The fault risk level of the target hardware is determined based on the fault propagation path. The processing strategy for the target hardware is determined based on the fault risk level.

5. The method of claim 3, wherein, After generating the physical location information of the target hardware and the prompt information of the processing strategy, the method further includes: The fault features are processed using a self-learning engine to obtain the fault feature rules, and the fault feature rules are stored in the preset database.

6. The method of claim 1, wherein, Obtain server runtime data, including: Obtain system logs and extract preset key fields from the system logs; The system log features are generated based on the preset key fields.

7. The method of claim 1, wherein, The method further includes: In the event of a server failure, the operational data within a preset time period prior to the server failure is retrieved. The running data is processed using a self-learning engine to obtain the fault characteristics of the running data; The fault features are processed using a third neural network model to obtain multiple correlation values ​​between the fault features and each of the server hardware and the confidence values ​​corresponding to each correlation value. The third neural network model is a model trained based on the correspondence between historical fault features and server hardware. Determine the fault feature whose correlation value is greater than a preset correlation value and whose confidence value corresponding to the correlation value is greater than a preset confidence value; determine the operating data corresponding to the fault feature; and generate the fault feature rule based on the fault feature and the operating data corresponding to the fault feature. The fault characteristic rules are stored in the preset database.

8. A server hardware failure early warning device, characterized by, include: The acquisition unit is used to acquire the server's operating data, which includes: hardware sensor data, hardware performance counter data, and system log features. The system log features are the characteristics of log information generated during the server's operation. The hardware sensor data includes the vibration amplitude of the hard drive, and the hardware performance counter data includes memory bandwidth utilization. The first processing unit is used to process the running data using an analysis engine to obtain a rule matching result. The rule matching result indicates whether the running data is successfully matched with the fault feature rules in the preset database. The fault feature rules indicate the correlation between the running data and the server hardware fault. The generation unit is used to generate a first-level warning message when the rule matching result indicates a successful match, so as to indicate that there is a risk of failure in the server hardware; The second processing unit is used to process the hardware sensor data using a first neural network model when the rule matching result indicates a matching failure, so as to obtain a result of whether to generate a warning information. The first neural network model is a model trained based on historical hardware sensor data in historical server hardware failure samples. The warning information indicates that the hardware sensor data is abnormal.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

10. A server hardware failure early warning system, comprising: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 7.

Images