Method and system for monitoring health state of server equipment

Abstract

The invention provides a method and system for monitoring the health state of server equipment, and relates to the technical field of server monitoring. The system comprises a data acquisition module, a message processing module, an event distribution module, a log conversion module, a mapping table module, a health state management module and an alarm module. According to the method, data and SEL logs are collected in parallel through multiple threads, events are converted into uniform IDL logs, the uniform IDL logs are associated to specific hardware equipment through a mapping table, the sub-health state, the overall health state and the type summarized health state of the equipment are updated based on the alarm level, and accurate alarms are generated when the equipment is abnormal. According to the invention, through the design of the multi-level health state architecture and the mapping table, the accurate positioning of the fault is realized; the monitoring real-time performance is guaranteed through multiple threads and a priority message queue; the resource consumption is reduced through the differentiation strategy, the method is suitable for large-scale server cluster deployment, and the problems that an existing monitoring mode is poor in real-time performance, inaccurate in positioning and the like are effectively solved.

Description

A method and system for monitoring the health status of server devices Technical Field

[0001] This invention relates to the field of server monitoring technology, specifically to a method and system for monitoring the health status of server equipment, which is particularly suitable for real-time and accurate health status monitoring and fault early warning of hardware equipment in large-scale server clusters. Background Technology

[0002] As the core hardware of a data center, the health status of servers directly affects data processing and service stability. Existing server health monitoring methods mainly fall into three categories: 1. Periodic polling monitoring: This involves periodically calling operating system commands, API interfaces, or SNMP protocols to collect sensor data. While simple to implement, it suffers from poor real-time performance and consumes significant network and system resources during large-scale deployments, leading to congestion or excessive load; 2. Offline log analysis monitoring: This involves processing massive amounts of log data offline to identify fault trends, but it suffers from significant latency and cannot provide real-time responses to critical equipment failures; 3. Centralized monitoring based on SNMP: This involves collecting device status data through an SNMP agent and reporting it to a central server. However, the monitoring items are independent of each other, lacking correlation analysis capabilities and failing to accurately determine the health status of individual devices.

[0003] The aforementioned existing technologies all have obvious shortcomings and are difficult to meet the core requirements of large-scale server clusters for health monitoring in terms of "real-time performance, accuracy, and low resource consumption." There is an urgent need for a technical solution that can achieve comprehensive, in-depth, and efficient monitoring. Summary of the Invention

[0004] This invention aims to provide a method and system for monitoring the health status of server equipment, solving the problems of poor real-time performance, high resource consumption, and inaccurate fault location in existing monitoring methods, and realizing multi-level, real-time, accurate monitoring and rapid fault early warning of server hardware equipment.

[0005] This invention is specifically implemented through the following technical solution: A system for monitoring the health status of server devices, comprising a data acquisition module, a message processing module, an event distribution module, a log conversion module, a mapping table module, a health status management module, and an alarm module; the data acquisition module includes multiple parallel monitoring threads for collecting status data of different hardware devices and SEL log events from the server's BMC; the message processing module includes a message queue and a dedicated receiving thread, the message queue employing a priority processing mechanism for receiving and caching standardized messages; the event distribution module distributes messages to corresponding dedicated processing functions according to device type; the log conversion module converts processed events into a unified IDL log format, the IDL log including public fields conforming to the IPMI specification and extended fields customized by the hardware device; the mapping table module stores a mapping table composed of DeviceTypeMapping structures for establishing the association between IDL logs and hardware devices; the health status management module incorporates a single-device health status structure and a device type summary health status structure for updating the device's sub-health status, overall health status, and type summary health status; the alarm module monitors changes in health status and generates and sends alarm information.

[0006] As a further improvement of the present invention, the IDL log structure includes common fields and extended fields. The common fields include RecID, SensorType, and SensorNum. The extended fields include ExtendEvtLevel and ExtendEvtData1. The ExtendEvtLevel field is used to identify the alarm level, including four levels: Info, Minor, Major, and Critical.

[0007] As a further improvement of the present invention, the DeviceTypeMapping structure of the mapping table includes a log matching field and a hardware device association field.

[0008] As a further improvement of the present invention, the single device health status structure includes a physical location identifier, an overall health status, and multiple sub-health status fields; the overall health status includes three levels: normal, severe, and emergency, which are determined by the sub-health status according to the device-specific strategy.

[0009] As a further improvement of the present invention, the device type summary health status structure includes a device quantity field and a summary health status field, wherein the summary health status is determined by the overall health status of all devices of that type according to a preset strategy.

[0010] This invention also provides a method for monitoring the health status of server devices, applied to the aforementioned system. The method includes the following steps: S1: Multi-threaded parallel acquisition of hardware device status data and SEL log events; S2: When abnormal data or new SEL events are detected, a standardized message is constructed and sent to a message queue according to priority; S3: A dedicated receiving thread retrieves the message and distributes it to the corresponding processing function according to device type; S4: The processed events are converted into unified IDL logs; S5: The matching fields of the IDL logs are parsed, a mapping table is queried, and the corresponding hardware device information is obtained; S6: The sub-health status of the target device is updated according to the alarm level of the IDL logs; S7: The overall health status of the device is determined from the sub-health status according to the device-specific policy, and then the summary health status of the device type is updated according to a preset policy; S8: Changes in health status are monitored, and when an anomaly occurs, alarm information containing device identifier, location, and fault context is generated and sent.

[0011] As a further improvement of the present invention, in step S6, when the alarm level is Info or Minor, the sub-health status is set to Normal; when the alarm level is Major, the sub-health status is set to Severe; and when the alarm level is Critical, the sub-health status is set to Emergency.

[0012] As a further improvement of the present invention, in step S7, the CPU and memory devices use the worst-case state determination method to determine the overall health status; the fan device uses the critical position determination method to determine the overall health status.

[0013] As a further improvement of the present invention, the worst-case state determination method means that when the value of a certain sub-health state reaches "critical", the overall health state is immediately set to "critical"; when the value of a certain sub-health state reaches "urgent", the overall health state is immediately set to "urgent". The critical position determination method means that even if the fans in multiple non-critical positions completely fail, the value of the overall health state value SummaryAlarmLevel is still "normal". Only when the fans in the critical position fail will the value of the overall health state value SummaryAlarmLevel become "critical" or "urgent".

[0014] As a further improvement of the present invention, in step S8, the alarm information is sent via one or more of the following methods: SNMP Trap, SMS, and email.

[0015] The beneficial effects of this invention are: 1. Achieving multi-level precise monitoring: Through a hierarchical architecture of single device sub-state - overall state - type summary state, combined with a mapping table to accurately associate logs and devices, fault location accuracy can reach the sub-state level; 2. Balancing real-time performance and efficiency: Multi-threaded parallel acquisition and a priority message queue mechanism ensure the real-time performance of data acquisition and event processing, while differentiated acquisition strategies reduce system resource consumption; 3. Strong scalability: When adding new devices or event types, only the mapping table and corresponding processing functions need to be expanded, without modifying the core framework; 4. Precise and efficient alarms: The multi-level linkage alarm mechanism carries detailed fault context, supports multiple alarm methods, and significantly improves operation and maintenance response efficiency; 5. Good compatibility: Supports integration with existing monitoring platforms and automated operation and maintenance systems, and is suitable for large-scale server cluster deployment scenarios. Attached Figure Description

[0016] Figure 1 is a system function diagram of the health status monitoring server device of the present invention; Figure 2 is a schematic diagram of the single device health status judgment mechanism of the present invention; Figure 3 is a schematic diagram of the FAN device type summary health status judgment mechanism of the present invention; Figure 4 is a schematic diagram of the mapping working mechanism of the present invention; Figure 5 is a schematic diagram of the warning mechanism of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] As shown in Figure 1, the server device health status monitoring system of the present invention adopts a multi-layer architecture design, realizing a complete monitoring link from data acquisition to alarm generation. The complete data flow of the server device health status monitoring system is as follows: Data acquisition: Multiple monitoring threads collect status data of various hardware devices in parallel, and the system also monitors SEL log events recorded by the BMC. That is to say, the data source for device health in the server is the SEL log and the data obtained by multiple threads monitoring hardware devices using certain strategies.

[0019] Message processing: Upon detecting abnormal data or a new SEL event, a standard message is constructed and sent to the message queue, where it is processed by a dedicated receiving thread.

[0020] Event dispatch: The receiving thread dispatches data to the corresponding processing function according to the message type for specialized processing, such as CPU data processing, NVME device processing, RAID card processing, etc.

[0021] Log transformation: Processed events are converted into a unified IDL log format within the system to standardize event information.

[0022] Log and Device Mapping Table: When the system generates an IDL log entry, it knows which hardware device's fault information each log entry points to, thereby updating the monitoring status of the corresponding hardware device. By querying the mapping table, IDL logs are associated with specific hardware devices, physical locations, and sub-components.

[0023] Alarm Levels: Alarm levels are predefined for each IDL log entry and stored in an array table. When an IDL log entry is received, its alarm level can be determined by looking up the table, and the hardware device's health status value is updated accordingly. There are four log levels: Info, Minor, Major, and Critical. Hardware health status has three levels: Normal, Critical, and Critical.

[0024] Health Status Update: Update the device's sub-health status and overall health status based on event type and alarm level. When the log level is Info or Minor, the hardware health status is set to Normal; when the log level is Major, the hardware health status is set to Critical; and when the log level is Critical, the hardware health status is set to Emergency.

[0025] Status monitoring and alarms: Monitor changes in the health sub-states of devices and the overall health status of individual devices, generate alarm information and send it through a unified interface, including SNMP traps, SMS, emails, etc.

[0026] This data flow forms a complete closed-loop system, from raw data acquisition to final alarm generation, enabling comprehensive and accurate health monitoring of server hardware. The layered design ensures loose coupling between the system's functional modules, improving maintainability and scalability.

[0027] By modularizing the above functions, the system for monitoring the health status of server devices according to the present invention can be obtained, including a data acquisition module, a message processing module, an event distribution module, a log conversion module, a mapping table module, a health status management module, and an alarm module.

[0028] The core of this design is to define a device health status structure and a mapping table between logs and hardware information for each hardware device. Each hardware device in the server has a defined health status structure containing its physical location identifier, overall health status, and multiple sub-health statuses. The overall health status of a single device is determined by the sub-health statuses and the health status policy for this type of device. The mapping table associates each IDL log entry with a specific hardware device. When an IDL log entry is received, the mapping table indicates which hardware device has failed, allowing for accurate updates to the health status value in that hardware device's health status structure.

[0029] Device Health Status Data Structure Single Device Health Status Structure: typedef struct {uint8 SocketID; / / Physical location identifier uint8 AllAlarmLevel; / / Overall CPU health status / / Sub-health status uint8 SubStatus1AlarmLevel; / / Sub-status 1 (e.g., temperature) uint8 SubStatus2AlarmLevel; / / Sub-status 2 (e.g., voltage) uint8 SubStatus3AlarmLevel; / / Sub-status 3 (e.g., power) / / ...other sub-status} CPUDeviceHealthInfo_T; Taking the CPU as an example, the SocketID variable in this structure stores the physical location identifier of the CPU, distinguishing CPUs in different locations in a server with multiple CPUs. The AllAlarmLevel variable identifies the overall health status of this CPU, while variables such as SubStatus1AlarmLevel store the CPU's sub-status, such as temperature, voltage, and performance. The overall health status of the CPU is determined by a strategy that combines the sub-health statuses of each CPU. For example, a "worst-case scenario" approach can be used: if the value of a sub-health status reaches "critical," the overall health status is immediately set to "critical"; if the value of a sub-health status reaches "urgent," the overall health status is immediately set to "urgent." Different devices can use different determination strategies.

[0030] A summary health status structure for a specific device type: `typedef struct {uint8 Count; / / Number of devices uint8 SummaryAlarmLevel; / / Summary health status} DeviceTypeSummaryInfo_T;` This structure records the summary health status of a specific type of device. `Count` records the number of devices of this type, and `SummaryAlarmLevel` records the health status value of this type of device. This value is obtained from the total health status of each device (`AllAlarmLevel`) and the set judgment strategy. As shown in Figure 2, for example, devices like CPUs use the "worst-case scenario determination method" to obtain the summary health status value `SummaryAlarmLevel`. As shown in Figure 3, devices like FANs use the "critical location FAN fault judgment method," meaning that even if multiple non-critical fans completely fail, the total health status value `SummaryAlarmLevel` will still be "normal." Only when a critical fan fails will the total health status value `SummaryAlarmLevel` become "critical" or "urgent."

[0031] This hierarchical structure enables the system to accurately record the health status of various aspects of the equipment, and to make highly customizable judgments on the health status of the equipment, which facilitates fault location and status management.

[0032] IDL Log Format IDL Log Format Structure: typedef struct { / / IPMI-compliant sel log format, common fields for various hardware components uint16_t RecID; uint8_t Type; INT32U TimeStamp; uint8_t GenID [2]; uint8_t EvMRev; uint8_t SensorType; uint8_t SensorNum; uint8_t EvtDirType; uint8_t EvtData1; uint8_t EvtData2; uint8_t EvtData3; / / Extended fields, meanings and usage of custom fields for various hardware devices uint8_t ExtendFlag; uint8_t ExtendEvtLevel; uint32_t ExtendEvtData1; uint32_t ExtendEvtData2; uint32_t ExtendEvtData3;} PACKED IDLEventRecord_T; All hardware fault information will be uniformly converted into the form of IDL logs. The format of IDL logs is as follows: the first half conforms to the IPMI specification and is consistent with SEL logs, while the second half is customized by each type of hardware device using its own practical method. Hardware fault information comes from two sources: first, SEL logs, which conform to the IPMI specification; each SEL log entry is converted into an IDL log; and second, fault information obtained from hardware devices monitored by multiple threads, which is also converted into IDL log format.

[0033] The mapping mechanism's mapping table structure is defined as follows: `typedef struct {uint8_trecord_type; / / Record type uint8_tsensor_type; / / Sensor type uint8_tevent_type; / / Event type uint8_tevent_offset; / / Event offset uint8_tsub_offset; / / Sub-offset uint16_t sensor_num; / / Sensor number uint8_tdevice_type; / / Mapped device type uint8_tdevice_sub_Type; / / Device sub-type uint8_tdevice_location_id; / / Device physical or logical location identifier uint8_tselect_para; / / Selection parameter (i.e., the parameter corresponding to the sub-state of the hardware device)} DeviceTypeMapping;` The mapping table is an array of `DeviceTypeMapping` structures. Each `DeviceTypeMapping` structure consists of two parts.

[0034] The first part (from the record_type to the sensor_num fields) contains data from the corresponding IDL log, used to match the data with the IDL log.

[0035] The latter part contains hardware information, including parameters such as device type (device_type), device location ID (device_location_id), and sub-state type (select_para), which are used to point to specific hardware devices.

[0036] As shown in Figure 4, after the query matching function receives an IDL log, it parses the data in the IDL log and obtains data such as record_type, sensor_type, event_type, event_offset, sub_offset, and sensor_num from the IDL log. These fields are then passed as parameters to the mapping query function. The function iterates through each entry in the mapping table, comparing the parameters in the log with the corresponding fields in the mapping table item by item. If a comparison is successful, the hardware device information is obtained, and the mapping table is queried to retrieve the hardware information. After obtaining the information of these devices, it can be determined which hardware device has failed, and subsequent code will accurately update the health status of this device.

[0037] Based on this hardware information, the system locates the data structure corresponding to the health status of the target device (e.g., the CpuDevHealthInfo_T structure for the CPU) in a pre-established device health status database. This structure stores all key sub-states of the device (such as temperature, voltage, fan status, etc.) and their overall health status. Subsequently, the system updates the individual sub-state data, automatically writing the device health level corresponding to the IDL log alarm level into the relevant sub-state field of the target device based on the "sub-state type" returned from the mapping table.

[0038] As shown in Figure 5, when an anomaly occurs in a single sub-state or the overall state, the system will further trigger the alarm processing module to notify the administrator or automatically handle the abnormal situation.

[0039] The monitoring system in this embodiment is deployed in a cluster of 8 servers. Each server is equipped with 2 CPUs, 16GB of memory, 2 NVMe hard drives, 1 RAID card, and 4 fans. • The data acquisition module starts 6 parallel monitoring threads, corresponding to CPU, memory, NVMe hard drives, RAID card, fans, and SEL log monitoring, respectively. • The message queue is set with 3 priority levels, with Critical level alarm messages having the highest priority and Minor level messages having the lowest. • The mapping table is pre-configured with association rules for 6 types of devices, including CPU and memory. Device location identification uses a combination of SocketID and slot number. • In the health status management module, the CPU uses the "worst-case scenario determination method," and the fans use the "critical location determination method" (fans #1 and #4 are critical locations).

[0040] The monitoring method in this embodiment includes the following specific operations: 1. Data acquisition: The CPU monitoring thread collects temperature and voltage data every 100ms, the fan monitoring thread collects speed data every 200ms, and the SEL log monitoring thread captures log events in real time; 2. Message processing: When the temperature of CPU0 exceeds 85℃ (Minor level alarm), a message is constructed and sent to the message queue with priority 2; 3. Event distribution: The receiving thread retrieves the message and distributes it to the CPU-specific processing function; 4. Log conversion: The processed temperature anomaly event is converted into an IDL log, and ExtendEvtLevel in the extended field is set to Minor; 5. Device association: The sensor_type (CPU temperature sensor), sensor_num (0#) and other fields of the IDL log are parsed, the mapping table is queried, the device type is determined to be CPU_DEV_TYPE, the location identifier is SocketID=0, and the sub-state type is CPUTEMP_TYPE; 6. Health status update: The temperature sub-health status of CPU0 is set to "Normal" (Minor). (Corresponding to the level), since all other sub-states are normal, the overall health status of CPU0 remains "normal", and the overall health status of CPU type is still "normal"; 7. Alarm generation: Since the alarm level is Minor, no external alarm is triggered, only the health status record is updated; if a CPU0 voltage abnormality is subsequently detected (Major level alarm), the corresponding sub-health status is set to "critical", the overall health status of CPU0 is updated to "critical", the overall health status of CPU type is updated to "critical", and the alarm module triggers email and SNMP Trap alarms, carrying SocketID=0 and voltage abnormality context.

[0041] The advantages of this invention are: 1. The precise design of the log and device mapping table is achieved by uniformly parsing various types of raw data into IDL logs. The mapping table is used to accurately match each field in the log record with predefined rules, so that each log can be accurately mapped to a specific device and the physical location of the device and its corresponding sub-state can be determined.

[0042] This mapping not only ensures consistency between different data sources, but also allows adding new devices or event types to be accomplished simply by expanding the mapping table, without requiring significant modifications to the core data processing framework.

[0043] 2. The real-time monitoring system adopts a multi-threaded and asynchronous message queue mechanism, which can collect and process monitoring logs from various data sources in real time, ensuring that parsing, mapping and status updates can be completed in a very short time after abnormal logs are collected.

[0044] This real-time capability ensures that changes in device status can be quickly captured and corresponding alarm responses can be triggered in a timely manner, reducing the impact of potential failure risks on the overall system.

[0045] 3. Monitoring of Individual Device Sub-states and Overall Status: Each device not only records the sub-state of individual indicators, but different strategies can be used to determine the overall health status value of the device for different device types. When a sub-state is abnormal, the system can update the device's information in a timely manner based on the mapping results and dynamically adjust the overall status, ensuring that the monitoring system has both fine-grained data and the ability to judge the overall macro-level health status.

[0046] This layered monitoring design allows for more detailed fault location, enabling problems to be detected at the sub-state level while simultaneously providing a rapid overall reflection of the equipment's operational status.

[0047] In summary, the log and device mapping table not only achieves standardization and accurate mapping of multiple data sources, but also incorporates real-time data acquisition and hierarchical monitoring mechanisms, fully demonstrating the significant advantages of this invention in real-time performance and single-device segmentation and overall status monitoring.

[0048] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A system for monitoring the health status of server equipment, characterized in that, The system includes a data acquisition module, a message processing module, an event distribution module, a log conversion module, a mapping table module, a health status management module, and an alarm module. The data acquisition module includes multiple parallel monitoring threads for collecting status data from different hardware devices and SEL log events from the server's BMC. The message processing module includes a message queue and a dedicated receiving thread. The message queue uses a priority processing mechanism to receive and cache standardized messages. The event distribution module distributes messages to corresponding dedicated processing functions according to device type. The log conversion module converts processed events into a unified IDL log format. The IDL log includes common fields conforming to the IPMI specification and extended fields defined by the hardware device. The mapping table module stores a mapping table composed of DeviceTypeMapping structures to establish the association between IDL logs and hardware devices. The health status management module has a built-in single device health status structure and a device type summary health status structure, which are used to update the device's sub-health status, overall health status, and type summary health status; the alarm module is used to monitor changes in health status and generate and send alarm information.

2. The system according to claim 1, characterized in that, The IDL log structure includes common fields and extended fields. The common fields include RecID, SensorType, and SensorNum. The extended fields include ExtendEvtLevel and ExtendEvtData1. The ExtendEvtLevel field is used to identify the alarm level, which includes four levels: Info, Minor, Major, and Critical.

3. The system according to claim 1, characterized in that, The DeviceTypeMapping structure of the mapping table includes a log matching field and a hardware device association field.

4. The system according to claim 1, characterized in that, The single device health status structure includes a physical location identifier, an overall health status, and multiple sub-health status fields; the overall health status includes three levels: normal, severe, and emergency, which are determined by the sub-health status according to the device-specific strategy.

5. The system according to claim 1, characterized in that, The device type summary health status structure includes a device quantity field and a summary health status field. The summary health status is determined by the overall health status of all devices of that type according to a preset strategy.

6. A method for monitoring the health status of server equipment, applied to the system as described in any one of claims 1-5, characterized in that, The method Includes the following steps: S1: Multi-threaded parallel acquisition of hardware device status data and SEL log events; S2: When abnormal data or a new SEL event is detected, a standardized message is constructed and sent to the message queue according to priority; S3: A dedicated receiving thread retrieves the message and distributes it to the corresponding processing function according to the device type; S4: The processed event is converted into a unified IDL log; S5: The matching fields of the IDL log are parsed, the mapping table is queried, and the corresponding hardware device information is obtained; S6: Update the sub-health status of the target device based on the alarm level of the IDL log; S7: Determine the overall health status of the device based on the sub-health status according to the device-specific policy, and then update the summary health status of the device type according to the preset policy. S8: Monitor changes in health status, and generate and send alarm information containing device identifier, location and fault context when an anomaly occurs.

7. The method according to claim 6, characterized in that, In step S6, when the alarm level is Info or Minor, the sub-health status is set to Normal; when the alarm level is Major, the sub-health status is set to Severe; and when the alarm level is Critical, the sub-health status is set to Emergency.

8. The method according to claim 6, characterized in that, In step S7, the CPU and memory devices use the worst-case scenario method to determine their overall health status; the fan device uses the critical location method to determine its overall health status.

9. The method according to claim 8, characterized in that, The worst-case scenario determination method means that if the value of a certain sub-health state reaches "critical", the overall health state is immediately set to "critical"; if the value of a certain sub-health state reaches "urgent", the overall health state is immediately set to "urgent". The critical location determination method means that even if multiple non-critical fans fail completely, the overall health state value SummaryAlarmLevel will still be "normal". Only when a critical fan fails will the overall health state value SummaryAlarmLevel become "critical" or "urgent".

10. The method according to claim 6, characterized in that, In step S8, the alarm information is sent via one or more of the following methods: SNMP Trap, SMS, and email.

Images