Monitoring and log integrated fault diagnosis system based on dynamic time window association

Abstract

The invention discloses a monitoring and log integrated fault diagnosis system based on dynamic time window association, which is suitable for operation and maintenance of a data center and a GPU (Graphics Processing Unit) cluster. When the monitoring index is triggered abnormally, the system adaptively calculates a retrieval window according to a sampling period, log delay, network jitter, clock skew and the like, automatically retrieves and aggregates related logs in the window, and generates a structured associated event. Calculating correlation degree through template matching, semantic similarity and time decay, and outputting confidence and candidate root causes; gPU temperature, power consumption, a video memory and PCIe errors are fused to construct a causal chain, early warning and self-healing are realized, a maintenance suppression window is provided for noise reduction, and positioning efficiency and stability are improved.

Description

Technical Field

[0001] This invention relates to the field of operation and maintenance monitoring and log management technology, and in particular to the technology of monitoring indicator and log data fusion modeling, dynamic time window correlation analysis, intelligent alarm and automated operation and maintenance handling in data centers, high-performance computing platforms and large-scale GPU cluster environments. Background Technology

[0002] As GPU clusters continue to expand, failures exhibit multi-source, multi-stage, and strongly coupled characteristics: hardware-level issues such as heat dissipation, power supply, fans, voltage and current sensors, and PCIe links; driver-level issues such as XID and ECC; system-level issues such as kernel errors and I / O timeouts; and business-level issues such as job failures and scheduling anomalies, all interacting with each other. Existing monitoring systems excel at collecting metrics and triggering alarms, but struggle to automatically form an interpretable chain of log evidence when anomalies occur; log platforms excel at centralized storage and retrieval, but mostly rely on post-event evidence collection, lacking automatic linkage and alignment mechanisms with monitoring anomalies.

[0003] Furthermore, static thresholds have significant limitations in GPU scenarios: high-load temperature / power fluctuations can easily trigger a large number of noise alarms; chronic degradation under low load, such as fan attenuation, link error accumulation, and a slow increase in PCIe error counts, is easily missed. Moreover, maintenance operations such as firmware upgrades, driver updates, ACS / ATS switching, container runtime upgrades, and topology rescans can cause short-term metric fluctuations and error logs, creating an alarm storm and masking real faults.

[0004] Therefore, it is necessary to provide a system that can achieve unified collection of monitoring and logs, unified timeline fusion, dynamic time window retrieval, correlation scoring, causal chain inference, maintenance noise reduction, and automated processing closed loop, so as to improve diagnostic efficiency and alarm accuracy. Summary of the Invention

[0005] This invention aims to solve the following technical problems:

[0006] (1) The separation of monitoring and logging makes fault location difficult and time-consuming; (2) Static thresholds cannot adapt to dynamic load and environmental changes, resulting in false alarms and missed alarms; (3) The alarm storm during maintenance is severe and affects the identification of real faults; (4) There is a lack of root cause inference and trend warning capabilities for GPU multi-dimensional features.

[0007] To achieve the above objectives, this invention provides an integrated monitoring and logging fault diagnosis system based on dynamic time window correlation, comprising at least the following functional modules:

[0008] Data acquisition module: Collects monitoring indicator data and log data;

[0009] Time axis fusion module: performs time alignment, cleaning, deduplication, and unified modeling on multi-source data;

[0010] Dynamic time window calculation module: Calculates a dynamic time window when an exception is triggered;

[0011] Log retrieval and aggregation module: Retrieves candidate logs within a dynamic time window and performs aggregation and filtering;

[0012] The correlation analysis module calculates the correlation between log indicators and generates structured correlation event objects based on matching scores, time decay weights, etc.

[0013] Diagnostic inference module: Integrates GPU features to construct anomaly causal chains, and outputs candidate root causes, confidence levels, and suggested actions;

[0014] Maintenance Suppression Window Module: Identifies maintenance windows and suppresses / downweights / marks alarms to avoid noisy alarms and statistical pollution;

[0015] Adaptive alarm module: dynamically adjusts thresholds, levels, and window parameters by combining historical baselines and load assessments;

[0016] Automated operation and maintenance module: Triggers policy actions based on diagnostic results, such as node isolation, GPU load migration, service restart, script repair, notification, etc.

[0017] Key mechanisms and exemplary formulas

[0018] Note: The following formulas are exemplary implementations, and this invention does not limit the use of this form; without changing the technical effect, equivalent substitutions can be achieved by using linear / nonlinear, piecewise functions, regression models, or learning models.

[0019] (1) Adaptive calculation of dynamic time window Δ

[0020] (1)

[0021] in: : Monitor the sampling period; Log write / transmission latency estimation; Network jitter estimation; Clock skew estimation; Abnormal amplitude; : Intensity of abnormal trends; For coefficients;

[0022] Indicates the upper and lower bounds of the cutoff function. These are the minimum and maximum values ​​of the window, used to prevent missed detections due to excessively small values ​​or the introduction of noise due to excessively large values.

[0023] Example definition of abnormal amplitude MMM:

[0024] (2)

[0025] in For indicator values, The historical baseline means and standard deviations To prevent zero constant.

[0026] (2) Log-metric correlation score R (time decay + matching fusion)

[0027] Candidate log entries The timestamp is The matching score is The weight is ,but:

[0028] (3)

[0029] in This represents the number of candidate log entries; The item is used to reflect temporal proximity (the closer the two are) The larger the weight.

[0030] (3) Event confidence score S

[0031] For example: (4)

[0032] in: Baseline deviation (which can be obtained by fusing the intensity of deviation from the baseline, such as temperature, power consumption, and error count); The hit rate of historical similar events (same template, same firmware / driver version, same rack / topology, etc.). These are the weighting coefficients.

[0033] (4) Maintain suppression window weight reduction

[0034] The weighting factor within the maintenance window is Non-maintenance window :

[0035] (5)

[0036] in To maintain the confidence / alarm score after suppression.

[0037] (5) Adaptive threshold T and load assessment D

[0038] Load assessment value DDD can be obtained by weighting and combining multiple indicators:

[0039] (6)

[0040] in This is a normalized load metric (such as CPU, memory, network, GPU utilization, etc.). For weights. Example of a dynamic threshold:

[0041] (7)

[0042] in As the initial threshold, This is the adjustment coefficient.

[0043] Beneficial effects

[0044] Compared with the prior art, the present invention has at least the following beneficial effects:

[0045] (1) Deep integration of monitoring and logs reduces manual comparison across systems;

[0046] (2) Dynamic time window adaptive retrieval reduces log missed detections and noise introduction;

[0047] (3) Relevance scoring and structured event objects enhance interpretability and reusability;

[0048] (4) Maintain the suppression window mechanism to reduce alarm storms and avoid statistical pollution;

[0049] (5) Causal chain inference and trend warning based on GPU multidimensional features to improve root cause localization capability;

[0050] (6) Automated operation and maintenance closed loop shortens fault recovery time and improves cluster stability. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the overall system architecture;

[0052] Figure 2 This is a schematic diagram of the dynamic time window calculation process; Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described are only for explaining the present invention and not for limiting the scope of protection; any equivalent substitutions without departing from the spirit of the present invention fall within the scope of protection.

[0054] Definition of terms and symbols

[0055] : The time when the exception is triggered; : Dynamic time window half width; : Monitor the sampling period; Log write / transmission latency estimate; Network jitter estimate; : Clock skew estimate; Abnormal amplitude (deviation from baseline) : Intensity of abnormal trends; Logs—Monitoring Relevance Score; Event confidence score; : Load assessment value; Dynamic threshold / alarm threshold; Log weight and matching score

[0056] Example 1: Monitoring and data collection, log collection, and dynamic time window association process

[0057] This embodiment is designed for large-scale GPU server scenarios, and realizes an automated closed loop of "monitoring - logging - related events - diagnosis". The process includes S11 to S16.

[0058] S11 Monitoring Data Acquisition (1) Collect hardware health indicators through BMC / IPMI, including GPU temperature, power consumption, fan speed, voltage and current sensors, BMC alarm status, power status, etc. (2) Optionally deploy a lightweight Agent on the node side to collect system indicators such as CPU, memory, disk, network, GPU utilization, etc. (3) The sampling period CCC can be configured to 15s / 30s / 60s, etc., without limitation. Monitoring data is written to a time-series database for real-time alarms and historical trends.

[0059] S12 Log Data Acquisition (1) Collect kernel logs, GPU driver logs (including XID, ECC), container runtime logs, job logs, scheduling logs, etc. through rsyslog, Filebeat, or equivalent log agents. (2) Logs should at least include: timestamp, host identifier, log level, and log content; optional parsing templates and key fields are available for subsequent matching.

[0060] S13 Timeline Fusion and Alignment (1) Unify the time zone of log and monitoring data and perform time correction; estimate clock deviation EEE. (2) Statistical log system write latency. Network jitter , etc., are used as input factors for dynamic window calculations.

[0061] S14 Dynamic Time Window Calculation and Candidate Log Retrieval (1) When monitoring indicators are in Triggering anomalies (such as temperature exceeding a threshold, abnormal power consumption, sudden increase in PCIe error count, etc.) to calculate the dynamic window. (2) In Retrieve candidate log sets within the time range. (3) The candidate set can be filtered by host identifier, GPU ID, PCIe BDF, container / job ID, etc., to reduce irrelevant noise.

[0062] S15 Association Analysis and Structured Event Object Generation (1) Perform keyword matching, log template matching, and semantic similarity calculation on candidate logs to obtain matching scores. (2) Calculate the correlation degree based on temporal proximity and matching score. The confidence level is obtained by combining baseline deviation and historical similarity. (3) Generate a structured associated event object (Event), which includes at least: host identifier, exception time information, exception type, associated log set, and associated log quantity. , Candidate root causes and summary of evidence.

[0063] S16 Output Diagnostic and Treatment Recommendations (1) When When the threshold is exceeded or the policy conditions are met, the root cause inference and suggested actions are output. (2) Trigger automated operation and maintenance policies: node isolation, GPU load migration, service restart, script repair or notification of operation and maintenance personnel, etc.

[0064] Example 2: Two-tier storage system, materialized view and report generation

[0065] To balance high-frequency writes and complex queries, this embodiment employs a dual-layer storage:

[0066] (1) Monitoring indicators are written into the time series database, supporting aggregation, downsampling, and quantile statistics;

[0067] (2) Logs and related event objects are written to a relational / retrieval database and composite indexes are created (host, time, exception type, version dimension, etc.).

[0068] (3) Construct materialized views at hourly / daily / versional levels and refresh them periodically;

[0069] (4) Generate trend reports and distribution reports based on views (by cluster, cabinet, firmware / driver version, anomaly type, etc.).

[0070] Example 3: GPU Feature Fusion, Anomaly Causal Chains, and Trend Early Warning

[0071] This embodiment addresses the multidimensional features and root cause complexity of GPU scenarios by adding a causal chain inference and prediction mechanism:

[0072] (1) Construct feature vectors: dynamic features (temperature, power consumption, video memory, error count, frequency reduction, etc.) + static features (firmware version, driver version, PSID, topology, rack location, etc.).

[0073] (2) Generate an abnormal causal chain based on features and associated event objects (fan abnormality → temperature rise → frequency reduction → job failure; or PCIe error count rise → driver error log appears → link quality abnormality).

[0074] (3) Provide trend warnings for chronic degradation (fan efficiency decline, link error accumulation) and output early maintenance suggestions.

[0075] Example 4: Maintenance of the Suppression Window Mechanism

[0076] This example addresses the alarm storm caused by batch maintenance and change operations:

[0077] (1) Maintenance identification methods include, but are not limited to: log keyword identification (upgrade start / finish, driver reload, PCI rescan), change management system records, automated task records, or manual marking;

[0078] (2) Alarm scoring in the maintenance window weight reduction And mark the event as "maintenance related";

[0079] (3) After maintenance is completed, a stability assessment is conducted on the short-term fluctuations after maintenance to avoid concentrated triggering of edge alarms at the moment maintenance ends;

[0080] (4) Maintenance-related events can be excluded from long-term statistics or displayed as a separate dimension to avoid contaminating failure rate analysis.

[0081] Example 5: Bidirectional Associative Reasoning

[0082] In addition to "Alarm → Find Logs", it also enables "Abnormal Logs → Reverse Indicator Lookup":

[0083] (1) When a specific error pattern (XID / ECC / PCIe Bus Error, etc.) appears in the log, check the monitoring indicators in reverse using the dynamic window around the time point of that log;

[0084] (2) If both abnormal metrics and abnormal logs occur simultaneously, increase the correlation confidence.

[0085] (3) Supports triggering early warnings from the log side, improving the ability to capture faults where "logs come first and then indicators deteriorate".

Claims

1. A monitoring and log integrated fault diagnosis method based on dynamic time window correlation, characterized in that, include: A) Collect monitoring metrics and log data of the target node; B) Perform timeline fusion and timestamp alignment on the monitoring indicator data and log data; C) When a monitored indicator triggers an abnormal event, calculate the time of the abnormal trigger. and dynamic time window and in A) Retrieve candidate log sets; B) Perform correlation analysis on the candidate log sets to generate structured correlation event objects; C) Output fault diagnosis results based on the structured correlation event objects and trigger automated operation and maintenance strategies.

2. The method of claim 1, wherein, The monitoring metrics data include at least one or more of the following: GPU temperature, GPU power consumption, memory utilization, PCIe error count, and fan speed.

3. The method of claim 1, wherein, The log data includes at least one or more of the following: operating system kernel logs, GPU driver logs, container runtime logs, and job / scheduling logs.

4. The method of claim 1, wherein, The dynamic time window At least based on the monitoring sampling period Log delay Network jitter Clock deviation Abnormal amplitude and trend strength Any one or more adaptive computations in the range.

5. The method of claim 4, wherein, The The calculation is performed in the following exemplary manner: (1) 6. The method of claim 1, wherein, The candidate log set is filtered by at least the host identifier, GPU identifier, or PCIe BDF identifier.

7. The method of claim 1, wherein, The correlation analysis includes calculating the log-indicator correlation score. The It is calculated based at least on log matching score and time decay weight.

8. The method according to claim 7, characterized in that, The RRR is calculated in the following exemplary manner: (2) 9. The method according to claim 1, characterized in that, The structured associated event object includes at least: host identifier, exception time information, exception type, associated log set, number of associated logs, and association score. and confidence score .

10. The method of claim 9, wherein, The confidence score At least based on correlation score Baseline deviation Hit rate similar to historical data Any one or more calculations in the above.

11. The method of claim 1, wherein, The fault diagnosis results include an abnormal causal chain, which is constructed by fusing monitoring indicator features and log event features, and outputs candidate root causes and their confidence levels.

12. The method of claim 12, wherein, The maintenance status is identified by any one or more of the following: log keywords, change management records, automated task records, or manual markings.

13. The method of claim 1, wherein, It also includes adaptive alarms: dynamically adjusting alarm thresholds or alarm levels based on historical baselines and load assessment values.

14. The method of claim 14, wherein, The load evaluation value For multi-index weighted fusion, the indexes at least include any one or more of CPU usage, memory usage, network indexes, GPU utilization.

Images