Full-link dynamic monitoring and abnormity simulation test system and method

Abstract

The invention provides a full-link dynamic monitoring and anomaly simulation test system and method. The system comprises a cross-level I / O load simulation engine module, a multi-level collaborative fault injection module, a full-link dynamic monitoring module and an intelligent root cause analysis module. The cross-level I / O load simulation engine module is used for simulating business pressure fluctuation, the multi-level collaborative fault injection module is used for synchronously injecting configurable anomalies and realizing cross-level fault conduction, and the full-link dynamic monitoring module realizes request-level end-to-end tracking and acquires monitoring data through a distributed probe. And the intelligent root cause analysis module positions a fault root and generates a repair suggestion in combination with LSTM time sequence prediction and Bayesian causal reasoning. According to the invention, the overall performance and fault response capability of the system can be comprehensively evaluated, and the accuracy and efficiency of system testing are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer system testing, and in particular to a testing system and method for full-link dynamic monitoring and abnormality simulation. Background Art

[0002] With the widespread application of distributed storage systems in key areas such as cloud computing and finance, their security and reliability have become core issues in digital transformation. The complexity and high reliability requirements of distributed storage systems now pose severe challenges to their testing and verification. First, traditional static testing cannot simulate variable load scenarios under dynamic resource scheduling, resulting in significant deviations between the evaluation results of performance indicators such as read and write latency and IOPS and actual business scenarios. Second, existing fault injection technologies mainly target a single layer (such as network packet loss or storage medium failure), and fail to build a collaborative fault model covering multiple levels of the I / O stack (application interface layer-file system layer-block device driver layer), and cannot effectively reproduce the cascading failure phenomenon caused by cross-level abnormal transmission. Third, the current monitoring system lacks full life cycle tracking capabilities. It can neither achieve full-link observability from request initiation to persistent storage, nor can it perform fine-grained monitoring of system status and multi-dimensional data synchronization analysis. Summary of the Invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] The present invention proposes a full-link dynamic monitoring and exception simulation test system. It generates composite loads of the application layer, file system layer, and block device layer through a cross-layer IO load simulation engine, adopts multi-anomaly superposition injection technology to simulate cascading failures of hardware failures, network anomalies, and silent data corruption, builds end-to-end tracing full-link dynamic monitoring, and combines the LSTM prediction model with the causal reasoning algorithm to realize intelligent root cause analysis.

[0005] Another object of the present invention is to propose a testing method for full-link dynamic monitoring and abnormal simulation.

[0006] To achieve the above objectives, the present invention proposes a full-link dynamic monitoring and abnormality simulation test system, comprising:

[0007] The cross-layer I / O load simulation engine module takes the set business scenario parameters and load characteristic parameters as input, and outputs dynamically generated composite load data for the application layer, file system layer, and block device layer to simulate business pressure fluctuations;

[0008] A multi-level collaborative fault injection module takes composite load data and configurable anomaly parameters as input, and outputs anomaly data of hardware, network, and data layers, as well as cross-level fault propagation data;

[0009] The full-link dynamic monitoring module takes composite load data and fault conduction data after anomaly injection as input and outputs request-level end-to-end tracing data collected by distributed probes, including millisecond-level performance indicators and contextual data.

[0010] The intelligent root cause analysis module takes request-level tracing data, composite load data, exception data, and fault conduction data as input, and outputs fault root cause location results and repair suggestions derived through LSTM time series prediction and Bayesian causal reasoning.

[0011] The full-link dynamic monitoring and abnormality simulation test system of the embodiment of the present invention may also have the following additional technical features:

[0012] In one embodiment of the present invention, the cross-level I / O load simulation engine module generates an 8:2 read-write mixed load so that the pressure peak reaches a preset IOPS value.

[0013] In one embodiment of the present invention, the multi-level coordinated fault injection module injects a preset disk delay into a designated storage node.

[0014] In one embodiment of the present invention, the full-link dynamic monitoring module includes:

[0015] a) Client-side probes dynamically intercept POSIX file operation APIs through API hook injection, capture request type, data location, and user context, and generate a globally unique tracking ID for each request;

[0016] b) Network probe, deploying XDP program based on the Linux kernel eBPF mechanism, to capture key indicators of the TCP layer with zero copy;

[0017] c) Storage probe, which hooks block device requests and file system operations in the form of a kernel module to monitor disk queue depth, IOPS and latency distribution;

[0018] d) The tracking ID generated by the client probe is passed to the storage probe through the network probe, and the events of the four major stages of the request life cycle are recorded in real time. Combined with PTP clock synchronization, strict timing correlation of cross-node events is ensured.

[0019] In one embodiment of the present invention, the full-link dynamic monitoring module also includes hierarchical processing of monitoring data. After the original event flow is classified through the streaming filter, normal data is stored in the time series database, abnormal events trigger the real-time analysis pipeline, and visualization is achieved through topological heat maps and fault propagation maps.

[0020] In one embodiment of the present invention, the LSTM prediction model of the intelligent root cause analysis module adopts a three-layer gated recurrent network structure, is trained based on historical monitoring data, predicts the indicator value range for future time periods, and introduces a dynamic threshold adjustment mechanism. When real-time data continues to deviate from the prediction interval and exceeds the set fault tolerance threshold, a multi-level alarm is triggered.

[0021] In one embodiment of the present invention, the causal reasoning engine of the intelligent root cause analysis module constructs a multi-level Bayesian network, whose nodes cover multi-dimensional causal relationships such as hardware indicators, network status, and application layer characteristics. It calculates the conditional probability between events through Gibbs sampling, and sorts the causal contribution based on the real-time context to generate a root cause path diagram with confidence score, a root cause report, and repair suggestions.

[0022] The intelligent root cause analysis module adopts a dual closed-loop feedback mechanism. The LSTM prediction error dynamically corrects the prior probability distribution of the causal network, and the causal analysis results feed back to the LSTM feature weight adjustment.

[0023] To achieve the above objectives, the present invention further proposes a testing method for full-link dynamic monitoring and abnormality simulation, comprising:

[0024] Taking the set business scenario parameters and load characteristic parameters as input, it outputs dynamically generated composite load data for the application layer, file system layer, and block device layer to simulate business pressure fluctuations;

[0025] Taking composite load data and configurable abnormal parameters as input, it outputs abnormal data of hardware, network, and data layers, as well as cross-layer fault transmission data;

[0026] Taking composite load data and fault conduction data after anomaly injection as input, it outputs request-level end-to-end tracing data collected by distributed probes, including millisecond-level performance indicators and context data.

[0027] Taking request-level tracing data, composite load data, exception data, and fault conduction data as input, it outputs fault root cause location results and repair suggestions obtained through LSTM time series prediction and Bayesian causal reasoning.

[0028] A full-link dynamic monitoring and abnormal simulation testing system and method in an embodiment of the present invention aims to solve the technical bottlenecks of distributed storage systems in dynamic load, multi-level failures and monitoring and tracking, realize quantitative reliability evaluation in real scenarios, and realize quantitative reliability evaluation of distributed storage systems under real load pressure and abnormal interference, so as to ensure the reliability and stability of distributed storage systems.

[0029] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 is a structural diagram of a test system for full-link dynamic monitoring and abnormality simulation according to an embodiment of the present invention;

[0032] Figure 2 This is a distributed storage test data diagram for full-link dynamic monitoring and abnormal simulation according to an embodiment of the present invention;

[0033] Figure 3 This is an example diagram of full-link dynamic monitoring and abnormal simulation testing according to an embodiment of the present invention;

[0034] Figure 4 This is a workflow diagram of a full-link dynamic monitoring module according to an embodiment of the present invention;

[0035] Figure 5 is a workflow diagram of an intelligent root cause analysis module according to an embodiment of the present invention;

[0036] Figure 6 This is a flow chart of a test method for full-link dynamic monitoring and abnormal simulation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0039] The following describes a full-link dynamic monitoring and abnormality simulation test system and method proposed according to an embodiment of the present invention with reference to the accompanying drawings.

[0040] First, the technical terms mentioned in this invention are introduced:

[0041] LSTM (Long Short-Term Memory): A specialized recurrent neural network (RNN) designed to address the vanishing / exploding gradient issues of traditional RNNs when training on long sequences of data. It effectively captures long-term dependencies in time series. The input gate determines which information is stored in the memory cell. The forget gate determines which information is discarded from the memory cell. The output gate determines which information is output.

[0042] API Hook: A technique for monitoring or changing system behavior by intercepting and modifying Application Programming Interface (API) calls.

[0043] eBPF (extended Berkeley Packet Filter): An efficient virtual machine running in the kernel that allows user-mode programs to safely inject custom code into kernel-mode execution without modifying the kernel source code or loading modules.

[0044] XDP (eXpress Data Path): A high-performance network data plane processing framework based on eBPF that processes packets directly at the network driver layer (the earliest possible point).

[0045] PTP (Precision Time Protocol): A network protocol used to achieve nanosecond-level clock synchronization.

[0046] Prometheus: An open source monitoring, alarm, and time series database combination that stores all information as time series data.

[0047] 3σ principle: A statistical rule that describes the range of data distribution in the normal distribution, which states that approximately 99.7% of the data fall within the interval of 3 standard deviations (σ) around the mean (μ), that is, (μ-3σ,μ+3σ).

[0048] Bayesian Network: A probability-based graphical model used to represent the causal relationship between variables and their uncertainty.

[0049] Gibbs sampling: An algorithm used in statistics for Markov Monte Carlo to approximate a sequence of samples from a multivariate probability distribution when direct sampling is difficult.

[0050] Figure 1 is a structural diagram of a test system for full-link dynamic monitoring and abnormal simulation according to an embodiment of the present invention, such as Figure 1Shown, including:

[0051] The cross-layer I / O load simulation engine module 100 takes the set business scenario parameters and load characteristic parameters as input, and outputs dynamically generated composite load data of the application layer, file system layer, and block device layer to simulate business pressure fluctuations;

[0052] The multi-level collaborative fault injection module 200 takes composite load data and configurable anomaly parameters as input and outputs anomaly data of hardware, network, and data layers as well as cross-level fault propagation data;

[0053] The full-link dynamic monitoring module 300 takes the composite load data and fault conduction data after anomaly injection as input and outputs request-level end-to-end tracing data collected by distributed probes, including millisecond-level performance indicators and context data;

[0054] The intelligent root cause analysis module 400 takes request-level tracing data, composite load data, abnormal data, and fault conduction data as input, and outputs fault root cause location results and repair suggestions obtained through LSTM time series prediction and Bayesian causal reasoning.

[0055] Specifically, it is mainly implemented by four major modules: a cross-level I / O load simulation engine module, a multi-level collaborative fault injection module, a full-link dynamic monitoring module, and an intelligent root cause analysis module. The cross-level I / O load simulation engine module simulates the multi-level I / O behavior of the distributed storage system, covering the composite loads of the application layer, file system layer, and block device layer, and provides a load template library to support real-time adjustment of load parameters and simulate pressure fluctuations under dynamic resource scheduling. The multi-level collaborative fault injection module provides a preset fault library, synchronously injects faults at the hardware layer, network layer, and data layer, and defines cross-level triggering rules, such as simulating storage medium bad blocks, SSD wear anomalies, and memory page errors at the hardware layer, simulating network packet loss rate, delay jitter, and brain split scenarios at the network layer, and simulating silent data corruption at the data layer. The full-link dynamic monitoring module implants lightweight probes at key nodes in the I / O path, assigns a unique tracking ID to each request, and records the initiation, routing, processing, and completion of request lifecycle events. It uses a distributed clock synchronization protocol to align multi-node timestamps, with errors controlled at the microsecond level. This enables full-link tracking from client requests to persistent storage, provides real-time visualization of multi-dimensional monitoring data, and supports data export and offline analysis. The intelligent root cause analysis module's LSTM prediction model is trained based on historical monitoring data, learns the dynamic patterns of the system under normal conditions, and predicts indicator values ​​in future time windows. When real-time monitoring data deviates significantly from the predicted values, an abnormality alarm is triggered. The causal reasoning engine builds a multi-dimensional causal relationship model based on Bayesian networks and causal graph algorithms. Combined with real-time collected contextual information, it calculates the conditional probability and causal contribution between events, reversely derives the most likely fault transmission path, outputs a root cause analysis report, and provides repair suggestions.

[0056] As attached Figure 2 As shown in the figure, the test system for full-link dynamic monitoring and anomaly simulation is mainly implemented by four modules: a cross-level I / O load simulation engine module, a multi-level collaborative fault injection module, a full-link dynamic monitoring module, and an intelligent root cause analysis module. The cross-level I / O load simulation engine module dynamically generates composite loads at the application layer, file system layer, and block device layer to simulate business pressure fluctuations. The multi-level collaborative fault injection module synchronously injects configurable anomalies (such as bad disk blocks, network packet loss, silent data corruption, etc.) into the hardware, network, and data layers, and supports cross-level fault propagation. The full-link dynamic monitoring module implements request-level end-to-end tracing through distributed probes, collecting millisecond-level performance indicators and contextual data. The intelligent root cause analysis module combines LSTM time series prediction with Bayesian causal reasoning to locate the root cause of the fault from multi-dimensional data and generate repair suggestions.

[0057] As attached Figure 3As shown in the figure, an example of client write latency surge testing and diagnosis is shown. The cross-level I / O load simulation engine module generates an 8:2 read-write mixed load, resulting in a peak pressure of 10,000 IOPS. The multi-level collaborative fault injection module injects a 300ms disk delay into storage node B. The full-link dynamic monitoring module's client probe records that the delay has increased from 50ms to 320ms. The network probe discovers that node B's TCP retransmission rate is 15%. The storage probe detects that node B's disk queue depth reaches 35 and reports monitoring data (including tracking ID) in real time. The intelligent root cause analysis module LSTM determines that the delay deviates from the predicted value by 4σ. The Bayesian network outputs the root cause as disk delay triggering a retry storm (confidence level 0.92%) and provides load optimization suggestions. As a result, the cross-level I / O load simulation engine module reduces the write IOPS to 4,000. The multi-level collaborative fault injection module stops disk delay and simulates network recovery instead.

[0058] As attached Figure 4As shown, the full-link dynamic monitoring module achieves end-to-end I / O path observability through distributed probes and high-precision tracing technology. First, lightweight probes are implanted in key locations such as the client, network, and storage nodes. The client probe dynamically intercepts POSIX file operation APIs through API hook injection, injects lightweight tracing logic in user mode, captures request type, data location, and user context in real time, and generates a globally unique tracing ID for each request. The network probe deploys the XDP program based on the Linux kernel's eBPF mechanism, captures key TCP layer metrics with zero copy, and efficiently reports data through the BPF ring buffer. The storage probe, as a kernel module, hooks block device requests and file system operations and associates the request tracing chain to monitor disk queue depth, IOPS, and latency distribution. The client probe generates a tracing ID (timestamp, user ID, and route), which the network probe transmits to the storage probe via TCP. The probe then records events from the four stages of the request lifecycle (initiation, routing, processing, and completion) in real time. Combined with PTP clock synchronization, this ensures strict temporal correlation of events across nodes, enabling full-link monitoring across user mode, kernel mode, and hardware. Secondly, the monitoring data is collected and dynamically compressed at the millisecond level for multi-dimensional indicators (resource utilization, performance indicators, fault events). The monitoring data is then processed in layers. After the original event flows through the streaming filter for classification, normal data is stored in a time series database (such as Prometheus), and abnormal events trigger a real-time analysis pipeline. Finally, visualization is achieved through topological heat maps and fault propagation maps, and the data interface is opened to support SQL queries and offline root cause analysis. In other words, dynamic compression ensures the feasibility of high-frequency indicator collection, heat maps provide global state perception, and fault maps reveal the abnormal transmission logic. The three together constitute an observability closed loop. The full-link dynamic monitoring module solves the problems of request-level full-link tracking, microsecond event alignment, and real-time correlation of multi-dimensional data in a distributed environment while ensuring low overhead. Typical application scenarios include cascade fault diagnosis (such as network packet loss → disk congestion → client timeout transmission chain analysis) and performance bottleneck location.

[0059] Specifically, the main steps to implement the full-link dynamic monitoring module are as follows:

[0060] Lightweight probes are implanted in key locations such as clients, networks, and storage nodes. Client probes dynamically intercept POSIX file operation APIs through APIHook injection, inject lightweight tracing logic into user mode, capture request types, data locations, and user contexts in real time, and generate a globally unique tracing ID for each request. Network probes deploy XDP programs based on the Linux kernel eBPF mechanism, capture key indicators at the TCP layer with zero copy, and efficiently report data through the BPF ring buffer. Storage probes hook block device requests and file system operations in the form of kernel modules, and associate request tracing chains to monitor disk queue depth, IOPS, and latency distribution. That is, the tracing ID (timestamp, user identity ID, and route) generated by the client probe is passed from the network probe to the storage probe via TCP, and events in the four major stages of the request life cycle (initiation, routing, processing, and completion) are recorded in real time. Combined with PTP clock synchronization, this ensures strict temporal correlation of events across nodes, enabling full-link monitoring across user mode, kernel mode, and hardware.

[0061] Through millisecond-level collection and dynamic compression of monitoring data of multi-dimensional indicators (resource utilization, performance indicators, fault events).

[0062] Monitoring data is processed in layers. After the original event flows through the streaming filter and is classified, normal data is stored in a time series database (such as Prometheus), and abnormal events trigger a real-time analysis pipeline.

[0063] Visualization is achieved through topology heat maps and fault propagation maps, and open data interfaces support SQL queries and offline root cause analysis. Dynamic compression ensures the feasibility of high-frequency metric collection, heat maps provide global status awareness, and fault maps reveal the logic of abnormal transmission. Together, these three form a closed loop of observability.

[0064] As attached Figure 5As shown, the intelligent root cause analysis module achieves accurate fault diagnosis by integrating deep learning with causal reasoning. First, the LSTM prediction model uses a three-layer gated recurrent network structure and is trained based on historical monitoring data (such as time series metrics like IOPS, latency, and CPU utilization over the past 72 hours). It learns the dynamic patterns of the system under normal conditions and uses a sliding window (typically 5-10 minutes) to predict the range of metric values ​​for future time periods. It also introduces a dynamic threshold adjustment mechanism (such as the 3σ principle). When real-time data continuously deviates from the predicted interval and exceeds the set fault tolerance threshold, a multi-level alarm (warning, critical, and fatal) is triggered. Secondly, the causal inference engine constructs a multi-level Bayesian network whose nodes encompass multi-dimensional causal relationships, including hardware metrics (disk SMART data), network status (TCP retransmission rate), and application-layer features (number of blocked threads). Gibbs sampling is used to calculate conditional probabilities between events. Causal contributions are ranked based on real-time context (such as fault injection markers and resource contention). Ultimately, a root cause path diagram with a confidence score (0-1) (e.g., "Network packet loss (0.87) → Storage node timeout (0.79) → Client retry storm (0.92)") is generated, along with a root cause report and remediation recommendations. Finally, the intelligent root cause analysis module utilizes a dual closed-loop feedback mechanism. The LSTM prediction error dynamically modifies the causal network's prior probability distribution, while the causal analysis results in turn feed back into the LSTM feature weights, enabling the system to continuously optimize diagnostic accuracy during operation.

[0065] Specifically, the main steps to implement the intelligent root cause analysis module are as follows:

[0066] The LSTM prediction model uses a three-layer gated recurrent network structure and is trained based on historical monitoring data (such as time series indicators such as IOPS, latency, and CPU utilization over the past 72 hours). It learns the dynamic patterns of the system under normal conditions and predicts the range of indicator values ​​for future time periods using a sliding window (usually 5-10 minutes). It also introduces a dynamic threshold adjustment mechanism (such as the 3σ principle). When real-time data continues to deviate from the prediction interval and exceeds the set fault tolerance threshold, multi-level alarms (warning / critical / fatal) are triggered.

[0067] The causal reasoning engine constructs a multi-level Bayesian network whose nodes cover multi-dimensional causal relationships such as hardware indicators (disk SMART data), network status (TCP retransmission rate), and application layer characteristics (number of blocked threads). It calculates the conditional probability between events through Gibbs sampling and ranks the causal contributions based on real-time context (such as fault injection markers and resource competition status). It ultimately generates a root cause path diagram with a confidence score (0-1) (such as "network packet loss (0.87) → storage node timeout (0.79) → client retry storm (0.92)"), a root cause report, and repair suggestions.

[0068] The intelligent root cause analysis module adopts a dual closed-loop feedback mechanism. The LSTM prediction error will dynamically correct the prior probability distribution of the causal network, and the causal analysis results will feed back to the LSTM feature weight adjustment, allowing the system to continuously optimize diagnostic accuracy during continuous operation.

[0069] According to the full-link dynamic monitoring and anomaly simulation test system of the embodiment of the present invention, through cross-level composite load generation, multi-anomaly collaborative injection and intelligent root cause analysis, it systematically solves the long-standing problems in the field of distributed storage testing, such as insufficient dynamic scenario coverage, difficulty in reproducing complex faults, and inefficient root cause location. It significantly improves the system reliability verification capability and enhances the stability of the distributed storage system.

[0070] In order to implement the above embodiment, Figure 6 As shown, this embodiment also provides a test method for full-link dynamic monitoring and abnormal simulation, including:

[0071] S1 takes the set business scenario parameters and load characteristic parameters as input and outputs dynamically generated composite load data for the application layer, file system layer, and block device layer to simulate business pressure fluctuations;

[0072] S2, which takes composite load data and configurable anomaly parameters as input and outputs anomaly data of hardware, network, and data layers, as well as cross-layer fault propagation data;

[0073] S3 takes the composite load data and fault conduction data after anomaly injection as input and outputs request-level end-to-end tracing data collected by distributed probes, including millisecond-level performance indicators and contextual data;

[0074] S4 takes request-level tracing data, composite load data, anomaly data, and fault conduction data as input, and outputs fault root cause location results and repair suggestions obtained through LSTM time series prediction and Bayesian causal reasoning.

[0075] Furthermore, it also includes hierarchical processing of monitoring data. After the original event flow is classified by the streaming filter, normal data is stored in the time series database, abnormal events trigger the real-time analysis pipeline, and visualization is achieved through topological heat maps and fault propagation maps.

[0076] Specifically, the method of the present invention is mainly implemented by four major modules: a cross-level I / O load simulation engine module, a multi-level collaborative fault injection module, a full-link dynamic monitoring module, and an intelligent root cause analysis module. The cross-level I / O load simulation engine module simulates the multi-level I / O behavior of the distributed storage system, covers the composite load of the application layer, file system layer, and block device layer, and provides a load template library to support real-time adjustment of load parameters and simulate pressure fluctuations under dynamic resource scheduling. The multi-level collaborative fault injection module provides a preset fault library, synchronously injects faults at the hardware layer, network layer, and data layer, and defines cross-level triggering rules, such as simulating storage medium bad blocks, SSD wear anomalies, and memory page errors at the hardware layer, simulating network packet loss rate, delay jitter, and brain split scenarios at the network layer, and simulating silent data corruption at the data layer. The full-link dynamic monitoring module implants lightweight probes at key nodes in the I / O path, assigns a unique tracking ID to each request, and records the initiation, routing, processing, and completion of request lifecycle events. It uses a distributed clock synchronization protocol to align multi-node timestamps, with errors controlled at the microsecond level. This enables full-link tracking from client requests to persistent storage, provides real-time visualization of multi-dimensional monitoring data, and supports data export and offline analysis. The intelligent root cause analysis module's LSTM prediction model is trained based on historical monitoring data, learns the dynamic patterns of the system under normal conditions, and predicts indicator values ​​in future time windows. When real-time monitoring data deviates significantly from the predicted values, an abnormality alarm is triggered. The causal reasoning engine builds a multi-dimensional causal relationship model based on Bayesian networks and causal graph algorithms. Combined with real-time collected contextual information, it calculates the conditional probability and causal contribution between events, reversely derives the most likely fault transmission path, outputs a root cause analysis report, and provides repair suggestions.

[0077] According to the full-link dynamic monitoring and anomaly simulation testing method of the embodiment of the present invention, through cross-level composite load generation, multi-anomaly collaborative injection and intelligent root cause analysis, etc., it systematically solves the long-standing problems in the field of distributed storage testing, such as insufficient dynamic scenario coverage, difficulty in reproducing complex faults and inefficient root cause location. It significantly improves the system reliability verification capability and enhances the stability of the distributed storage system.

[0078] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0079] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

Claims

1. A full-link dynamic monitoring and abnormal simulation test system, characterized in that: include: The cross-layer I / O load simulation engine module takes the set business scenario parameters and load characteristic parameters as input, and outputs dynamically generated composite load data for the application layer, file system layer, and block device layer to simulate business pressure fluctuations; A multi-level collaborative fault injection module takes composite load data and configurable anomaly parameters as input, and outputs anomaly data of hardware, network, and data layers, as well as cross-level fault propagation data; The full-link dynamic monitoring module takes composite load data and fault conduction data after anomaly injection as input and outputs request-level end-to-end tracing data collected by distributed probes, including millisecond-level performance indicators and contextual data. The intelligent root cause analysis module takes request-level tracing data, composite load data, exception data, and fault conduction data as input, and outputs fault root cause location results and repair suggestions derived through LSTM time series prediction and Bayesian causal reasoning.

2. The full-link dynamic monitoring and abnormal simulation test system according to claim 1 is characterized in that: The cross-level I / O load simulation engine module generates an 8:2 read-write mixed load, so that the pressure peak reaches a preset IOPS value.

3. The full-link dynamic monitoring and abnormal simulation test system according to claim 1 is characterized in that: The multi-level coordinated fault injection module injects a disk delay of a preset time into a designated storage node.

4. The full-link dynamic monitoring and abnormal simulation test system according to claim 1 is characterized in that: The full-link dynamic monitoring module includes: a) Client-side probes dynamically intercept POSIX file operation APIs through API hook injection, capture request type, data location, and user context, and generate a globally unique tracking ID for each request; b) Network probe, deploying XDP program based on the Linux kernel eBPF mechanism, to capture key indicators of the TCP layer with zero copy; c) Storage probe, which hooks block device requests and file system operations in the form of a kernel module to monitor disk queue depth, IOPS and latency distribution; d) The tracking ID generated by the client probe is passed to the storage probe through the network probe, and the events of the four major stages of the request life cycle are recorded in real time. Combined with PTP clock synchronization, strict timing correlation of cross-node events is ensured.

5. The full-link dynamic monitoring and abnormal simulation test system according to claim 4 is characterized in that: The full-link dynamic monitoring module also includes hierarchical processing of monitoring data. After the original event flow is classified by the streaming filter, normal data is stored in the time series database, abnormal events trigger the real-time analysis pipeline, and visualization is achieved through topological heat maps and fault propagation maps.

6. The full-link dynamic monitoring and abnormal simulation test system according to claim 1 is characterized in that: The LSTM prediction model of the intelligent root cause analysis module adopts a three-layer gated recurrent network structure and is trained based on historical monitoring data to predict the range of indicator values ​​for future time periods. It also introduces a dynamic threshold adjustment mechanism. When real-time data continuously deviates from the prediction interval and exceeds the set fault tolerance threshold, a multi-level alarm is triggered.

7. The full-link dynamic monitoring and abnormal simulation test system according to claim 1 is characterized in that: The causal reasoning engine of the intelligent root cause analysis module constructs a multi-level Bayesian network whose nodes cover causal relationships in multiple dimensions, including hardware indicators, network status, and application layer features. It calculates the conditional probability between events through Gibbs sampling, ranks causal contributions based on real-time context, and generates a root cause path diagram with confidence score, a root cause report, and repair suggestions.

8. The full-link dynamic monitoring and abnormal simulation test system according to claim 1 is characterized in that: The intelligent root cause analysis module adopts a dual closed-loop feedback mechanism. The LSTM prediction error dynamically corrects the prior probability distribution of the causal network, and the causal analysis results feed back to the LSTM feature weight adjustment.

9. A full-link dynamic monitoring and abnormal simulation testing method, characterized in that: include: Taking the set business scenario parameters and load characteristic parameters as input, it outputs dynamically generated composite load data for the application layer, file system layer, and block device layer to simulate business pressure fluctuations; Taking composite load data and configurable abnormal parameters as input, it outputs abnormal data of hardware, network, and data layers, as well as cross-layer fault transmission data; Taking composite load data and fault conduction data after anomaly injection as input, it outputs request-level end-to-end tracing data collected by distributed probes, including millisecond-level performance indicators and context data. Taking request-level tracing data, composite load data, exception data, and fault conduction data as input, it outputs fault root cause location results and repair suggestions obtained through LSTM time series prediction and Bayesian causal reasoning.

10. The full-link dynamic monitoring and abnormal simulation testing method according to claim 9 is characterized in that: It also includes hierarchical processing of monitoring data. After the original event flows through the streaming filter and is classified, normal data is stored in the time series database, abnormal events trigger the real-time analysis pipeline, and visualization is achieved through topological heat maps and fault propagation maps.

Images