Distributed storage node fault detection system

Abstract

The invention proposes a distributed storage node fault detection system, and relates to the technical field of distributed storage, and the system comprises a data collection module which is configured to collect node operation state data; the feature extraction module is configured to perform feature calculation on the node operation state data to obtain a feature vector; the distributed anomaly detection module adopts a three-layer detection architecture and comprises a local layer, a region layer and a global layer; the fault diagnosis module is configured to receive the anomaly detection result of the global layer, perform fault classification and generate a fault diagnosis report; and the early warning decision module is configured to perform response strategy matching through a preset strategy template and a dynamic generation rule according to the fault diagnosis report, perform strategy priority ranking, and select a preset strategy with the highest priority as a final recommendation strategy. According to the method, the abnormal state of the storage node can be found in time, the fault type can be accurately identified, the fault root cause can be quickly positioned, and intelligent early warning decision support can be provided.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a distributed storage node fault detection system. Background Art

[0002] As the scale of distributed storage systems continues to expand, the number of nodes is growing exponentially, the system architecture is becoming increasingly complex, and the frequent occurrence of various types of failures has become an important factor affecting system reliability. At present, mainstream distributed storage systems generally adopt threshold-based fault detection methods, which set fixed thresholds for key indicators such as CPU usage, memory usage, and IO performance, and trigger alarms when monitoring indicators exceed the preset thresholds. This method is simple to implement and easy to deploy, and performs well in small-scale systems.

[0003] With the development of fault detection technology, researchers have proposed fault detection methods based on machine learning. This type of method collects system operation data, extracts multi-dimensional features such as time domain features and frequency domain features, and uses algorithms such as cluster analysis and anomaly detection to model the system state and identify abnormal states. Compared with traditional threshold detection methods, machine learning methods have stronger adaptive capabilities and can better handle complex fault scenarios.

[0004] However, existing fault detection methods still have some technical defects: first, the detection method with fixed thresholds is difficult to adapt to the dynamic changes of system load and is prone to false positives and false negatives; second, the fault detection of a single node cannot effectively identify complex faults caused by the coordination of multiple nodes; finally, the detection results are not interpretable enough, making it difficult to provide effective decision support for operation and maintenance personnel. These problems seriously restrict the reliability and maintainability of distributed storage systems. Summary of the invention

[0005] In view of this, the present invention proposes a distributed storage node fault detection system, which realizes comprehensive monitoring of the operating status of storage nodes, precise identification of abnormal behaviors, accurate diagnosis of fault types, and intelligent decision-making of early warning strategies by constructing a three-layer detection architecture with multi-module collaboration, thereby improving the reliability and maintainability of the distributed storage system and reducing operation and maintenance costs.

[0006] The technical solution of the present invention is achieved in this way:

[0007] The present invention provides a distributed storage node fault detection system, comprising:

[0008] A data collection module configured to collect node operation status data and perform preprocessing and standardization on the node operation status data;

[0009] A feature extraction module is configured to perform feature calculation on the node operation status data, perform feature dimension reduction through principal component analysis, and obtain a feature vector;

[0010] The distributed anomaly detection module adopts a three-layer detection architecture, including a local layer, a regional layer, and a global layer. The local layer is configured to perform real-time clustering analysis on feature vectors through a clustering algorithm, the regional layer is configured to perform fusion processing on the clustering analysis results of the local layer, and the global layer is configured to perform anomaly detection on the fused information;

[0011] A fault diagnosis module is configured to receive the anomaly detection results of the global layer, classify the faults, perform feature association analysis based on association rule mining, locate the root cause using a location algorithm, and generate a fault diagnosis report;

[0012] The early warning decision module is configured to classify confirmed faults based on multi-level early warning thresholds according to the fault diagnosis report, set the early warning suppression time window, match the response strategy through preset strategy templates and dynamically generated rules, and use a weighted scoring mechanism to prioritize the strategies, and select the preset strategy with the highest priority as the final recommended strategy.

[0013] On the basis of the above technical solution, preferably, the node operation status data is divided into three categories according to the monitoring indicator type, namely system-level indicator data, storage-level indicator data and network-level indicator data, wherein:

[0014] System-level indicator data includes CPU usage, memory usage, system load, and process status;

[0015] Storage-level metrics include IO performance, disk usage, read / write bandwidth, and disk health status;

[0016] Network-level metrics include network latency, packet loss rate, bandwidth usage, and connection status.

[0017] Based on the above technical solution, preferably, the process in which the data acquisition module performs preprocessing and standardization on the node operation status data includes:

[0018] The missing values ​​of the collected original data were processed, and the multivariate linear regression interpolation method was used to repair the data by combining temporal correlation and spatial correlation;

[0019] Adaptive median filtering is used to smooth outliers, and the window size is dynamically adjusted according to the degree of data fluctuation;

[0020] The Z-score method was used to standardize the data, and the skewness correction coefficient was introduced to correct the asymmetry of the data distribution.

[0021] On the basis of the above technical solution, preferably, when the feature extraction module performs feature calculation on the node operation status data, a multi-dimensional feature extraction method is adopted, by calculating the time domain features, frequency domain features, time-frequency features and high-order statistical features, and optimizing the extracted features; wherein the time domain features include first-order moments, second-order moments, third-order moments and fourth-order moments, the frequency domain features obtain the power spectral density and frequency band energy distribution through fast Fourier transform, the time-frequency features perform multi-scale decomposition based on the db4 wavelet basis, and the high-order statistical features extract nonlinear features through the ARIMA model.

[0022] Based on the above technical solution, preferably, the optimization processing of the extracted features includes:

[0023] Calculate the feature importance score SI(f i ), sort the features according to their importance scores, select the top N features to be retained, and the feature importance score SI(f i ) is calculated as follows:

[0024] SI(f i )=α 1 IG(f i )+α 2 GR(f i )+α 3 ·Relief(f i )

[0025] In the formula, f i represents the i-th feature, IG(f i ) is the information gain value, GR(f i ) is the information gain rate, Relief(f i ) is the score of the Relief algorithm, α 1 , α 2 , α 3 is the weight coefficient and satisfies α 1 +α 2 +α 3 =1;

[0026] The correlation between features is calculated based on the Pearson correlation coefficient, and highly correlated redundant features are eliminated by setting the correlation threshold θ:

[0027]

[0028] In the formula, R(f i ,f j ) is the feature f i With feature f j The feature correlation, Cov(f i ,f j ) is the feature f iWith feature f j The covariance of and The features f i With feature f j The standard deviation of

[0029] The retained features are subjected to principal component analysis and dimensionality reduction processing to obtain the reduced-dimensional feature vector F = {f′ 1 ,f′ 2 ,...,f′ k}, where k is the feature dimension after dimensionality reduction, and the k value is determined by calculating the cumulative variance contribution rate, so that the cumulative variance contribution rate of the first k principal components is not less than the preset threshold η.

[0030] Based on the above technical solution, preferably, in the distributed anomaly detection module, the execution process of the clustering algorithm is as follows:

[0031] A1. Normalize the feature vectors and map them to the interval [0,1]. Take each feature vector as a point and calculate the Euclidean distance d between the two points. ij ;

[0032] A2. Calculate the local density of each point:

[0033] ρ i,t =∑ j χ(d ij -d c )·τ(t)

[0034] In the formula, ρ i,t is the local density, χ is the indicator function, d c is the cutoff distance, τ(t)=e -λt is the time attenuation factor, λ is the attenuation coefficient, and t is the time interval;

[0035] A3. Calculate the minimum distance δ from each point to the high-density point i =min(d ij ), where j satisfies ρ j,t >ρ i,t ;

[0036] A4. Calculate the comprehensive score:

[0037] γ i =ρ i,t ·δ i ·(1+ε·En(i))

[0038] In the formula, γ i is the comprehensive score, En(i) is the local entropy value of point i, and ε is the entropy weight coefficient;

[0039] A5. Determine the local density threshold:

[0040] th den =μ·mean(ρ)+σ·std(ρ)

[0041] In the formula, th den represents the local density threshold, μ is the mean adjustment coefficient, σ is the standard deviation adjustment coefficient, mean(ρ) is the mean value of the local density, and std(ρ) is the standard deviation of the local density;

[0042] Calculate the distance anomaly threshold:

[0043] th dis =κ 1 ·d max +κ 2 ·std(D)

[0044] In the formula, th dis is the distance anomaly threshold, κ 1 is the distance reference coefficient, κ 2 is the fluctuation adjustment coefficient, d max is the maximum distance of the current cluster, std(D) is the standard deviation of the distance within the cluster;

[0045] A6. Based on the comprehensive score γ i Determine the cluster center according to the local density threshold th den Perform cluster division and use the distance anomaly threshold th dis Identify outliers;

[0046] A7. Generate local clustering results based on cluster division and outlier identification results, including cluster information {c,n c ,d max} and a set of outliers. The cluster information includes the cluster center c and the number of vectors in the cluster n c and the maximum distance d max .

[0047] Based on the above technical solution, preferably, the fusion processing of the cluster analysis results of the local layer includes:

[0048] B1. Receive the local clustering results of each node in the region, including the cluster center set C = {c 1 ,c 2 ,...,c m}、N={n 1 ,n 2 ,...,n m}、Maximum distance set And the set of outliers A = a 1 ,a 2 ,...,ak ;

[0049] B2. Calculate the inter-cluster similarity matrix S, for any two cluster centers c x and c y , and its similarity is calculated as follows:

[0050]

[0051] In the formula, s(c x ,c y ) represents the cluster center c x and c y similarity, σ is the Gaussian kernel parameter, determined by cross-validation; ‖c x -c y ‖ represents the Euclidean distance between the centers of two clusters, n x ,n y represents the number of vectors in the two clusters, Represents the maximum distance between two clusters;

[0052] B3. Construct a hierarchical clustering tree T and use the adaptive threshold method to merge clusters:

[0053] Merge th =β 1 ×(mean(S)+β 2 ×std(S)

[0054] In the formula, Merge th is the merging threshold, β 1 is the benchmark coefficient, β 2 is the variance adjustment coefficient, mean(S) is the mean of the similarity matrix S, and std(S) is the standard deviation of the similarity matrix S;

[0055] B4. Calculate the regional anomaly index RAI:

[0056] RAI(c)=w 1 ×DS(c)+w 2 ×IS(c)+w 3 ×TS(c)+w 4 ×AD(c)

[0057] Where DS(c) is the density score, which indicates the density of the cluster; IS(c) is the isolation score, which indicates the degree of isolation of the cluster from other clusters; TS(c) is the temporal correlation score, which indicates the temporal evolution characteristics of the cluster; AD(c) is the anomaly distribution feature, which reflects the distribution characteristics of the anomalies around the cluster; w 1 、w 2 、w 3 is the weight coefficient and satisfies w 1 +w2 +w 3 =1;

[0058] B5. Generate regional status descriptor RSD = {C * ,N * ,D * ,RAI * ,T *}, where C * is the set of cluster centers after merging, N * is the updated set of vector numbers within the cluster, D * is the updated maximum distance set, RAI * is the set of abnormal indices, T * Is the timestamp.

[0059] Based on the above technical solution, preferably, the process of performing anomaly detection at the global layer includes:

[0060] C1. Construct the global state matrix GSM = [RSD 1 ;RSD 2 ;...;RSD r ], where r is the total number of regions;

[0061] C2. Calculate the inter-regional correlation strength matrix RCM, and establish a regional correlation relationship diagram by analyzing the similarity of performance indicators, network topology distance, and similarity of abnormal patterns:

[0062] RCM(a,b)=∈ 1 ×PS(a,b)+∈ 2 ×TS(a,b)+∈ 3 ×AS(a,b)

[0063] Where RCM(a,b) represents the correlation strength matrix between region a and region b, PS(a,b) is the performance index similarity between region a and region b, TS(a,b) is the topological distance similarity between region a and region b, AS(a,b) is the anomaly pattern similarity between region a and region b, ∈ 1 ,∈ 2 ,∈ 3 is the weight coefficient, and ∈ 1 +∈ 2 +∈ 3 =1;

[0064] C3. Based on the sliding time window mechanism, analyze the anomaly pattern in the time and space dimensions, including tracking the evolution trend of the anomaly state, identifying the propagation path of the anomaly between regions, and predicting the possible anomaly diffusion direction;

[0065] C4, comprehensively evaluate the degree of abnormality, perform weighted fusion of local abnormal features, regional abnormal features and associated abnormal features to obtain the global abnormality score GAS;

[0066] C5, using an adaptive threshold method to determine anomalies, the threshold is dynamically adjusted according to the statistical characteristics of historical anomaly data;

[0067] C6. Output the standardized anomaly detection result set ADR, which includes the global anomaly score GAS, the location information AL of the abnormal area, the quantitative index AQ of the degree of anomaly, the propagation situation feature vector PF, the impact range prediction IR and the timestamp T.

[0068] On the basis of the above technical solution, preferably, the fault diagnosis module receives the abnormal detection result set ADR as input and processes it through the following subunits:

[0069] The fault classification unit builds a multi-level classification model based on the XGBoost decision tree algorithm, takes the global anomaly score GAS, the anomaly area feature AL and the propagation feature vector PF as feature input, and trains the model in combination with the historical fault sample library to achieve the probability prediction of the fault type and output the fault type vector FT containing the confidence level;

[0070] The feature association analysis unit uses the Apriori algorithm to mine the association rules between abnormal patterns and fault types based on the time series evolution data of abnormal regions and the propagation relationship between regions, constructs the fault feature map G, and establishes the feature association intensity matrix FCM for each fault type;

[0071] The root cause location unit takes the area where the anomaly first appears as the starting point, combines the propagation situation feature PF and network topology information, analyzes the fault propagation path through the random walk algorithm, uses the PageRank algorithm to evaluate the importance of the node, and outputs the root cause probability ranking list RC and the fault impact range assessment report IR;

[0072] The diagnostic report generation unit integrates the fault type FT, feature association map G and root cause analysis result RC to generate a standardized diagnostic report DR including fault location, impact assessment and repair suggestions, and provides visual fault diagnosis results.

[0073] On the basis of the above technical solution, preferably, the early warning decision module is processed by the following sub-units based on the anomaly detection result set ADR and the fault diagnosis report DR:

[0074] The fault level assessment unit comprehensively analyzes the global abnormality score GAS, fault type probability distribution FT and impact range prediction IR, establishes a multi-dimensional fault severity assessment system, and dynamically generates the fault level assessment result SL;

[0075] The warning control unit implements a warning trigger mechanism based on a time window according to the fault level SL and the propagation situation feature PF, and determines the alarm sequence through the warning priority score;

[0076] The strategy matching unit selects a matching processing solution from the preset strategy library based on the current fault type FT and the root cause analysis result RC;

[0077] The decision optimization unit performs multi-objective optimization on candidate processing strategies, comprehensively considers factors such as fault severity SL, impact range IR, processing cost and risk, and outputs the optimal strategy execution plan and execution sequence.

[0078] Compared with the prior art, the present invention has the following beneficial effects:

[0079] (1) The present invention constructs five collaborative functional modules, namely data collection, feature extraction, distributed anomaly detection, fault diagnosis and early warning decision-making, adopts a three-layer detection architecture to implement anomaly detection, and combines machine learning algorithms to classify and locate the root cause of faults, thus realizing the full process automation from data collection, anomaly detection to fault diagnosis. The system can timely detect the abnormal state of storage nodes, accurately identify the fault type, quickly locate the root cause of the fault, and provide intelligent early warning decision support, thereby improving the reliability and maintainability of the distributed storage system;

[0080] (2) In the feature extraction process, the present invention realizes multi-dimensional feature expression of node operation status by calculating time domain features, frequency domain features, time-frequency features and high-order statistical features, and combining feature importance scoring, correlation analysis and principal component analysis for feature optimization. This method not only improves the feature expression ability, but also reduces data redundancy and computational complexity through dimensionality reduction processing;

[0081] (3) The present invention adopts a three-layer detection architecture of local layer, regional layer, and global layer, and realizes step-by-step analysis from single node to global through a hierarchical anomaly detection mechanism. This architecture can effectively reduce data transmission overhead and achieve reasonable distribution of computing load. At the same time, it improves the accuracy and scalability of anomaly detection through multi-level information fusion, providing efficient and reliable technical support for fault detection of large-scale distributed storage systems;

[0082] (4) The local layer uses an improved clustering algorithm to introduce local density calculation, time decay factor and local entropy value to achieve real-time clustering analysis of node status. This method adaptively selects cluster centers through a comprehensive scoring mechanism and uses dynamic thresholds for cluster division. This not only improves clustering accuracy, but also effectively adapts to the dynamic changes of node status, providing high-quality local analysis results for the regional layer.

[0083] (5) The regional layer merges clusters by calculating the inter-cluster similarity matrix and adaptive threshold, and introduces the regional anomaly index RAI to comprehensively evaluate the density, isolation and time correlation of clusters. This method realizes the effective identification and feature extraction of abnormal patterns in the region, and provides structured regional status information for the global layer by generating a standardized regional status descriptor RSD, effectively reducing the amount of data transmission;

[0084] (6) The global layer constructs a regional correlation graph, implements spatiotemporal analysis of anomalies based on a sliding time window mechanism, and uses multi-dimensional feature fusion to obtain a global anomaly score GAS. This method can not only accurately identify complex cross-regional faults, but also predict the propagation path and impact range of anomalies. By outputting a standardized anomaly detection result set ADR, it provides comprehensive anomaly feature information for the fault diagnosis module. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0086] Figure 1 It is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0087] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] like Figure 1 As shown, the present invention provides a distributed storage node fault detection system, comprising:

[0089] A data collection module configured to collect node operation status data and perform preprocessing and standardization on the node operation status data;

[0090] A feature extraction module is configured to perform feature calculation on the node operation status data, perform feature dimension reduction through principal component analysis, and obtain a feature vector;

[0091] The distributed anomaly detection module adopts a three-layer detection architecture, including a local layer, a regional layer, and a global layer. The local layer is configured to perform real-time clustering analysis on feature vectors through a clustering algorithm, the regional layer is configured to perform fusion processing on the clustering analysis results of the local layer, and the global layer is configured to perform anomaly detection on the fused information;

[0092] A fault diagnosis module is configured to receive the anomaly detection results of the global layer, classify the faults, perform feature association analysis based on association rule mining, locate the root cause using a location algorithm, and generate a fault diagnosis report;

[0093] The early warning decision module is configured to classify confirmed faults based on multi-level early warning thresholds according to the fault diagnosis report, set the early warning suppression time window, match the response strategy through preset strategy templates and dynamically generated rules, and use a weighted scoring mechanism to prioritize the strategies, and select the preset strategy with the highest priority as the final recommended strategy.

[0094] Specifically, in one embodiment of the present invention, the data acquisition module, as a basic component of the distributed storage node fault detection system, is mainly responsible for collecting node operation status data and preprocessing and standardizing the collected data. The node operation status data collected by this module can be divided into three categories: system-level indicator data, storage-level indicator data, and network-level indicator data. Among them, the system-level indicator data includes CPU utilization, memory utilization, system load, and process status; the storage-level indicator data includes IO performance, disk utilization, read and write bandwidth, and disk health status; the network-level indicator data includes network delay, packet loss rate, bandwidth utilization, and connection status. These indicator data comprehensively cover the operation status characteristics of the storage node, providing a reliable data basis for subsequent anomaly detection and fault diagnosis.

[0095] After collecting the node operation status data, the data acquisition module needs to preprocess and standardize the raw data to improve data quality and eliminate noise interference. The specific implementation process includes three steps: missing value processing, outlier smoothing, and data standardization. The process is as follows:

[0096] In order to solve the problem of missing data in the collection process, the multivariate linear regression interpolation method is used. This method fully considers the temporal correlation and spatial correlation of the data and repairs the missing data by establishing a regression model. The specific implementation steps are as follows:

[0097] First, a multivariate linear regression model is established based on historical data. For the missing indicator y, k indicators with strong correlation with it are selected as independent variables x. 1 ,x 2 ,...,x k , establish the regression equation:

[0098] y=β 0 +β 1 x 1 +β 2 x 2 +...+β k x k +ε

[0099] Among them, β 0 is the intercept term, β 1 ,β 2 ,...,β k is the regression coefficient, and ε is the random error term.

[0100] Estimate the regression coefficients using the method of least squares:

[0101] β=(X T X) -1 X T Y

[0102] Among them, X is the independent variable matrix and Y is the dependent variable vector.

[0103] Substitute the current values ​​of the relevant indicators into the regression equation to obtain the predicted values ​​of the missing values:

[0104]

[0105] Combined with the confidence interval, correction is made to ensure the rationality of the interpolation result. The confidence interval of the predicted value is defined as:

[0106]

[0107] where ∈ is the error bound calculated from the t-distribution:

[0108]

[0109] Among them, s is the regression standard deviation, t α / 2,n-k-1 is the critical value of the t distribution, which is usually obtained from the t distribution table and depends on the confidence level α and the degrees of freedom nk-1. 0 is the independent variable vector of the points to be predicted.

[0110] Adaptive median filtering is used to smooth outliers, and the window size is dynamically adjusted according to the degree of data fluctuation:

[0111] 1. Initialize the filter window size S min and the maximum window size S max :

[0112] S min : Initial window size, set to 3; S max : Maximum window size, determined by the data sampling frequency.

[0113] 2. Calculate the local volatility of the data in the current window:

[0114]

[0115] Where n is the number of data points in the window, x i is the data point in the window, is the mean of the data in the window.

[0116] 3. Dynamically adjust the window size according to the volatility:

[0117]

[0118] Among them, k is the adjustment coefficient, which is used to control the growth rate of the window size; Indicates a floor operation.

[0119] 4. Perform median filtering on the data in the current window: calculate the median med of the data in the window; if the deviation between the current value and the median exceeds the preset threshold θ, replace it with the median: if |x-med|>θ, then x=med.

[0120] The Z-score method is used to standardize the data, and the skewness correction coefficient is introduced to correct the asymmetry of the data distribution. The specific implementation steps are as follows:

[0121] Calculate the skewness coefficient of the data:

[0122]

[0123] Where n is the number of samples, x i is the original data, is the sample mean.

[0124] Calculate the correction factor based on the skewness coefficient:

[0125] α=1 / (1+|SK|)

[0126] Perform Z-score normalization:

[0127] z=α×(x-μ) / σ

[0128] Among them, μ is the sample mean and σ is the sample standard deviation.

[0129] Verify the standardized results to ensure that: the mean of the standardized data is close to 0; the standard deviation is close to 1; and the skewness of the data distribution is effectively corrected.

[0130] Through the above three steps, the data acquisition module can effectively improve the data quality and provide a reliable data foundation for subsequent feature extraction and anomaly detection.

[0131] Specifically, in one embodiment of the present invention, the feature extraction module is mainly responsible for performing feature calculation and optimization processing on the node operation status data. It adopts a multi-dimensional feature extraction method, calculates time domain features, frequency domain features, time-frequency features and high-order statistical features, and optimizes the extracted features. Finally, it performs dimensionality reduction through principal component analysis to obtain the final feature vector.

[0132] The time domain features include first-order moment, second-order moment, third-order moment and fourth-order moment. The frequency domain features obtain power spectral density and frequency band energy distribution through fast Fourier transform. The time-frequency features are decomposed at multiple scales based on db4 wavelet basis. The high-order statistical features extract nonlinear features through ARIMA model.

[0133] In this embodiment, the feature optimization process is as follows:

[0134] Calculate the feature importance score SI(f i ), sort the features according to their importance scores, select the top N features to be retained, and the feature importance score SI(f i ) is calculated as follows:

[0135] SI(f i )=α 1 IG(f i )+α 2 GR(f i )+α 3 ·Relief(f i )

[0136] In the formula, f i represents the i-th feature, IG(f i ) is the information gain value, GR(f i ) is the information gain rate, Relief(f i ) is the score of the Relief algorithm, α 1 , α 2 , α 3 is the weight coefficient and satisfies α 1 +α 2 +α 3 =1.

[0137] Specifically, the Relief algorithm is a feature selection algorithm used to evaluate the importance of features. It evaluates the importance of features by analyzing the ability of each feature to distinguish neighboring samples. The algorithm flow is:

[0138] For each feature f, initialize the weight W[f] = 0

[0139] Repeat m times:

[0140] Randomly select a sample R

[0141] Find the k nearest similar samples of R (called Nearest Hits, H for short)

[0142] Find the k nearest samples of different classes of R (called Nearest Misses, M for short)

[0143] For each feature f:

[0144] W[f]=W[f]-Σ(diff(f,R,H) / (m×k))+Σ(diff(f,R,M) / (m×k))

[0145] Among them, for numerical features, diff(f,R,H)=|value(f,R)-value(f,H)| / (max(f)-min(f)); for discrete features, diff(f,R,H)=0,if value(f,R)=value(f,H),diff(f,R,H)=1,ifvalue(f,R)≠value(f,H).

[0146] The correlation between features is calculated based on the Pearson correlation coefficient, and highly correlated redundant features are eliminated by setting the correlation threshold θ:

[0147]

[0148] In the formula, R(f i ,f j ) is the feature f i With feature f j The feature correlation, Cov(f i ,f j ) is the feature f i With feature f j The covariance of and The features f i With feature f j The standard deviation of .

[0149] Perform principal component analysis on the retained features to reduce their dimensionality: calculate the feature covariance matrix, solve the eigenvalues ​​and eigenvectors, sort by eigenvalues, select the main components, calculate the cumulative variance contribution rate, determine the feature dimension k after dimensionality reduction, and generate the eigenvector F after dimensionality reduction. 1 ,f′ 2 ,...,f′ k}, where the criterion for determining the dimension k of the dimension reduction is: the cumulative variance contribution rate of the first k principal components is not less than a preset threshold η, and η is specifically set to 85% or 90%.

[0150] Specifically, in one embodiment of the present invention, a three-layer detection architecture is used to detect node anomalies. From a functional dimension, the local layer is responsible for real-time monitoring at the node level, the regional layer performs collaborative analysis within the region, and the global layer completes system-level anomaly detection. From the perspective of data granularity, the data processed by each layer is abstracted level by level: the local layer processes the original monitoring data, the regional layer processes the clustered feature data, and the global layer processes the regional state descriptor. This hierarchical design provides good system scalability and supports horizontal expansion and independent optimization and upgrading of the system. At the same time, by dispersing the computing pressure, a balanced distribution of the computing load is achieved, and the system response speed is improved. In addition, the multi-level cross-validation mechanism effectively improves the detection accuracy and reduces the false alarm rate.

[0151] Specifically, the execution process of the entire distributed anomaly detection module is a progressive process: the local layer performs real-time clustering analysis and generates results; the regional layer collects and fuses the clustering results of nodes in the region and calculates the regional anomaly index; the global layer constructs a state matrix, performs spatiotemporal analysis, and finally generates anomaly detection results. The data flow between layers is clear, ensuring the effective transmission and processing of information.

[0152] First, as the first layer of distributed anomaly detection, the local layer is mainly responsible for real-time anomaly detection of a single node, and quickly identifies local anomaly patterns through clustering analysis. Its characteristics are: it can quickly respond to changes in node status; it uses a lightweight algorithm that does not affect the normal business of the node; it can dynamically adjust parameters according to data characteristics. Among them, the clustering algorithm execution process is:

[0153] A1. Normalize the feature vectors and map them to the interval [0,1]. Take each feature vector as a point and calculate the Euclidean distance d between the two points. ij ;

[0154] A2. Calculate the local density of each point:

[0155] ρ i,t =∑ j χ(d ij -d c )·τ(t)

[0156] In the formula, ρ i,t is the local density, χ is the indicator function, d c is the cutoff distance, τ(t)=e -λt is the time attenuation factor, λ is the attenuation coefficient, and t is the time interval; ij -d c <0, χ=1, otherwise χ=0; d c The value may be the 2% quantile of all point-to-point distances of the data point clock. In this embodiment, the value range of λ is 0.1-0.5.

[0157] A3. Calculate the minimum distance δ from each point to the high-density point i =min(d ij ), where j satisfies ρ j,t >ρ i,t ;

[0158] A4. Calculate the comprehensive score:

[0159] γ i =ρ i,t ·ε i ·(1+ε·En(i))

[0160] In the formula, γ i is the comprehensive score, En(i) is the local entropy value of point i, and ε is the entropy weight coefficient; where En(i) = -∑(p j ×log(p j )), where p j is the proportion of the jth type of points in the neighborhood of point i, and the local entropy value represents the uncertainty of data distribution in the local area. In this embodiment, the value range of ε is 0.1-0.3.

[0161] A5. Determine the local density threshold:

[0162] th den =μ·mean(ρ)+σ·std(ρ)

[0163] In the formula, th den represents the local density threshold, μ is the mean adjustment coefficient, σ is the standard deviation adjustment coefficient, mean(ρ) is the mean value of the local density, and std(ρ) is the standard deviation of the local density;

[0164] Calculate the distance anomaly threshold:

[0165] th dis =κ 1 ·d max +κ 2 ·std(D)

[0166] In the formula, th dis is the distance anomaly threshold, κ 1 is the distance reference coefficient, κ 2 is the fluctuation adjustment coefficient, d max is the maximum distance of the current cluster, std(D) is the standard deviation of the distance within the cluster;

[0167] In this embodiment, μ is in the range of 1.0-1.5, σ is in the range of 2.0-3.0, and κ is 1 The value range is 0.6-0.8, κ 2 The value range is 1.5-2.0.

[0168] A6. Based on the comprehensive score γ i Determine the cluster center according to the local density threshold th den Perform cluster division and use the distance anomaly threshold th dis Identify outliers;

[0169] A7. Generate local clustering results based on cluster division and outlier identification results, including cluster information {c,n c ,d max} and a set of outliers. The cluster information includes the cluster center c and the number of vectors in the cluster n c and the maximum distance d max .

[0170] As the middle layer of distributed anomaly detection, the regional layer is mainly responsible for fusing and analyzing the clustering results of multiple nodes in the same region to identify regional-level anomaly patterns. Its features are: integrating the detection results of multiple nodes; filtering local noise through result fusion; and supporting dynamic addition and removal of nodes. The fusion processing flow is as follows:

[0171] B1. Receive the local clustering results of each node in the region, including the cluster center set C = {c 1 ,c 2 ,...,c m}、N={n 1 ,n 2 ,...,n m}、Maximum distance set And the set of outliers A = a 1 ,a 2 ,...,a k ;

[0172] B2. Calculate the inter-cluster similarity matrix S, for any two cluster centers c x and c y , and its similarity is calculated as follows:

[0173]

[0174] In the formula, s(c x ,c y ) represents the cluster center c x and c y similarity, σ is the Gaussian kernel parameter, determined by cross-validation; ‖c x -c y ‖ represents the Euclidean distance between the centers of two clusters, n x ,n y represents the number of vectors in the two clusters, Represents the maximum distance between two clusters;

[0175] B3. Construct a hierarchical clustering tree T and use the adaptive threshold method to merge clusters:

[0176] Merge th =β 1 ×(mean(S)+β 2 ×std(S)

[0177] In the formula, Merge th is the merging threshold, β 1 is the benchmark coefficient, β 2 is the variance adjustment coefficient, mean(S) is the mean of the similarity matrix S, and std(S) is the standard deviation of the similarity matrix S; in this embodiment, β 1 The value range is 0.8-1.2, β 2 The value range is 1.5-2.5.

[0178] When merging, it is iterated from bottom to top, each time selecting the cluster pair with the highest similarity for merging, and updating the merged cluster features: recalculating the cluster center and the maximum distance.

[0179] B4. Calculate the regional anomaly index RAI:

[0180] RAI(c)=w 1 ×DS(c)+w 2 ×IS(c)+w 3 ×TS(c)+w 4 ×AD(c)

[0181] Where DS(c) is the density score, which indicates the density of the cluster; IS(c) is the isolation score, which indicates the degree of isolation of the cluster from other clusters; TS(c) is the temporal correlation score, which indicates the temporal evolution characteristics of the cluster; AD(c) is the anomaly distribution feature, which reflects the distribution characteristics of the anomalies around the cluster; w 1 、w 2 、w 3 is the weight coefficient and satisfies w 1 +w 2 +w 3 =1;

[0182] In this embodiment, n(c) represents the number of points in the cluster, d max (c) is the maximum distance within the cluster; dis(c,c j ) represents cluster c and other clusters c j The center distance between them, min(dis(c,c j )) represents the distance from cluster c to the nearest cluster, mean(dis(c,all other-c)) represents the average distance from cluster c to all other clusters; TS(c) = e -λ*Δt *sta(c), where λ is the time decay coefficient, Δt is the time interval from the current moment to the first appearance of cluster c, and sta(c) is the stability measure of cluster c, which is calculated based on the changes of cluster c in the historical window period; AD(c) = count(r) / total(r), where r represents the influence radius of cluster c, which is d max 1.5 times of, count(r) represents the number of outliers within the impact radius r, and total(r) represents the total number of points within the impact radius r.

[0183] B5. Generate regional status descriptor RSD = {C * ,N * ,D * ,RAI * ,T *}, where C * is the set of cluster centers after merging, N * is the updated set of vector numbers within the cluster, D * is the updated maximum distance set, RAI * is the set of abnormal indices, T * Is the timestamp.

[0184] As the highest layer of distributed anomaly detection, the global layer is responsible for integrating the detection results of all regions and performing system-level anomaly analysis. Its characteristics are: it can discover cross-regional anomaly patterns; it can predict the propagation trend of anomalies; and it provides a basis for system-level response. The anomaly detection process is as follows:

[0185] C1. Construct the global state matrix GSM = [RSD 1 ;RSD 2 ;...;RSD r ], where r is the total number of regions;

[0186] C2. Calculate the inter-regional correlation strength matrix RCM, and establish a regional correlation relationship diagram by analyzing the similarity of performance indicators, network topology distance, and similarity of abnormal patterns:

[0187] RCM(a,b)=∈ 1 ×PS(a,b)+∈ 2 ×TS(a,b)+∈ 3 ×AS(a,b)

[0188] Where RCM(a,b) represents the correlation strength matrix between region a and region b, PS(a,b) is the performance index similarity between region a and region b, TS(a,b) is the topological distance similarity between region a and region b, AS(a,b) is the anomaly pattern similarity between region a and region b, ∈ 1 ,∈ 2 ,∈ 3 is the weight coefficient, and ∈ 1 +∈ 2 +∈ 3 =1;

[0189] In this embodiment, PS(a,b)=1-‖P a -P b ‖ / max(‖P a ‖,‖P b ‖), P a , P b is the performance index vector of regions a and b, ‖P a -P b ‖ is the Euclidean distance between two vectors, ‖P a ‖,‖P b ‖ is the norm of the two vectors. hop(a,b) is the minimum number of hops between areas a and b, max hop is the maximum number of hops in the network. AS(a,b)=Jaccard(A a ,A b )*SIM(Y a ,Y b ), A a ,A b is the set of outliers in regions a and b, Jaccard(A a ,A b ) is the Jaccard similarity of two outlier sets, Y a ,Y b is the abnormal pattern feature vector of regions a and b, SIM(Y a ,Y b ) is the cosine similarity of the abnormal pattern feature vector.

[0190] C3. Based on the sliding time window mechanism, analyze the anomaly pattern in the time and space dimensions, including tracking the evolution trend of the anomaly state, identifying the propagation path of the anomaly between regions, and predicting the possible anomaly diffusion direction; specifically, use the sliding time window mechanism to perform time series analysis on the anomaly index RAI of each region, calculate the trend change rate and fluctuation characteristics. Based on the regional correlation intensity matrix RCM, analyze the diffusion path of the anomaly state between different regions and construct a propagation map. Combine the historical propagation path and the current anomaly situation to predict the possible anomaly diffusion direction and impact range.

[0191] C4, comprehensively evaluate the degree of abnormality, perform weighted fusion of local abnormal features, regional abnormal features and associated abnormal features to obtain the global abnormality score GAS;

[0192] GAS=ω 1 *LAF+ω 2 *RAF+ω 3 *CAF

[0193] Among them, LAF is the local abnormal feature, LAF = ∑ (RAI * weight local ), weight local is the weight based on local density and centrality; RAF is the regional anomaly feature, RAF = En(RCM) * max(RAI), En(RCM) is the local entropy value of RCM; CAF is the associated anomaly feature, CAF = prop score *Im score , prop score is a score based on propagation speed and direction consistency, Im score = Number of affected nodes / total number of nodes.

[0194] C5. Adopting an adaptive threshold method to determine anomalies, the threshold is dynamically adjusted according to the statistical characteristics of historical anomaly data; specifically including:

[0195] Calculate the mean and standard deviation of historical GAS scores; consider time trends and seasonal factors; introduce an adaptive adjustment coefficient to dynamically adjust according to the current abnormal proportion; set minimum and maximum threshold boundaries to ensure that the threshold fluctuates within a reasonable range.

[0196] C6. Output the standardized anomaly detection result set ADR, which includes the global anomaly score GAS, the location information AL of the abnormal area, the quantitative index AQ of the degree of anomaly, the propagation situation feature vector PF, the impact range prediction IR and the timestamp T.

[0197] Specifically, in one embodiment of the present invention, the fault diagnosis module receives anomaly detection result set ADR as input, and realizes accurate diagnosis and location of faults through coordinated processing of four sub-units including fault classification, feature association analysis, root cause location and diagnostic report generation.

[0198] The fault classification unit builds a multi-level classification model based on the XGBoost decision tree algorithm, takes the global anomaly score GAS, the anomaly region feature AL, and the propagation feature vector PF as feature input, and combines the historical fault sample library for model training to achieve probability prediction of fault types and output the fault type vector FT containing confidence. Specifically, the XGBoost decision tree algorithm is used to build a three-layer classification model. The first layer: distinguishes the major fault categories (hardware fault, software fault, network fault), the second layer: subdivides the fault subcategories (such as disk fault, memory fault, network congestion, etc.), and the third layer: identifies the specific fault type. Output the fault type probability vector FT, which contains the confidence score of each type of fault and provides the probability distribution of the fault type.

[0199] Based on the time series evolution data of the abnormal area and the propagation relationship between regions, the feature association analysis unit uses the Apriori algorithm to mine the association rules between abnormal patterns and fault types, construct the fault feature map G, and establish the feature association strength matrix FCM for each fault type; specifically, extract the time series change characteristics of the performance indicators in the abnormal area, analyze the correlation and causal relationship between the indicators, and identify the change pattern of the key performance indicators. Use the Apriori algorithm to mine association rules, set the minimum support and confidence thresholds, extract high-frequency abnormal pattern combinations, and establish a mapping relationship between abnormal patterns and fault types. Construct the fault feature association graph G, where nodes represent performance indicators or abnormal events, edges represent the association relationship between indicators, and edge weights reflect the association strength. Construct the fault feature association strength matrix FCM and calculate the direct association strength and indirect association strength.

[0200] The root cause location unit takes the area where the anomaly first appears as the starting point, combines the propagation situation feature PF and network topology information, analyzes the fault propagation path through the random walk algorithm, uses the PageRank algorithm to evaluate the importance of the node, and outputs the root cause probability ranking list RC and the fault impact range assessment report IR;

[0201] The diagnostic report generation unit integrates the fault type FT, feature association map G and root cause analysis result RC to generate a standardized diagnostic report DR containing fault location, impact assessment and repair suggestions, and provides visual fault diagnosis results. The report content may include:

[0202] Fault overview: including fault type, severity and impact scope;

[0203] Root cause analysis: including root cause location results and credibility assessment;

[0204] Propagation analysis: including fault propagation path and diffusion trend;

[0205] Impact assessment: including performance impact and potential risk analysis;

[0206] Repair suggestions: including treatment plans and expected results.

[0207] Specifically, in one embodiment of the present invention, the early warning decision module receives the anomaly detection result set ADR and the fault diagnosis report DR as input, and realizes intelligent decision-making of graded early warning and processing strategy for faults through the coordinated processing of four sub-units: fault level assessment, early warning control, strategy matching and decision optimization.

[0208] The fault level assessment unit comprehensively analyzes the global anomaly score GAS, fault type probability distribution FT and impact range prediction IR, establishes a multi-dimensional fault severity assessment system, and dynamically generates a fault level assessment result SL; specifically, based on the assessment results, the faults are divided into four levels: emergency level, severe level, warning level and prompt level.

[0209] The early warning control unit implements an early warning trigger mechanism based on the time window according to the fault level SL and the propagation situation feature PF, and determines the alarm order through the early warning priority score; for example: emergency level: no suppression time; severe level: 5-minute suppression window; warning level: 15-minute suppression window; prompt level: 30-minute suppression window. Faults of the same type are merged within the time window, faults of different levels are triggered according to the highest level, and unified early warning is issued after root cause analysis of related faults.

[0210] The strategy matching unit selects a matching processing solution from the preset strategy library based on the current fault type FT and the root cause analysis result RC;

[0211] The decision optimization unit performs multi-objective optimization on candidate processing strategies, comprehensively considers factors such as fault severity SL, impact range IR, processing cost and risk, and outputs the optimal strategy execution plan and execution sequence.

[0212] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed storage node fault detection system, characterized in that: include: A data collection module configured to collect node operation status data and perform preprocessing and standardization on the node operation status data; A feature extraction module is configured to perform feature calculation on the node operation status data, perform feature dimension reduction through principal component analysis, and obtain a feature vector; The distributed anomaly detection module adopts a three-layer detection architecture, including a local layer, a regional layer, and a global layer. The local layer is configured to perform real-time clustering analysis on feature vectors through a clustering algorithm, the regional layer is configured to perform fusion processing on the clustering analysis results of the local layer, and the global layer is configured to perform anomaly detection on the fused information; A fault diagnosis module is configured to receive the anomaly detection results of the global layer, classify the faults, perform feature association analysis based on association rule mining, locate the root cause using a location algorithm, and generate a fault diagnosis report; The early warning decision module is configured to classify confirmed faults based on multi-level early warning thresholds according to the fault diagnosis report, set the early warning suppression time window, match the response strategy through preset strategy templates and dynamically generated rules, and use a weighted scoring mechanism to prioritize the strategies, and select the preset strategy with the highest priority as the final recommended strategy.

2. A distributed storage node fault detection system as claimed in claim 1, characterized in that: Node operation status data is divided into three categories according to the monitoring indicator type, namely system-level indicator data, storage-level indicator data, and network-level indicator data, among which: System-level indicator data includes CPU usage, memory usage, system load, and process status; Storage-level metrics include IO performance, disk usage, read / write bandwidth, and disk health status; Network-level metrics include network latency, packet loss rate, bandwidth usage, and connection status.

3. A distributed storage node fault detection system as claimed in claim 2, characterized in that: The data acquisition module performs preprocessing and standardization on the node operation status data, including: The missing values ​​of the collected original data were processed, and the multivariate linear regression interpolation method was used to repair the data by combining temporal correlation and spatial correlation; Adaptive median filtering is used to smooth outliers, and the window size is dynamically adjusted according to the degree of data fluctuation; The Z-score method was used to standardize the data, and the skewness correction coefficient was introduced to correct the asymmetry of the data distribution.

4. A distributed storage node fault detection system as claimed in claim 2, characterized in that: When the feature extraction module calculates the features of the node operation status data, it uses a multi-dimensional feature extraction method to calculate the time domain features, frequency domain features, time-frequency features and high-order statistical features, and optimizes the extracted features; The time domain features include first-order moment, second-order moment, third-order moment and fourth-order moment. The frequency domain features obtain power spectral density and frequency band energy distribution through fast Fourier transform. The time-frequency features are decomposed at multiple scales based on db4 wavelet basis. The high-order statistical features extract nonlinear features through ARIMA model.

5. A distributed storage node fault detection system as claimed in claim 4, characterized in that: The optimization process of the extracted features includes: Calculate the feature importance score SI(f i ), sort the features according to their importance scores, select the top N features to be retained, and the feature importance score SI(f i ) is calculated as follows: SI(f i )=α1·IG(f i )+α2·GR(f i )+α3·Relief(f i ) In the formula, f i represents the i-th feature, IG(f i ) is the information gain value, GR(f i ) is the information gain rate, Relief(f i ) is the score of the Relief algorithm, α1, α2, α3 are weight coefficients and satisfy α1+α2+α3=1; The correlation between features is calculated based on the Pearson correlation coefficient, and highly correlated redundant features are eliminated by setting the correlation threshold θ: In the formula, R(fi,f j ) is the feature fi and feature f j The feature correlation, Cov(fi,f j ) is the feature fi and feature f j The covariance of fi and σ fj are feature fi and feature f respectively j The standard deviation of The retained features are subjected to principal component analysis and dimensionality reduction processing to obtain the reduced feature vector F = {f′1, f′2, ..., f′k}, where k is the feature dimension after dimensionality reduction. The k value is determined by calculating the cumulative variance contribution rate, so that the cumulative variance contribution rate of the first k principal components is not less than the preset threshold η.

6. A distributed storage node fault detection system as claimed in claim 2, characterized in that: In the distributed anomaly detection module, the execution process of the clustering algorithm is as follows: A1. Normalize the feature vectors and map them to the interval [0,1]. Take each feature vector as a point and calculate the Euclidean distance d between the two points. ij ; A2. Calculate the local density of each point: r i,t ∑ j x(d ij -d c )·τ(t) In the formula, ρ i,t is the local density, χ is the indicator function, d c is the cutoff distance, τ(t)=e -λt is the time attenuation factor, λ is the attenuation coefficient, and t is the time interval; A3. Calculate the minimum distance δ from each point to the high-density point i =min(d ij ), where j satisfies ρ j,t >ρ i,t ; A4. Calculate the comprehensive score: γ i =ρ i,t ·δ i ·(1+ε·En(i)) In the formula, γ i is the comprehensive score, En(i) is the local entropy value of point i, and ε is the entropy weight coefficient; A5. Determine the local density threshold: th den =μ·mean(ρ)+σ·std(ρ) In the formula, th den represents the local density threshold, μ is the mean adjustment coefficient, σ is the standard deviation adjustment coefficient, mean(ρ) is the mean value of the local density, and std(ρ) is the standard deviation of the local density; Calculate the distance anomaly threshold: <h2 style=";text-align:left;direction:ltr">th<h2 style=";text-align:left;direction:ltr"> dis <h2 style=";text-align:left;direction:ltr"> =k1·d<h2 style=";text-align:left;direction:ltr"> max <h2 style=";text-align:left;direction:ltr"> +κ2·std(D) In the formula, th dis is the distance anomaly threshold, κ1 is the distance benchmark coefficient, k2 is the fluctuation adjustment coefficient, d max is the maximum distance of the current cluster, std(D) is the standard deviation of the distance within the cluster; A6. Based on the comprehensive score γ i Determine the cluster center according to the local density threshold th den Perform cluster division and use the distance anomaly threshold th dis Identify outliers; A7. Generate local clustering results based on cluster division and outlier identification results, including cluster information {c,n c ,d max } and a set of outliers. The cluster information includes the cluster center c and the number of vectors in the cluster n c and the maximum distance d max .

7. A distributed storage node fault detection system as claimed in claim 6, characterized in that: The fusion processing of the cluster analysis results of the local layer includes: B1, receiving the local clustering results of each node in the area, including the cluster center set C = {c1, c2, ..., c m }、The number of vectors in the cluster N={n1,n2,...,n m }、Maximum distance set And the set of outliers A=a1,a2,...,a k ; B2. Calculate the inter-cluster similarity matrix S, for any two cluster centers c x and c y , and its similarity is calculated as follows: In the formula, s(c x ,c y ) represents the cluster center c x and c y similarity, σ is the Gaussian kernel parameter, determined by cross-validation; ‖c x -c y ‖ represents the Euclidean distance between the centers of two clusters, n x ,n y represents the number of vectors in the two clusters, Represents the maximum distance between two clusters; B3. Construct a hierarchical clustering tree T and use the adaptive threshold method to merge clusters: Merge th =β1×(mean(S)+β2×std(S)) In the formula, Merge th is the merging threshold, β1 is the benchmark coefficient, β2 is the variance adjustment coefficient, mean(S) is the mean of the similarity matrix S, and std(S) is the standard deviation of the similarity matrix S; B4. Calculate the regional anomaly index RAI: RAl(c)=w1×DS(c)+w2×IS(c)+w3×TS(c)+w4×AD(C) Where DS(c) is the density score, which indicates the density of the cluster; IS(c) is the isolation score, which indicates the degree of isolation of the cluster from other clusters; TS(c) is the temporal correlation score, which indicates the temporal evolution characteristics of the cluster; AD(c) is the anomaly distribution feature, which reflects the distribution characteristics of the anomalies around the cluster; w1, w2, and w3 are weight coefficients, and they satisfy w1+w2+w3=1; B5. Generate regional status descriptor RSD = {C * ,N * ,D * ,RAI * ,T * }, where C * is the set of cluster centers after merging, N * is the updated set of vector numbers within the cluster, D * is the updated maximum distance set, RAI * is the set of abnormal indices, T * Is the timestamp.

8. A distributed storage node fault detection system as claimed in claim 7, characterized in that: The process of performing anomaly detection at the global layer includes: C1, construct the global state matrix GSM = [RSD1; RSD2; ...; RSD r ], where r is the total number of regions; C2. Calculate the inter-regional correlation strength matrix RCM, and establish a regional correlation relationship diagram by analyzing the similarity of performance indicators, network topology distance, and similarity of abnormal patterns: RCM(a,b)=∈1×PS(a,b)+∈2×TS(a,b)+∈3×AS(a,b) Where RCM(a,b) represents the correlation strength matrix between region a and region b, PS(a,b) is the performance index similarity between region a and region b, TS(a,b) is the topological distance similarity between region a and region b, AS(a,b) is the anomaly pattern similarity between region a and region b, ∈1, ∈2, ∈3 are weight coefficients, and ∈1+∈2+∈3=1; C3. Based on the sliding time window mechanism, analyze the anomaly pattern in the time and space dimensions, including tracking the evolution trend of the anomaly state, identifying the propagation path of the anomaly between regions, and predicting the possible anomaly diffusion direction; C4, comprehensively evaluate the degree of abnormality, perform weighted fusion of local abnormal features, regional abnormal features and associated abnormal features to obtain the global abnormality score GAS; C5, using an adaptive threshold method to determine anomalies, the threshold is dynamically adjusted according to the statistical characteristics of historical anomaly data; C6. Output the standardized anomaly detection result set ADR, which includes the global anomaly score GAS, the location information AL of the abnormal area, the quantitative index AQ of the degree of anomaly, the propagation situation feature vector PF, the impact range prediction IR and the timestamp T.

9. A distributed storage node fault detection system as claimed in claim 8, characterized in that: The fault diagnosis module receives the anomaly detection result set ADR as input and processes it through the following subunits: The fault classification unit builds a multi-level classification model based on the XGBoost decision tree algorithm, takes the global anomaly score GAS, the anomaly area feature AL and the propagation feature vector PF as feature input, and trains the model in combination with the historical fault sample library to achieve the probability prediction of the fault type and output the fault type vector FT containing the confidence level; The feature association analysis unit uses the Apriori algorithm to mine the association rules between abnormal patterns and fault types based on the time series evolution data of abnormal regions and the propagation relationship between regions, constructs the fault feature map G, and establishes the feature association intensity matrix FCM for each fault type; The root cause location unit takes the area where the anomaly first appears as the starting point, combines the propagation situation feature PF and network topology information, analyzes the fault propagation path through the random walk algorithm, uses the PageRank algorithm to evaluate the importance of the node, and outputs the root cause probability ranking list RC and the fault impact range assessment report IR; The diagnostic report generation unit integrates the fault type FT, feature association map G and root cause analysis result RC to generate a standardized diagnostic report DR including fault location, impact assessment and repair suggestions, and provides visual fault diagnosis results.

10. A distributed storage node fault detection system as claimed in claim 9, characterized in that: The early warning decision module is based on the anomaly detection result set ADR and the fault diagnosis report DR, and is processed through the following sub-units: The fault level assessment unit comprehensively analyzes the global abnormality score GAS, fault type probability distribution FT and impact range prediction IR, establishes a multi-dimensional fault severity assessment system, and dynamically generates the fault level assessment result SL; The warning control unit implements a warning trigger mechanism based on a time window according to the fault level SL and the propagation situation feature PF, and determines the alarm sequence through the warning priority score; The strategy matching unit selects a matching processing solution from the preset strategy library based on the current fault type FT and the root cause analysis result RC; The decision optimization unit performs multi-objective optimization on candidate processing strategies, comprehensively considers factors such as fault severity SL, impact range IR, processing cost and risk, and outputs the optimal strategy execution plan and execution sequence.

Images