Storage System Performance Monitoring and Analysis Method

Abstract

The present invention relates to the field of computer storage technology, and more specifically, to a storage system performance monitoring and analysis method. The present invention utilizes a data acquisition sensor to collect the disk I / O read / write rate and overall response time of a storage system, and transmits the data to a processing layer in real time via a message queue protocol. Then, a real-time stream processing platform is built, and a sliding window algorithm is used to calculate the real-time statistical values of the two indicators to reflect the real-time fluctuations of the disk I / O and system response. Subsequently, the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system is calculated based on the Pearson correlation coefficient, and their changes are continuously monitored to determine the degree of influence of the two. Finally, a method based on a structural causal model is used, combined with domain knowledge and a causal discovery algorithm, to construct a causal model of storage system performance indicators, generate a directed acyclic graph to display the causal relationship, improve the operation and maintenance efficiency of the storage system, and reduce the operation and maintenance costs.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, in particular to a storage system performance monitoring and analysis method. Background Art

[0002] Computer storage is an important technology. In today's digital age, storage systems play a key role in various businesses, and their performance is directly related to the stable operation of the business and user experience.

[0003] When e-commerce companies launch big promotions, the number of orders will surge, resulting in low disk I / O read / write rates. The disk I / O read / write rate determines the speed at which order data is read and written, while the overall response time reflects the total time from when a user initiates an order request to when the system gives feedback. When the disk I / O read / write rate is low, the data processing speed will slow down, which will directly lead to longer system processing time for order requests and increased overall response time. Users will notice significant order delays. However, existing monitoring methods only focus on a single disk I / O read / write rate metric and do not consider it in conjunction with the overall response time. This makes it difficult for operations and maintenance personnel to detect that when the system processes a large number of orders, the overall response time increases significantly due to the read / write rate bottleneck, thereby reducing order processing efficiency. To address this technical problem, we provide a storage system performance monitoring and analysis method. Summary of the Invention

[0004] The purpose of the present invention is to provide a storage system performance monitoring and analysis method to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, one of the objectives of the present invention is to provide a storage system performance monitoring and analysis method, comprising the following steps:

[0006] S1, collects the disk I / O read / write rate and overall response time of the storage system through data acquisition sensors, and transmits them to the processing layer in real time through the message queue protocol;

[0007] S2. Build a real-time stream processing platform using a real-time stream processing framework and use a sliding window algorithm to calculate real-time statistics of disk I / O read / write rates and overall response time. These real-time statistics are used to reflect real-time fluctuations in disk I / O and system response.

[0008] S3. Substitute the real-time statistical values into the Pearson correlation coefficient formula to calculate the dynamic correlation between the disk I / O read / write rate and the overall storage system response time. This is used as the key feature. By continuously monitoring changes in the key feature, the impact of changes in the disk I / O read / write rate on the overall storage system response time is determined.

[0009] S4. Use a method based on structural causal modeling to construct a causal model for storage system performance indicators. Based on domain knowledge, determine the causal relationships among variables in the model. Use disk I / O read / write rates and the overall storage system response time as model variables. Use a constraint-based causal discovery algorithm to analyze historical data, determine the causal structure between variables, and construct a directed acyclic graph (DAG) that reflects the causal relationships among storage system performance indicators.

[0010] S5. When the processing layer detects that the overall response time of the storage system exceeds the preset performance threshold, it inputs key features into the causal model based on the causal relationship of each indicator in the directed acyclic graph, and uses the counterfactual reasoning method to locate the performance bottleneck. A comprehensive analysis is then performed to determine the root cause of the performance bottleneck.

[0011] As a further improvement of the present technical solution, in S2, when calculating the real-time statistical value using the sliding window algorithm, a dynamic adaptive sliding window strategy is adopted. The specific steps are as follows:

[0012] Set an initial window size, include the same amount of disk I / O read / write rate and overall response time data as the initial window size into the window, and set the fluctuation threshold, stability threshold, and window adjustment coefficient.

[0013] In the current window, real-time statistics of disk I / O read / write rates and overall response time are calculated. These statistics include the mean and median. The standard deviation of the data in the window is also calculated to measure data volatility. The standard deviation is compared with the volatility threshold and an adjustment factor is applied based on the comparison results.

[0014] As new data arrives, the window slides according to the adjusted size, removing the oldest data and adding the latest data, continuously calculating real-time statistics and dynamically adjusting the window size.

[0015] As a further improvement to this technical solution, in S3, the time lag of data is taken into account when calculating the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system. The specific steps are as follows:

[0016] Based on storage system experience, set a maximum latency, which represents the maximum time range during which changes in disk I / O read and write rates affect the overall storage system response time.

[0017] Starting from the initial moment, gradually increase the lag time until the maximum lag time is reached. For each lag time, shift the disk I / O read and write rate data backward by t time units, where t is the lag time.

[0018] Substitute the translated disk I / O read / write rate data and the original storage system overall response time data into the Pearson correlation coefficient formula to calculate the correlation. Compare the absolute values of the correlation coefficients calculated under different lag times, and select the correlation coefficient corresponding to the lag time with the largest absolute value as the final dynamic correlation.

[0019] As a further improvement of this technical solution, in S3, when calculating the dynamic correlation, a weighted Pearson correlation coefficient is used. The specific steps are as follows:

[0020] With exponential decay function ω(i)=e -λi Define a weight function, where λ is the decay coefficient, i is the serial number of the time point, and the value of λ is determined by historical experience. For the disk I / O read and write rate data xi and the storage system overall response time data y i , calculate the weighted mean and

[0021] Substitute the data into the weighted Pearson correlation coefficient formula, that is, where r ω is the weighted Pearson correlation coefficient, n represents the total amount of data, and i represents the i-th disk I / O read / write rate data and the overall storage system response time data.

[0022] As a further improvement of this technical solution, in S4, when constructing the storage system performance indicator causal model using the method based on the structural causal model, external environmental factors are introduced. The specific steps are as follows:

[0023] By analyzing the storage system, we identify external environmental factors that affect storage system performance, including network bandwidth, system CPU utilization, and memory utilization. While collecting disk I / O read and write rates and the overall storage system response time, we use appropriate sensors to collect data on these external environmental factors and correlate them with disk I / O and response time data.

[0024] Based on domain knowledge, we preliminarily determined the causal relationship between external environmental factors and the disk I / O read / write rate and the overall response time of the storage system. We then input historical data containing external environmental factors into a constraint-based causal discovery algorithm, analyzed the data, and determined the causal structure between all variables. Based on the results of the causal structure discovery, we constructed a directed acyclic graph that included the disk I / O read / write rate, the overall response time of the storage system, and external environmental factors.

[0025] As a further improvement of the present technical solution, in S4, when using the constraint-based causal discovery algorithm to analyze historical data, a hybrid causal discovery strategy is adopted. The specific steps are as follows:

[0026] A constraint-based causal discovery algorithm is used to perform preliminary causal structure discovery on historical data. The constraint-based causal discovery algorithm constructs a preliminary directed acyclic graph by testing the conditional independence between variables and gradually removing irrelevant edges.

[0027] The FCI algorithm is used to correct and improve the preliminary directed acyclic graph, and the directed acyclic graph is adjusted in combination with domain knowledge. Finally, the adjusted directed acyclic graph is evaluated using the cross-validation method to calculate the accuracy of the model. Based on the evaluation results, the model is optimized.

[0028] As a further improvement of this technical solution, after constructing the directed acyclic graph in S4, uncertainty analysis is performed on it. The specific steps are as follows:

[0029] Perform multiple random samplings from historical data, each time obtaining a new data set. For each sampled data set, a directed acyclic graph is constructed using a constraint-based causal discovery algorithm.

[0030] The frequency of each causal relationship in all directed acyclic graphs is counted, and the frequency of the causal relationship is used as the confidence level of the causal relationship. For causal relationships with confidence levels lower than the confidence threshold, secondary data collection and analysis are performed, and performance bottlenecks are located based on the analysis results.

[0031] As a further improvement of this technical solution, when S5 uses the counterfactual reasoning method to locate the performance bottleneck, it combines the Bayesian network for reasoning. The specific steps are as follows:

[0032] Convert the constructed directed acyclic graph into a Bayesian network and use the maximum likelihood estimation method to determine the conditional probability distribution for each node of the directed acyclic graph;

[0033] Observe the actual state of the storage system. When the disk I / O read / write rate is observed to be a certain value, set the disk I / O read / write rate node as the evidence node and use this value as its value. Then propose different counterfactual hypotheses, assuming that the disk I / O read / write rate changes to another value. For each counterfactual hypothesis, change the value of the corresponding node.

[0034] Use the Bayesian inference algorithm to calculate the probability distribution of the storage system's overall response time under different counterfactual assumptions. By comparing the probability distributions under different counterfactual assumptions, identify the factors that have the greatest impact on performance bottlenecks. Based on the results of Bayesian inference and the actual system conditions, a comprehensive analysis is conducted to determine the root cause of the performance bottleneck.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] In the storage system performance monitoring and analysis method, the Pearson correlation coefficient is used to calculate the dynamic correlation between the disk I / O read / write rate and the storage system response time. By continuously monitoring changes in key features, the impact of changes in the disk I / O read / write rate on the overall response time of the storage system can be accurately determined, providing a quantitative basis for in-depth analysis of performance issues. A causal model of storage system performance indicators is constructed based on the structural causal model. The causal structure of variables is determined by combining domain knowledge and causal discovery algorithms, and a directed acyclic graph is constructed to clearly present the causal relationship between various performance indicators. This enables operation and maintenance personnel to quickly locate the root cause of performance bottlenecks when facing performance problems, improve problem-solving efficiency, and reduce operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the overall workflow diagram of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] See also Figure 1 As shown, this embodiment provides a storage system performance monitoring and analysis method, including the following steps:

[0040] S1. Collect the disk I / O read / write rate and overall response time of the storage system through data acquisition sensors, and transmit them to the processing layer in real time through the message queue protocol.

[0041] S2. Use the real-time stream processing framework to build a real-time stream processing platform, and use the sliding window algorithm to calculate the real-time statistical values of disk I / O read and write rates and overall response time. The real-time statistical values are used to reflect the real-time fluctuations of disk I / O and system response.

[0042] In S2, when using the sliding window algorithm to calculate real-time statistical values, a dynamic adaptive sliding window strategy is adopted. The specific steps are as follows:

[0043] In order to use the dynamic adaptive sliding window strategy to calculate real-time statistical values, it is first necessary to determine the initial window size and the criteria for measuring data fluctuation and stability, and to clarify the rules for adjusting the window size. This will provide a basic framework for subsequent calculations and adjustments.

[0044] Set an initial window size W0, include the same amount of disk I / O read / write rate and overall response time data as the initial window size into the window, and set the fluctuation threshold σth , Stable threshold σ st As well as the window adjustment coefficients k1 (0<k1<1) and k2 (k2>1), the window size and judgment criteria in the initial state are clarified, providing a clear starting point for subsequent dynamic adjustments and making the algorithm operational.

[0045] Real-time statistics and standard deviations of disk I / O read / write rates and overall response time are calculated within the current window to fully understand the centralization trend and fluctuations of the current data. The standard deviation of the data within the window is also calculated to measure the degree of data fluctuation. The standard deviation is then compared with the fluctuation threshold and adjusted using an adjustment factor based on the comparison results.

[0046] Let the median of the disk I / O read and write rate data in the current window be M x , with a standard deviation of σ x , the median of the overall response time data is M y , with a standard deviation of σ y , if σ x >σ th or σ y >σ th , then adjust the window size to W1=k1×W, if σ x <σ st or σ y <σ st , then adjust the window size to W2=k2×W, otherwise, keep the window size W unchanged, taking into account the mean, median and standard deviation, which can fully reflect the characteristics of the data.

[0047] As new data arrives, the window slides according to the adjusted size, removing the oldest data and adding the latest data. Real-time statistics are continuously calculated and the window size is dynamically adjusted, making the monitoring of disk I / O read and write rates and overall response time more real-time and accurate, providing more effective data support for storage system performance analysis.

[0048] S3. Substitute the real-time statistical value into the Pearson correlation coefficient formula to calculate the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system. Use this as the key feature and continuously monitor changes in the key feature to determine the impact of changes in the disk I / O read / write rate on the overall response time of the storage system.

[0049] In S3, the time lag of data is taken into account when calculating the dynamic correlation between disk I / O read / write rates and the overall storage system response time. The specific steps are as follows:

[0050] When calculating the dynamic correlation between disk I / O read / write rate and the overall response time of the storage system, considering that the impact of changes in disk I / O read / write rate on the overall response time of the storage system may not be immediately apparent and there is a certain delay process, a maximum lag time T is set based on the experience of the storage system. max , which represents the maximum time range in which changes in disk I / O read / write rates affect the overall response time of the storage system. It provides a reasonable time boundary for determining the true correlation between disk I / O read / write rates and the overall response time of the storage system, avoiding unnecessary calculation and analysis.

[0051] In order to accurately find the correlation between the disk I / O read and write rate and the overall response time of the storage system, it is necessary to consider different lag times. Starting from the initial moment, gradually increase the lag time t until the maximum lag time T is reached. max For each lag time t, the disk I / O read / write rate is shifted backward by t time units, expressed as lag time t, which makes the analysis of the correlation between the disk I / O read / write rate and the overall response time of the storage system more comprehensive and accurate, and can capture the possible delay relationship between them.

[0052] Substituting the translated disk I / O read / write rate data and the original storage system overall response time data into the Pearson correlation coefficient formula to calculate the correlation is to quantify the degree of association between the two at different lag times. The absolute values of the correlation coefficients calculated at different lag times are compared, and the correlation coefficient corresponding to the lag time with the largest absolute value is selected as the final dynamic correlation.

[0053] The dynamic correlation between disk I / O read and write rates and the overall storage system response time is accurately determined in a quantitative manner, and time lag is taken into account, making the correlation calculation more realistic and helping to identify potential performance issues and bottlenecks.

[0054] In S3, the weighted Pearson correlation coefficient is used to calculate dynamic correlation. The specific steps are as follows:

[0055] When calculating the dynamic correlation between disk I / O read / write rate and the overall response time of the storage system, the impact of data at different time points on the current system performance may be different. The exponential decay function ω(i) = e -λi Define a weight function, where λ is the decay coefficient, i is the serial number of the time point, and the value of λ is determined by historical experience. For the disk I / O read and write rate data xi and the storage system overall response time data y i , calculate the weighted mean and This enables correlation calculations to more accurately reflect the current performance status of the system, improves the accuracy of analysis of the relationship between disk I / O read and write rates and the overall response time of the storage system, and provides more valuable information for performance monitoring and analysis.

[0056] After obtaining the weighted mean, the data is substituted into the weighted Pearson correlation coefficient formula for calculation. This is to quantify the correlation between the disk I / O read and write rate and the overall response time of the storage system. At the same time, the weight difference of data at different time points is taken into account. The data is substituted into the weighted Pearson correlation coefficient formula, that is, where r ω The weighted Pearson correlation coefficient is used, where n represents the total amount of data, and i represents the i-th disk I / O read / write rate data and the overall storage system response time data. By calculating the weighted Pearson correlation coefficient, we can more accurately determine the impact of changes in disk I / O read / write rates on the overall storage system response time, providing a more reliable basis for storage system performance analysis and optimization, and helping to promptly identify and resolve potential performance issues.

[0057] S4. Use a method based on structural causal models to construct a causal model of storage system performance indicators. Determine the causal relationship of variables in the model based on domain knowledge. Use disk I / O read and write rates and the overall response time of the storage system as model variables. Use a constraint-based causal discovery algorithm to analyze historical data, determine the causal structure between variables, and construct a directed acyclic graph that reflects the causal relationship of storage system performance indicators.

[0058] In S4, when constructing a causal model for storage system performance indicators using a method based on a structural causal model, external environmental factors are introduced. The specific steps are as follows:

[0059] Storage system performance isn't solely determined by disk I / O read / write rates and overall response time; external environmental factors also have a significant impact. Network bandwidth affects data transmission speed, system CPU utilization reflects the system's computing resource usage, and memory utilization reflects the level of memory resource utilization.

[0060] By analyzing the storage system, we can determine the external environmental factors that affect the storage system performance. These factors include network bandwidth, system CPU usage, and memory usage. While collecting disk I / O read and write rates and the overall storage system response time, we use corresponding sensors to collect data on these external environmental factors and correlate them with disk I / O and response time data to form a multivariate data set. Where n is the number of data samples, X io Disk I / O read and write rate data, Y re is the overall response time data of the storage system, Z bw , Zcpu , Z mem They are network bandwidth data, system CPU usage data, and memory usage data.

[0061] Domain knowledge is based on experience accumulated through long-term practice and research. It can be used to preliminarily determine the approximate causal relationship between external environmental factors and disk I / O read / write rates and the overall response time of the storage system. This provides prior information for subsequent causal structure discovery. Historical data containing external environmental factors is input into a constraint-based causal discovery algorithm, which analyzes the data and determines the causal structure between all variables. Specifically, the following is a description:

[0062] Missing values in the disk I / O read / write rate, storage system overall response time, network bandwidth, system CPU usage, and memory usage data are filled. For continuous variables, partial correlation analysis can be used to perform conditional independence tests, and for discrete variables, the chi-square test can be used.

[0063] For the two continuous variables, disk I / O read / write rate and storage system overall response time, calculate their partial correlation coefficients given other variables. If the partial correlation coefficient is between [0, 0.1], they are considered independent under this condition. Set a significance level α. When the partial correlation coefficient value obtained from the test is less than α, reject the null hypothesis of independence between the variables and assume that there is a causal relationship between them. Starting from all variable pairs, gradually increase the number of conditional variables for testing. First, test whether variables A and B are unconditionally independent, then test whether A and A are independent given variable C, and so on.

[0064] The results of the conditional independence test can be used to determine whether there are edges between variables, thereby constructing the skeleton of the causal graph. The causal graph skeleton shows the basic connection relationship between variables and provides a basis for subsequently determining the direction of the edges. All variables are represented as vertices of the graph. Initially, there are no edges in the graph. If the disk I / O read / write rate and network bandwidth are not independent under all conditions, an undirected edge connecting them is added to the graph.

[0065] Find ABC in the causal graph skeleton and there is no edge connection between A and C. According to the result of the conditional independence test, the direction of the edge is determined to be A→B←C. Using some directional propagation rules, according to the edges with determined directions and the structure of the graph, the directions of other edges are further determined, and the undirected graph is gradually converted into a directed acyclic graph.

[0066] The causal structure obtained through the above steps may contain errors and needs to be verified and optimized to improve the accuracy of the causal structure. The dataset is divided into a training set and a validation set. Causal structure discovery is performed on the training set, and then the accuracy of the causal structure is verified on the validation set. If domain knowledge indicates that network bandwidth does not directly affect the disk I / O read and write rate, but such an edge exists in the causal structure, the data and verification process need to be rechecked.

[0067] In S4, when using the constraint-based causal discovery algorithm to analyze historical data, a hybrid causal discovery strategy is adopted. The specific steps are as follows:

[0068] In order to extract the causal relationship between variables from historical data, it is necessary to first build a basic causal structure framework. The constraint-based causal discovery algorithm can systematically exclude connections between variables where there is no causal relationship by testing the conditional independence between variables, thereby preliminarily constructing a directed acyclic graph, providing a starting point for subsequent precise analysis.

[0069] For discrete variables, the chi-square test is used. The chi-square statistic formula is Where Oij is the observed frequency, Eij is the expected frequency, and partial correlation analysis is used for continuous variables. The partial correlation coefficient formula is: where r XY 、r XZ 、r YZ are the correlation coefficients between variables X and Y, X and Z, and Y and Z, respectively.

[0070] According to the results of the conditional independence test, if variables X and Y are conditionally independent under a given variable set S, the edge connecting X and Y is deleted. This constraint-based method has a certain theoretical basis and can efficiently construct a preliminary causal structure using the conditional independence information in the data, avoiding the blind search of all possible causal relationships and having high computational efficiency.

[0071] The FCI algorithm is used to correct and improve the preliminary directed acyclic graph, and the directed acyclic graph is adjusted in combination with domain knowledge to obtain a more accurate and reliable directed acyclic graph, which reduces the error in causal relationship judgment and provides a more accurate basis for subsequent performance analysis and decision-making. The directed acyclic graph is adjusted in combination with domain knowledge. In order to evaluate the accuracy and reliability of the adjusted directed acyclic graph, the cross-validation method is finally used to evaluate the adjusted directed acyclic graph, calculate the accuracy of the model, and optimize the model based on the evaluation results.

[0072] The dataset D is randomly divided into k mutually disjoint subsets D1, D2, ..., D k , for each subset D i, use it as the validation set, and the remaining k-1 subsets as the training set. Train the causal model on the training set and verify it on the validation set to obtain k evaluation results. Calculate the accuracy by the ratio of the number of correctly predicted samples to the total number of samples: According to the accuracy evaluation results, adjust the significance level α of the algorithm's conditional independence test to obtain a causal model with better performance and more reliability. It can more accurately reflect the causal relationship between variables and provide stronger support for storage system performance analysis and decision-making.

[0073] After constructing the directed acyclic graph in S4, uncertainty analysis is performed on it. The specific steps are as follows:

[0074] Since historical data may contain noise, incompleteness and other problems, a single directed acyclic graph may not accurately reflect the true causal relationship between variables. Multiple random samplings are performed from historical data, and a new data set is obtained each time. For each sampled data set, a constraint-based causal discovery algorithm is used to construct a directed acyclic graph.

[0075] Assume that the historical data set is D = {x1, x2, ..., x N}, where N is the number of data samples, and each sampling randomly selects n samples from D to form a new data set D i For discrete variables, the chi-square test is used, and for continuous variables, partial correlation analysis is used to generate multiple directed acyclic graphs, which provide a rich data basis for the subsequent analysis of the stability and confidence of causal relationships.

[0076] In order to evaluate the reliability of each causal relationship, it is necessary to perform statistical analysis on the directed acyclic graph obtained by multiple samplings. Taking the frequency of occurrence of the causal relationship as the confidence level can intuitively reflect the stability of the causal relationship under different data sets, thereby judging its reliability.

[0077] Count the frequency of each causal relationship in all directed acyclic graphs, and use the frequency of the causal relationship as the confidence of the causal relationship. Suppose a total of M samplings are performed, and a causal relationship A→B appears in m directed acyclic graphs, then the confidence of the causal relationship is A confidence value is assigned to each causal relationship to facilitate the screening and evaluation of causal relationships.

[0078] For causal relationships with confidence levels lower than the confidence threshold, it indicates that their reliability is low, which may be caused by data noise, incompleteness, or limitations of the causal discovery algorithm. Secondary data collection and analysis are performed, and performance bottlenecks are located based on the analysis results.

[0079] S5. When the processing layer detects that the overall response time of the storage system exceeds the preset performance threshold, it inputs key features into the causal model based on the causal relationship of each indicator in the directed acyclic graph, and uses the counterfactual reasoning method to locate the performance bottleneck. A comprehensive analysis is then performed to determine the root cause of the performance bottleneck.

[0080] When S5 uses counterfactual reasoning to locate performance bottlenecks, it combines Bayesian network reasoning. The specific steps are as follows:

[0081] Although the directed acyclic graph shows the causal relationship between variables, it lacks a quantitative description of the probabilistic relationship between variables. Converting it into a Bayesian network and using the maximum likelihood estimation method to determine the conditional probability distribution of each node can provide a basis for subsequent probabilistic reasoning, allowing us to analyze the mutual influence between variables in a probabilistic way.

[0082] Suppose there is a node X in the Bayesian network i , the set of all its parent nodes is πi, for discrete variables, the conditional probability distribution P(X i |π i ) can be calculated by maximum likelihood estimation. Suppose we have N sample data, for π i Every combination of values π i j and X i Every value of X i k , and their frequencies are and Then the conditional probability Bayesian networks combine the causal structure of directed acyclic graphs with the ability of probabilistic reasoning, enabling a more comprehensive description of the relationships between variables. This allows us to analyze the impact of different variable values on the overall response time of the storage system based on probability.

[0083] Observe the actual status of the storage system. When the disk I / O read / write rate is observed to be a certain value, set the disk I / O read / write rate node as the evidence node and use this value as its value. Then propose different counterfactual hypotheses, assuming that the disk I / O read / write rate changes to another value. For each counterfactual hypothesis, change the value of the corresponding node, providing different scenarios for subsequent Bayesian reasoning. This allows us to determine the factor that has the greatest impact on performance bottlenecks by comparing the results under different scenarios.

[0084] The Bayesian inference algorithm is used to calculate the probability distribution of the overall response time of the storage system under different counterfactual assumptions. It can quantify the possibility of the overall response time of the storage system under different scenarios. For the overall response time node Y of the storage system re, given the evidence node E and the node value Xhyp under the counterfactual hypothesis, according to Bayes' theorem In actual calculations, the probability distribution is updated by passing messages between nodes in the Bayesian network, and the probability distribution is updated by comparing P(Y re |E,X hyp ) to determine the factors that have the greatest impact on Yre. By comparing probability distributions, we can intuitively identify the factors that have the greatest impact on performance bottlenecks, provide a clear direction for performance optimization, and provide a scientific basis for subsequent performance optimization, which helps improve the performance and stability of the storage system.

[0085] The above storage system performance monitoring and analysis method collects the disk I / O read / write rate and overall response time of the storage system through data acquisition sensors, and transmits them to the processing layer in real time through the message queue protocol. Then, a real-time stream processing platform is built, and the sliding window algorithm is used to calculate the real-time statistical values of these two indicators to reflect the real-time fluctuations of disk I / O and system response. Then, based on the Pearson correlation coefficient, the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system is calculated, and their changes are continuously monitored to determine the degree of impact of the two. Finally, a method based on structural causal model is used, combined with domain knowledge and causal discovery algorithm, to construct a causal model of storage system performance indicators, generate a directed acyclic graph to show the causal relationship, improve the operation and maintenance efficiency of the storage system, and reduce the operation and maintenance cost.

[0086] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A storage system performance monitoring and analysis method, characterized by: The following steps are involved: S1, collects the disk I / O read / write rate and overall response time of the storage system through data acquisition sensors, and transmits them to the processing layer in real time through the message queue protocol; S2. Build a real-time stream processing platform using a real-time stream processing framework and use a sliding window algorithm to calculate real-time statistics of disk I / O read / write rates and overall response time. These real-time statistics are used to reflect real-time fluctuations in disk I / O and system response. S3. Substitute the real-time statistical values into the Pearson correlation coefficient formula to calculate the dynamic correlation between the disk I / O read / write rate and the overall storage system response time. This is used as the key feature. By continuously monitoring changes in the key feature, the impact of changes in the disk I / O read / write rate on the overall storage system response time is determined. S4. Use a method based on structural causal modeling to construct a causal model for storage system performance indicators. Based on domain knowledge, determine the causal relationships among variables in the model. Use disk I / O read / write rates and the overall storage system response time as model variables. Use a constraint-based causal discovery algorithm to analyze historical data, determine the causal structure between variables, and construct a directed acyclic graph (DAG) that reflects the causal relationships among storage system performance indicators. S5. When the processing layer detects that the overall response time of the storage system exceeds the preset performance threshold, it inputs key features into the causal model based on the causal relationship between the indicators in the directed acyclic graph, and uses counterfactual reasoning to locate the performance bottleneck. A comprehensive analysis is conducted to determine the root cause of the performance bottleneck. When using counterfactual reasoning to locate the performance bottleneck, S5 combines reasoning with Bayesian networks. The specific steps are as follows: Convert the constructed directed acyclic graph into a Bayesian network and use the maximum likelihood estimation method to determine the conditional probability distribution for each node of the directed acyclic graph; Observe the actual state of the storage system. When the disk I / O read / write rate is observed to be a certain value, set the disk I / O read / write rate node as the evidence node and use this value as its value. Then propose different counterfactual hypotheses, assuming that the disk I / O read / write rate changes to another value. For each counterfactual hypothesis, change the value of the corresponding node. Use the Bayesian inference algorithm to calculate the probability distribution of the storage system's overall response time under different counterfactual assumptions. By comparing the probability distributions under different counterfactual assumptions, identify the factors that have the greatest impact on performance bottlenecks. Based on the results of Bayesian inference and the actual system conditions, a comprehensive analysis is conducted to determine the root cause of the performance bottleneck.

2. The storage system performance monitoring and analysis method according to claim 1, wherein: In S2, when calculating real-time statistical values using a sliding window algorithm, a dynamic adaptive sliding window strategy is adopted. The specific steps are as follows: Set an initial window size, include the same amount of disk I / O read / write rate and overall response time data as the initial window size into the window, and set the fluctuation threshold, stability threshold, and window adjustment coefficient. In the current window, real-time statistics of disk I / O read / write rates and overall response time are calculated. These statistics include the mean and median. The standard deviation of the data in the window is also calculated to measure data volatility. The standard deviation is compared with the volatility threshold and an adjustment factor is applied based on the comparison results. As new data arrives, the window slides according to the adjusted size, removing the oldest data and adding the latest data, continuously calculating real-time statistics and dynamically adjusting the window size.

3. The storage system performance monitoring and analysis method according to claim 2, wherein: In S3, the time lag of data is taken into account when calculating the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system. The specific steps are as follows: Based on storage system experience, set a maximum latency, which represents the maximum time range during which changes in disk I / O read and write rates affect the overall storage system response time. Starting from the initial moment, gradually increase the lag time until the maximum lag time is reached. For each lag time, shift the disk I / O read and write rate data backward. time units, Expressed as lag time; Substitute the translated disk I / O read / write rate data and the original storage system overall response time data into the Pearson correlation coefficient formula to calculate the correlation. Compare the absolute values of the correlation coefficients calculated under different lag times, and select the correlation coefficient corresponding to the lag time with the largest absolute value as the final dynamic correlation.

4. The storage system performance monitoring and analysis method according to claim 3, wherein: In S3, when calculating the dynamic correlation, the weighted Pearson correlation coefficient is used. The specific steps are as follows: Exponential decay function Define the weight function, where is the attenuation coefficient, is the serial number of the time point, The value is determined by historical experience. and storage system overall response time data , calculate the weighted mean and ; Substitute the data into the weighted Pearson correlation coefficient formula, that is, ;in is the weighted Pearson correlation coefficient, Indicates the total amount of data, Indicates the Disk I / O read and write rate data and storage system overall response time data.

5. The storage system performance monitoring and analysis method according to claim 4, characterized in that: In S4, when constructing a causal model of storage system performance indicators using a method based on a structural causal model, external environmental factors are introduced. The specific steps are as follows: By analyzing the storage system, we identify external environmental factors that affect storage system performance, including network bandwidth, system CPU utilization, and memory utilization. While collecting disk I / O read and write rates and the overall storage system response time, we use appropriate sensors to collect data on these external environmental factors and correlate them with disk I / O and response time data. Based on domain knowledge, we preliminarily determined the causal relationship between external environmental factors and the disk I / O read / write rate and the overall response time of the storage system. We then input historical data containing external environmental factors into a constraint-based causal discovery algorithm, analyzed the data, and determined the causal structure between all variables. Based on the results of the causal structure discovery, we constructed a directed acyclic graph that included the disk I / O read / write rate, the overall response time of the storage system, and external environmental factors.

6. The storage system performance monitoring and analysis method according to claim 5, characterized in that: In S4, when analyzing historical data using the constraint-based causal discovery algorithm, a hybrid causal discovery strategy is adopted. The specific steps are as follows: A constraint-based causal discovery algorithm is used to perform preliminary causal structure discovery on historical data. The constraint-based causal discovery algorithm constructs a preliminary directed acyclic graph by testing the conditional independence between variables and gradually removing irrelevant edges. The FCI algorithm is used to correct and improve the preliminary directed acyclic graph, and the directed acyclic graph is adjusted in combination with domain knowledge. Finally, the adjusted directed acyclic graph is evaluated using the cross-validation method to calculate the accuracy of the model. Based on the evaluation results, the model is optimized.

7. The storage system performance monitoring and analysis method according to claim 6, characterized in that: After constructing the directed acyclic graph in S4, uncertainty analysis is performed on it. The specific steps are as follows: Perform multiple random samplings from historical data, each time obtaining a new data set. For each sampled data set, a directed acyclic graph is constructed using a constraint-based causal discovery algorithm. The frequency of each causal relationship in all directed acyclic graphs is counted, and the frequency of the causal relationship is used as the confidence level of the causal relationship. For causal relationships with confidence levels lower than the confidence threshold, secondary data collection and analysis are performed, and performance bottlenecks are located based on the analysis results.

Images