Storage system performance monitoring and analyzing method

Abstract

The invention relates to the technical field of computer storage, in particular to a storage system performance monitoring and analyzing method, which comprises the following steps of: acquiring disk I / O (Input / Output) read-write rate and overall response time of a storage system by using a data acquisition sensor, transmitting the read-write rate and the overall response time to a processing layer in real time through a message queue protocol, and then, establishing a real-time stream processing platform, calculating real-time statistical values of the two indexes by using a sliding window algorithm to reflect real-time fluctuation conditions of disk I / O and system response, then calculating dynamic correlation between the disk I / O read-write rate and the overall response time of the storage system based on a Pearson's correlation coefficient, continuously monitoring changes of the dynamic correlation, judging the influence degree of the two, and finally, calculating the dynamic correlation between the disk I / O read-write rate and the overall response time of the storage system. A method based on a structure causal model is applied, domain knowledge and a causal discovery algorithm are combined, a storage system performance index causal model is constructed, a directed acyclic graph is generated to show a causal relationship, the operation and maintenance efficiency of the storage system is improved, and the operation and maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to a storage system performance monitoring and analysis method. Background Art

[0002] Computer storage is an important technology. In today's digital age, storage systems play a key role in various businesses, and their performance is directly related to the stable operation of the business and user experience.

[0003] When an e-commerce company launches a big promotion, the number of orders will surge, resulting in a low disk I / O read / write rate. The disk I / O read / write rate determines the reading and writing speed of order data, and the overall response time reflects the total time from the user initiating an order request to the system giving feedback. When the disk I / O read / write rate is low, the data processing speed will slow down, which will directly lead to the system processing order requests. The time and overall response time increase, and users will obviously feel the order jam. However, the existing monitoring method only focuses on a single disk I / O read / write rate indicator, and does not consider it in conjunction with the overall response time. As a result, it is difficult for operation and maintenance personnel to detect that when the system processes a large number of orders, the overall response time increases significantly due to the read / write rate bottleneck, thereby causing the order processing efficiency to decrease. In order to solve this technical problem, we provide a storage system performance monitoring and analysis method. Summary of the invention

[0004] The purpose of the present invention is to provide a storage system performance monitoring and analysis method to solve the problems raised in the above background technology.

[0005] To achieve the above object, one of the objects of the present invention is to provide a storage system performance monitoring and analysis method, comprising the following steps:

[0006] S1, collects the disk I / O read / write rate and overall response time of the storage system through data acquisition sensors, and transmits them to the processing layer in real time through the message queue protocol;

[0007] S2. Use the real-time stream processing framework to build a real-time stream processing platform, and use the sliding window algorithm to calculate the real-time statistical values ​​of the disk I / O read / write rate and the overall response time. The real-time statistical values ​​are used to reflect the real-time fluctuations of the disk I / O and system response;

[0008] S3. Substitute the real-time statistical value into the Pearson correlation coefficient formula to calculate the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system, and use it as the key feature. By continuously monitoring the changes in the key features, determine the impact of the changes in the disk I / O read / write rate on the overall response time of the storage system.

[0009] S4. Use the method based on the structural causal model to construct a causal model of the storage system performance metrics. Determine the causal relationships of the variables in the model based on domain knowledge. Take the disk I / O read / write rate and the overall response time of the storage system as model variables. Use the constraint-based causal discovery algorithm to analyze the historical data, determine the causal structure between the variables, and construct a directed acyclic graph reflecting the causal relationships of the storage system performance metrics.

[0010] S5. When the processing layer detects that the overall response time of the storage system exceeds the preset performance threshold, input the key features into the causal model according to the causal relationships of each metric in the directed acyclic graph, and use the counterfactual reasoning method to locate the performance bottleneck, and comprehensively analyze to determine the root cause of the performance bottleneck.

[0011] As a further improvement of this technical solution, when calculating the real-time statistical value using the sliding window algorithm in S2, adopt a dynamic adaptive sliding window strategy, and the specific steps are as follows:

[0012] Set an initial window size, include the same number of disk I / O read / write rate and overall response time data as the initial window size into the window, and at the same time set the fluctuation threshold, stability threshold, and window adjustment coefficient;

[0013] In the current window, calculate the real-time statistical values of the disk I / O read / write rate and the overall response time. The statistical values include the mean and median. At the same time, calculate the standard deviation of the data in the window. The standard deviation is used to measure the degree of data fluctuation, and compare the standard deviation with the fluctuation threshold, and use the adjustment coefficient for adjustment according to the comparison result;

[0014] As new data arrives, the window slides according to the adjusted size, removes the earliest data, adds the latest data, and continuously calculates the real-time statistical value and dynamically adjusts the window size.

[0015] As a further improvement of this technical solution, when calculating the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system in S3, consider the time lag of the data, and the specific steps are as follows:

[0016] According to the experience of the storage system, set a maximum lag time, which represents the longest time range for the change of the disk I / O read / write rate to affect the overall response time of the storage system;

[0017] Starting from the initial moment, gradually increase the lag time until the maximum lag time is reached. For each lag time, shift the disk I / O read / write rate data backward by t time units, where t represents the lag time;

[0018] Substitute the translated disk I / O read / write rate data and the original overall storage system response time data into the Pearson correlation coefficient formula to calculate the correlation. Compare the absolute values of the correlation coefficients calculated at different lag times, and select the correlation coefficient corresponding to the lag time with the largest absolute value as the final dynamic correlation.

[0019] As a further improvement of this technical solution, when calculating the dynamic correlation in S3, the weighted Pearson correlation coefficient is used, and the specific steps are as follows:

[0020] Define the weight function with the exponential decay function ω(i) = e -λi Define the weight function, where λ is the decay coefficient and i is the serial number of the time point. The value of λ is determined by historical experience. For the disk I / O read / write rate data xi and the overall storage system response time data y i , calculate the weighted means and

[0021] Substitute the data into the weighted Pearson correlation coefficient formula, that is where r ω is the weighted Pearson correlation coefficient, n represents the total amount of data, and i represents the i-th disk I / O read / write rate data and the overall storage system response time data.

[0022] As a further improvement of this technical solution, when using the method based on the structural causal model to construct the causal model of the storage system performance metrics in S4, external environmental factors are introduced, and the specific steps are as follows:

[0023] By analyzing the storage system, determine the external environmental factors that affect the storage system performance. The external environmental factors include network bandwidth, system CPU usage rate, and memory usage rate. When collecting the disk I / O read / write rate and the overall storage system response time, use corresponding sensors to collect the data of external environmental factors and associate them with the disk I / O and response time data;

[0024] Based on domain knowledge, initially determine the causal relationships between external environmental factors, disk I / O read / write rate, and the overall storage system response time. Input the historical data containing external environmental factors into the constraint-based causal discovery algorithm, analyze the data, determine the causal structure among all variables, and construct a directed acyclic graph containing the disk I / O read / write rate, the overall storage system response time, and external environmental factors according to the results of the causal structure discovery.

[0025] As a further improvement of this technical solution, when using the constraint-based causal discovery algorithm to analyze the historical data in S4, a hybrid causal discovery strategy is adopted, and the specific steps are as follows:

[0026] Use a constraint-based causal discovery algorithm to conduct preliminary causal structure discovery on historical data. The constraint-based causal discovery algorithm gradually deletes irrelevant edges by testing the conditional independence between variables and constructs a preliminary directed acyclic graph;

[0027] Use the FCI algorithm to correct and improve the preliminary directed acyclic graph, adjust the directed acyclic graph in combination with domain knowledge, and finally use the cross-validation method to evaluate the adjusted directed acyclic graph, calculate the accuracy of the model, and optimize the model according to the evaluation results.

[0028] As a further improvement of this technical solution, in step S4, after constructing the directed acyclic graph, conduct uncertainty analysis on it. The specific steps are as follows:

[0029] Conduct multiple random samplings from historical data. Each sampling obtains a new data set. For each sampled data set, use a constraint-based causal discovery algorithm to construct a directed acyclic graph;

[0030] Count the frequency of each causal relationship in all directed acyclic graphs. Use the frequency of the causal relationship as the confidence level of this causal relationship. For causal relationships with a confidence level lower than the confidence level threshold, conduct secondary data collection and analysis, and locate the performance bottleneck according to the analysis results.

[0031] As a further improvement of this technical solution, when using the counterfactual reasoning method to locate the performance bottleneck in step S5, combine Bayesian networks for reasoning. The specific steps are as follows:

[0032] Convert the constructed directed acyclic graph into a Bayesian network, and use the maximum likelihood estimation method to determine the conditional probability distribution for each node of the directed acyclic graph;

[0033] Observe the actual state of the storage system. When the disk I / O read / write rate is observed to be a certain value, set the disk I / O read / write rate node as an evidence node, and use this value as its value, and propose different counterfactual hypotheses. Assume that the disk I / O read / write rate changes to another value. For each counterfactual hypothesis, change the values of the corresponding nodes;

[0034] Use the Bayesian inference algorithm to calculate the probability distribution of the overall response time of the storage system under different counterfactual hypotheses. By comparing the probability distributions under different counterfactual hypotheses, determine the factor that has the greatest impact on the performance bottleneck. According to the results of Bayesian inference and combined with the actual system situation, comprehensively analyze and determine the root cause of the performance bottleneck.

[0035] Compared with the prior art, the beneficial effects of the present invention:

[0036] In the storage system performance monitoring and analysis method, the Pearson correlation coefficient is used to calculate the dynamic correlation between the disk I / O read / write rate and the storage system response time, continuously monitor the changes in key features, and accurately judge the impact degree of the disk I / O read / write rate change on the overall response time of the storage system, providing a quantitative basis for in-depth analysis of performance problems. Based on the structural causal model, a causal model of storage system performance indicators is constructed, and the causal structure of variables is determined by combining domain knowledge and causal discovery algorithms, and a directed acyclic graph is constructed to clearly present the causal relationship between performance indicators. This enables operation and maintenance personnel to quickly locate the root cause of performance bottlenecks when facing performance problems, improving the problem-solving efficiency and reducing the operation and maintenance costs. Brief Description of the Drawings

[0037] Figure 1 This is the overall workflow diagram of the present invention. Detailed Embodiment

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0039] Please refer to Figure 1 As shown, this embodiment provides a storage system performance monitoring and analysis method, including the following steps:

[0040] S1. Collect the disk I / O read / write rate and the overall response time of the storage system through data acquisition sensors, and transmit them to the processing layer in real time through the message queue protocol.

[0041] S2. Build a real-time stream processing platform using a real-time stream processing framework, and use the sliding window algorithm to calculate the real-time statistical values of the disk I / O read / write rate and the overall response time. The real-time statistical values are used to reflect the real-time fluctuations of the disk I / O and the system response.

[0042] When using the sliding window algorithm to calculate the real-time statistical values in S2, a dynamic adaptive sliding window strategy is adopted, and the specific steps are as follows:

[0043] In order to calculate the real-time statistical values using the dynamic adaptive sliding window strategy, it is first necessary to determine the initial window size and the criteria for measuring data fluctuations and stability, and clarify the adjustment rules for the window size, so as to provide a basic framework for subsequent calculations and adjustments.

[0044] Set an initial window size W0, include the same number of disk I / O read / write rate and overall response time data as the initial window size into the window, and set the fluctuation threshold σth Stable threshold σ st And window adjustment coefficients k1 (0 < k1 < 1) and k2 (k2 > 1) define the window size and judgment criteria in the initial state, providing a clear starting point for subsequent dynamic adjustment and making the algorithm operable.

[0045] Calculating the real-time statistical values and standard deviations of disk I / O read / write rate and overall response time within the current window is to comprehensively understand the central tendency and fluctuation of current data. At the same time, calculating the standard deviation of data within the window, which is used to measure the degree of data fluctuation, and comparing the standard deviation with the fluctuation threshold, and making adjustments using the adjustment coefficient according to the comparison result.

[0046] Let the median of the disk I / O read / write rate data within the current window be M x , and the standard deviation be σ x , let the median of the overall response time data be M y , and the standard deviation be σ y , if σ x > σ th or σ y > σ th , then adjust the window size to W1 = k1 × W. If σ x < σ st or σ y < σ st , then adjust the window size to W2 = k2 × W. Otherwise, keep the window size W unchanged, which comprehensively considers the mean, median, and standard deviation and can fully reflect the characteristics of the data.

[0047] As new data arrives, the window slides according to the adjusted size, removing the earliest data and adding the latest data, continuously calculating real-time statistical values and dynamically adjusting the window size, making the monitoring of disk I / O read / write rate and overall response time more real-time and accurate, and providing more effective data support for the performance analysis of the storage system.

[0048] S3. Substitute the real-time statistical values into the Pearson correlation coefficient formula to calculate the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system, and use it as a key feature. By continuously monitoring the changes in the key feature, judge the impact degree of the change in the disk I / O read / write rate on the overall response time of the storage system.

[0049] When calculating the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system in S3, consider the time lag of the data. The specific steps are as follows:

[0050] When calculating the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system, considering that the impact of changes in the disk I / O read / write rate on the overall response time of the storage system may not be immediately apparent and there is a certain delay process, according to the experience of the storage system, a maximum lag time T is set. max This represents the longest time range for the impact of changes in the disk I / O read / write rate on the overall response time of the storage system, providing a reasonable time boundary for determining the true correlation between the disk I / O read / write rate and the overall response time of the storage system, and avoiding unnecessary calculations and analyses.

[0051] To accurately find the correlation between the disk I / O read / write rate and the overall response time of the storage system, it is necessary to consider the situation of different lag times. Starting from the initial moment, the lag time t is gradually increased until the maximum lag time T is reached. max For each lag time t, the disk I / O read / write rate is shifted backward by t time units, denoted as the lag time t, making the analysis of the correlation between the disk I / O read / write rate and the overall response time of the storage system more comprehensive and accurate, and capable of capturing the possible delay relationship between them.

[0052] Substituting the shifted disk I / O read / write rate data and the original overall response time data of the storage system into the Pearson correlation coefficient formula to calculate the correlation is to quantify the degree of association between the two under different lag times, compare the absolute values of the correlation coefficients calculated under different lag times, and select the correlation coefficient corresponding to the lag time with the largest absolute value as the final dynamic correlation.

[0053] The dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system is accurately determined in a quantitative manner, and the time lag is considered, making the calculation of the correlation more in line with the actual situation, and helping to discover potential performance problems and bottlenecks.

[0054] In S3, when calculating the dynamic correlation, the weighted Pearson correlation coefficient is adopted, and the specific steps are as follows:

[0055] When calculating the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system, the impact of data at different time points on the current system performance may vary. Define the weight function with the exponential decay function ω(i) = e -λi where λ is the decay coefficient and i is the serial number of the time point. The value of λ is determined by historical experience. For the disk I / O read / write rate data xi and the overall response time data y of the storage system i , calculate the weighted means and The correlation calculation can more accurately reflect the current performance state of the system, improve the accuracy of the analysis of the relationship between the disk I / O read / write rate and the overall response time of the storage system, and provide more valuable information for performance monitoring and analysis.

[0056] After obtaining the weighted mean, substituting the data into the weighted Pearson correlation coefficient formula for calculation is to quantify the correlation between the disk I / O read / write rate and the overall response time of the storage system. Considering the weight differences of data at different time points, substituting the data into the weighted Pearson correlation coefficient formula, that is where r ω is the weighted Pearson correlation coefficient, n represents the total amount of data, i represents the i-th disk I / O read / write rate data and the overall response time data of the storage system. By calculating the weighted Pearson correlation coefficient, it is possible to more precisely judge the impact degree of the change in the disk I / O read / write rate on the overall response time of the storage system, provide a more reliable basis for the analysis and optimization of the storage system performance, and help to discover and solve potential performance problems in a timely manner.

[0057] S4. Use the method based on the structural causal model to construct the causal model of the storage system performance indicators, determine the causal relationship of the variables in the model based on domain knowledge, take the disk I / O read / write rate and the overall response time of the storage system as the model variables, use the constraint-based causal discovery algorithm to analyze the historical data, determine the causal structure between the variables, and construct a directed acyclic graph reflecting the causal relationship of the storage system performance indicators.

[0058] When using the method based on the structural causal model to construct the causal model of the storage system performance indicators in S4, external environmental factors are introduced, and the specific steps are as follows:

[0059] The performance of the storage system is not only determined by the disk I / O read / write rate and the overall response time. External environmental factors will also have an important impact on it. The network bandwidth will affect the data transmission speed, the system CPU usage rate reflects the occupation of the system's computing resources, and the memory usage rate reflects the utilization degree of the memory resources.

[0060] By analyzing the storage system, determine the external environmental factors that affect the storage system performance. The external environmental factors include network bandwidth, system CPU usage rate, and memory usage rate. While collecting the disk I / O read / write rate and the overall response time of the storage system, use the corresponding sensors to collect the data of the external environmental factors, and associate them with the disk I / O and response time data to form a multi-variable data set where n is the number of data samples, X io is the disk I / O read / write rate data, Y re is the overall response time data of the storage system, Z bw , Zcpu , Z mem are the network bandwidth data, the system CPU usage data, and the memory usage data respectively.

[0061] Domain knowledge is the experience accumulated based on long-term practice and research. Using domain knowledge, the approximate causal relationships between external environmental factors and the disk I / O read / write rate and the overall response time of the storage system can be initially determined, providing prior information for subsequent causal structure discovery. Input the historical data containing external environmental factors into the constraint-based causal discovery algorithm, analyze the data, and determine the causal structure among all variables as follows:

[0062] Fill in the missing values in the disk I / O read / write rate, the overall response time of the storage system, the network bandwidth, the system CPU usage, and the memory usage data. For continuous variables, partial correlation analysis can be used for conditional independence testing. For discrete variables, chi-square testing can be used.

[0063] For the two continuous variables of the disk I / O read / write rate and the overall response time of the storage system, calculate their partial correlation coefficients under the condition of given other variables. If the partial correlation coefficient is between [0, 0.1], it is considered that they are independent under this condition. Set a significance level α. When the obtained partial correlation coefficient value from the test is less than α, reject the original hypothesis of independence between the variables and consider that there is a causal relationship between them. Starting from all variable pairs, gradually increase the number of conditional variables for testing. First, test whether variables A and B are unconditionally independent, then test whether A and A are independent under the condition of given variable C, and so on.

[0064] Based on the results of conditional independence testing, it can be determined whether there is an edge between variables, thus constructing the skeleton of the causal graph. The causal graph skeleton shows the basic connection relationships between variables and provides a basis for subsequent determination of the direction of the edges. Represent all variables as the vertices of the graph. Initially, there are no edges in the graph. If the disk I / O read / write rate and the network bandwidth are not independent under all conditions, add an undirected edge connecting them in the graph.

[0065] In the causal graph skeleton, find A - B - C where there is no edge connection between A and C. Determine the direction of the edge as A → B ← C according to the results of conditional independence testing. Use some direction propagation rules. Based on the edges with determined directions and the structure of the graph, further determine the directions of other edges, and gradually convert the undirected graph into a directed acyclic graph.

[0066] The causal structure obtained through the above steps may have errors and needs to be verified and optimized to improve the accuracy of the causal structure. The dataset is divided into a training set and a validation set. Causal structure discovery is performed on the training set, and then the accuracy of the obtained causal structure is verified on the validation set. If domain knowledge indicates that network bandwidth does not directly affect disk I / O read and write rates, but there is such an edge in the causal structure, the data and the verification process need to be rechecked.

[0067] In S4, when using the constraint-based causal discovery algorithm to analyze historical data, a hybrid causal discovery strategy is adopted. The specific steps are as follows:

[0068] To mine the causal relationships between variables from historical data, a basic causal structure framework needs to be constructed first. The constraint-based causal discovery algorithm can systematically exclude the connections where there is no causal relationship between variables by testing the conditional independence between variables, thereby initially constructing a directed acyclic graph, providing a starting point for subsequent precise analysis.

[0069] For discrete variables, the chi-square test is used. The chi-square statistic formula is where Oij is the observed frequency and Eij is the expected frequency. For continuous variables, partial correlation analysis is used. The partial correlation coefficient formula is where r XY 、r XZ 、r YZ are the correlation coefficients between variables X and Y, X and Z, and Y and Z respectively.

[0070] According to the conditional independence test results, if variables X and Y are conditionally independent given the variable set S, then the edge connecting X and Y is deleted. This constraint-based method has a certain theoretical basis and can efficiently construct a preliminary causal structure by using the conditional independence information in the data, avoiding blind search for all possible causal relationships and having relatively high computational efficiency.

[0071] Use the FCI algorithm to correct and improve the preliminary directed acyclic graph, and adjust the directed acyclic graph in combination with domain knowledge to obtain a more accurate and reliable directed acyclic graph, reducing the error in causal relationship judgment and providing a more accurate basis for subsequent performance analysis and decision-making. Adjust the directed acyclic graph in combination with domain knowledge. To evaluate the accuracy and reliability of the adjusted directed acyclic graph, finally use the cross-validation method to evaluate the adjusted directed acyclic graph, calculate the accuracy of the model, and optimize the model according to the evaluation results.

[0072] Randomly divide the dataset D into k non-overlapping subsets D1, D2, …, D k , for each subset D i, use it as the validation set, and use the remaining k - 1 subsets as the training set. Train the causal model on the training set and validate it on the validation set to obtain k evaluation results. Calculate the accuracy by the ratio of the number of correctly predicted samples to the total number of samples. According to the accuracy evaluation results, adjust the significance level α of the conditional independence test of the algorithm to obtain a causal model with better performance and higher reliability, which can more accurately reflect the causal relationship between variables and provide more powerful support for the performance analysis and decision-making of the storage system.

[0073] In S4, after constructing the directed acyclic graph, perform uncertainty analysis on it. The specific steps are as follows:

[0074] Since historical data may have problems such as noise and incompleteness, a single directed acyclic graph may not accurately reflect the true causal relationship between variables. Conduct multiple random samplings from the historical data. Each sampling obtains a new data set. For each sampled data set, use the constraint-based causal discovery algorithm to construct a directed acyclic graph.

[0075] Let the historical data set be D = {x1, x2,..., x N}, where N is the number of data samples. Each sampling randomly selects n samples from D to form a new data set D i . For discrete variables, use the chi-square test, and for continuous variables, use partial correlation analysis to generate multiple directed acyclic graphs, providing a rich data basis for subsequent analysis of the stability and confidence of causal relationships.

[0076] To evaluate the reliability of each causal relationship, it is necessary to perform statistical analysis on the directed acyclic graphs obtained from multiple samplings. Take the occurrence frequency of the causal relationship as the confidence level, which can intuitively reflect the stability of the causal relationship under different data sets, and thus judge its reliability.

[0077] Statistically count the frequency of each causal relationship in all directed acyclic graphs. Take the frequency of the causal relationship as the confidence level of this causal relationship. Suppose a total of M samplings are conducted, and a certain causal relationship A → B appears in m directed acyclic graphs. Then the confidence level of this causal relationship Assigns a confidence level value to each causal relationship, facilitating the screening and evaluation of causal relationships.

[0078] For causal relationships with a confidence level lower than the confidence level threshold, it indicates that their reliability is relatively low, which may be caused by data noise, incompleteness, or limitations of the causal discovery algorithm. Conduct secondary data collection and analysis, and locate the performance bottleneck according to the analysis results.

[0079] S5. When the processing layer detects that the overall response time of the storage system exceeds the preset performance threshold, it inputs the key features into the causal model according to the causal relationships of the various metrics in the directed acyclic graph, and uses the counterfactual reasoning method to locate the performance bottleneck, and comprehensively analyzes to determine the root cause of the performance bottleneck.

[0080] When S5 uses the counterfactual reasoning method to locate the performance bottleneck, it combines Bayesian networks for reasoning. The specific steps are as follows:

[0081] Although the directed acyclic graph shows the causal relationships between variables, it lacks a quantitative description of the probabilistic relationships between variables. Converting it into a Bayesian network and using the maximum likelihood estimation method to determine the conditional probability distribution of each node can provide a basis for subsequent probabilistic reasoning, enabling us to analyze the mutual influence between variables in terms of probability.

[0082] Let there be a node X in the Bayesian network i , and the set of all its parent nodes is πi. For discrete variables, the conditional probability distribution P(X i |π i ) can be calculated by maximum likelihood estimation. Suppose we have N sample data. For each combination of values π i of π i j and each value X i of X i k , the frequencies of their occurrences are and respectively. Then the conditional probability The Bayesian network combines the causal structure of the directed acyclic graph and the ability of probabilistic reasoning, and can more comprehensively describe the relationships between variables, enabling us to analyze the impact of different variable values on the overall response time of the storage system based on probability.

[0083] Observe the actual state of the storage system. When it is observed that the disk I / O read / write rate is a certain value, the disk I / O read / write rate node is set as the evidence node, and this value is used as its value, and different counterfactual hypotheses are proposed. Suppose the disk I / O read / write rate is changed to another value. For each counterfactual hypothesis, the values of the corresponding nodes are changed, providing different scenarios for subsequent Bayesian reasoning, enabling us to determine the factor that has the greatest impact on the performance bottleneck by comparing the results in different scenarios.

[0084] Use the Bayesian inference algorithm to calculate the probability distribution of the overall response time of the storage system under different counterfactual hypotheses, which can quantify the likelihood of the overall response time of the storage system in different scenarios. For the overall response time node Y of the storage system re, given the evidence node E and the value Xhyp of the node under the counterfactual hypothesis, according to Bayes' theorem In actual calculations, the probability distribution is updated by passing messages between the nodes of the Bayesian network. By comparing the distributions of P(Y re |E, X hyp ) under different counterfactual hypotheses, the factors that have the greatest impact on Yre are determined. By comparing the probability distributions, the factors that have the greatest impact on the performance bottleneck can be intuitively identified, providing a clear direction for performance optimization and a scientific basis for subsequent performance optimization, which helps to improve the performance and stability of the storage system.

[0085] The above storage system performance monitoring and analysis method collects the disk I / O read and write rates and the overall response time of the storage system through data acquisition sensors, and transmits them to the processing layer in real time through the message queue protocol. Then, a real-time stream processing platform is built, and the sliding window algorithm is used to calculate the real-time statistical values of these two metrics, reflecting the real-time fluctuations of disk I / O and system response. Next, the dynamic correlation between the disk I / O read and write rate and the overall response time of the storage system is calculated based on the Pearson correlation coefficient, and its changes are continuously monitored to judge the degree of influence between the two. Finally, the method based on the structural causal model is used, combined with domain knowledge and causal discovery algorithms, to construct a causal model of the storage system performance metrics, and a directed acyclic graph is generated to show the causal relationship, improving the operation and maintenance efficiency of the storage system and reducing the operation and maintenance costs.

[0086] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A storage system performance monitoring and analysis method, characterized in that: The following steps are involved: S1, collects the disk I / O read / write rate and overall response time of the storage system through data acquisition sensors, and transmits them to the processing layer in real time through the message queue protocol; S2. Use the real-time stream processing framework to build a real-time stream processing platform, and use the sliding window algorithm to calculate the real-time statistical values ​​of the disk I / O read / write rate and the overall response time. The real-time statistical values ​​are used to reflect the real-time fluctuations of the disk I / O and system response; S3. Substitute the real-time statistical value into the Pearson correlation coefficient formula to calculate the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system, and use it as the key feature. By continuously monitoring the changes in the key features, determine the impact of the changes in the disk I / O read / write rate on the overall response time of the storage system. S4. Use the method based on structural causal model to build a causal model of storage system performance indicators. Determine the causal relationship of variables in the model based on domain knowledge. Use disk I / O read / write rate and storage system overall response time as model variables. Use constraint-based causal discovery algorithm to analyze historical data, determine the causal structure between variables, and construct a directed acyclic graph that reflects the causal relationship of storage system performance indicators. S5. When the processing layer detects that the overall response time of the storage system exceeds the preset performance threshold, the key features are input into the causal model according to the causal relationship of each indicator in the directed acyclic graph, and the performance bottleneck is located using the counterfactual reasoning method. A comprehensive analysis is performed to determine the root cause of the performance bottleneck.

2. The storage system performance monitoring and analysis method according to claim 1, characterized in that: In S2, when the sliding window algorithm is used to calculate the real-time statistical value, a dynamic adaptive sliding window strategy is adopted. The specific steps are as follows: Set an initial window size, include the same amount of disk I / O read / write rate and overall response time data as the initial window size into the window, and set the fluctuation threshold, stability threshold, and window adjustment coefficient. In the current window, calculate the real-time statistics of disk I / O read / write rate and overall response time. The statistics include mean and median. At the same time, calculate the standard deviation of the data in the window. The standard deviation is used to measure the degree of data fluctuation. The standard deviation is compared with the fluctuation threshold and adjusted using the adjustment coefficient based on the comparison result. As new data arrives, the window slides according to the adjusted size, removing the earliest data and adding the latest data, continuously calculating real-time statistics and dynamically adjusting the window size.

3. The storage system performance monitoring and analysis method according to claim 2, characterized in that: In the S3, the time lag of data is taken into account when calculating the dynamic correlation between the disk I / O read / write rate and the overall response time of the storage system. The specific steps are as follows: Based on the experience of the storage system, a maximum lag time is set to indicate the maximum time range in which changes in the disk I / O read / write rate affect the overall response time of the storage system; Starting from the initial moment, gradually increase the lag time until the maximum lag time is reached. For each lag time, shift the disk I / O read / write rate data backward by t time units, where t is the lag time. Substitute the translated disk I / O read / write rate data and the original storage system overall response time data into the Pearson correlation coefficient formula to calculate the correlation, compare the absolute values ​​of the correlation coefficients calculated under different lag times, and select the correlation coefficient corresponding to the lag time with the largest absolute value as the final dynamic correlation.

4. The storage system performance monitoring and analysis method according to claim 3, characterized in that: In S3, when calculating the dynamic correlation, the weighted Pearson correlation coefficient is used, and the specific steps are as follows: With exponential decay function ω(i) = e -λi Define a weight function, where λ is the decay coefficient, i is the serial number of the time point, and the value of λ is determined by historical experience. For the disk I / O read / write rate data xi and the storage system overall response time data y i , calculate the weighted mean and Substitute the data into the weighted Pearson correlation coefficient formula, that is, where r ω is the weighted Pearson correlation coefficient, n represents the total amount of data, and i represents the i-th disk I / O read / write rate data and the overall response time data of the storage system.

5. The storage system performance monitoring and analysis method according to claim 4, characterized in that: In S4, when constructing a storage system performance indicator causal model using a method based on a structural causal model, external environmental factors are introduced. The specific steps are as follows: By analyzing the storage system, the external environmental factors that affect the performance of the storage system are determined, including network bandwidth, system CPU usage, and memory usage. While collecting disk I / O read / write rates and the overall response time of the storage system, corresponding sensors are used to collect data on external environmental factors and associate them with disk I / O and response time data. Based on domain knowledge, the causal relationship between external environmental factors and disk I / O read / write rates and the overall response time of the storage system is preliminarily determined. The historical data containing external environmental factors is input into the constraint-based causal discovery algorithm. The data is analyzed to determine the causal structure between all variables. According to the results of the causal structure discovery, a directed acyclic graph including disk I / O read / write rates, the overall response time of the storage system and external environmental factors is constructed.

6. The storage system performance monitoring and analysis method according to claim 5, characterized in that: In S4, when using the constraint-based causal discovery algorithm to analyze historical data, a hybrid causal discovery strategy is adopted, and the specific steps are as follows: A constraint-based causal discovery algorithm is used to perform preliminary causal structure discovery on historical data. The constraint-based causal discovery algorithm constructs a preliminary directed acyclic graph by testing the conditional independence between variables and gradually removing irrelevant edges. The FCI algorithm is used to correct and improve the preliminary directed acyclic graph, and the directed acyclic graph is adjusted in combination with domain knowledge. Finally, the adjusted directed acyclic graph is evaluated using the cross-validation method to calculate the accuracy of the model. Based on the evaluation results, the model is optimized.

7. The storage system performance monitoring and analysis method according to claim 6, characterized in that: After constructing the directed acyclic graph in S4, uncertainty analysis is performed on it. The specific steps are as follows: Perform multiple random samplings from historical data, and obtain a new data set each time. For each sampled data set, use a constraint-based causal discovery algorithm to construct a directed acyclic graph; The frequency of each causal relationship in all directed acyclic graphs is counted, and the frequency of the causal relationship is used as the confidence of the causal relationship. For causal relationships with confidence levels lower than the confidence threshold, secondary data collection and analysis are performed, and performance bottlenecks are located based on the analysis results.

8. The storage system performance monitoring and analysis method according to claim 7, characterized in that: When S5 uses the counterfactual reasoning method to locate the performance bottleneck, it combines the Bayesian network for reasoning. The specific steps are as follows: Convert the constructed directed acyclic graph into a Bayesian network, and use the maximum likelihood estimation method to determine the conditional probability distribution for each node of the directed acyclic graph; Observe the actual state of the storage system. When the disk I / O read / write rate is observed to be a certain value, set the disk I / O read / write rate node as the evidence node and use the value as its value. Then propose different counterfactual hypotheses. Suppose the disk I / O read / write rate changes to another value. For each counterfactual hypothesis, change the value of the corresponding node. Use the Bayesian reasoning algorithm to calculate the probability distribution of the overall response time of the storage system under different counterfactual assumptions. By comparing the probability distributions under different counterfactual assumptions, determine the factors that have the greatest impact on the performance bottleneck. Based on the results of Bayesian reasoning and the actual system situation, a comprehensive analysis is performed to determine the root cause of the performance bottleneck.

Images