A method for analyzing the performance anomaly of distributed storage systems

Abstract

The present invention relates to the field of data processing, and discloses a method for analyzing the automatic performance anomaly of a distributed storage system, comprising: presetting performance collection points in each process in the distributed system, and inputting the parent-child relationship of the performance collection points; collecting the performance indicators of each performance collection point, and obtaining the rate of change of the performance indicators of the performance collection point with the time interval according to the performance indicators of the performance collection point; when the performance collection point generates anomalies or jitters within the set time period, the performance collection point is the target performance collection point, and the correlation between the target performance collection point and each descendant performance collection point is obtained; the correlation between the target performance collection point and each descendant performance collection point is sorted, and the descendant performance collection point and leaf node with the greatest correlation are obtained; according to the descendant performance collection point and leaf node with the greatest correlation, the performance anomaly propagation path is obtained. The present invention can realize the rapid and automatic positioning of the submodule causing the performance jitter.

Description

Technical Field

[0001] The invention relates to the field of data processing, and in particular to an automatic performance anomaly analysis method for a distributed storage system. Background Art

[0002] When a cluster has performance anomalies, this anomaly generally refers to increased IO latency, and it takes a long time to analyze which part of the IO path is causing the problem. In a distributed system, the IO path is very long, from the client, network, primary replica, network, secondary replica, write disk, network, return to primary replica, network, client, any abnormality in any of these parts may cause the overall performance of the cluster to be abnormal. And considering the large number of nodes, it takes a long time to quickly locate which disk, which node, which switch caused the problem, which is very unfavorable to product development efficiency.

[0003] The existing performance analysis system collects performance data from various statistical points and obtains the delay size of each stage on the IO path. By analyzing these data, it can be determined which stage has the largest delay and is the bottleneck point that needs to be optimized; it can also be used for performance prediction, using performance data to predict task execution time, thereby performing better task scheduling.

[0004] However, these analysis methods cannot quickly solve the analysis of performance anomalies. These methods only focus on the size of each statistical point, but ignore the stability of these statistical points. For performance anomaly analysis, the focus is on the stability of each statistical point. Although some io stages account for a small proportion of the overall delay, when the delays of other stages are very stable, a performance anomaly in a processing stage with a small delay ratio will also cause system performance jitter. Background tasks, disk defragmentation, network surges, excessive metadata and other factors can cause performance jitter, and which ones are normal and which factors are the main factors cannot be quickly located in the existing analysis methods. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for automatically analyzing abnormal performance of a distributed storage system, comprising the following steps:

[0006] Step 1: In each process in the distributed system, pre-set the performance collection point and enter the parent-child relationship of the performance collection point;

[0007] Step 2: collecting performance indicators of each performance collection point, and obtaining the change rate of the performance indicator of the performance collection point with the time interval according to the performance indicator of the performance collection point;

[0008] Step 3: When an abnormality or jitter occurs at a performance collection point within a set time period, the performance collection point is used as a target performance collection point, and the performance curve of the target performance collection point is used as a target curve. The target curve is fitted with the performance change trend of each descendant performance collection point of the target performance collection point to obtain the correlation between the target performance collection point and each descendant performance collection point;

[0009] Step 4: According to the obtained correlation between the target performance collection point and each descendant performance collection point, the correlation between the benchmark performance collection point and each descendant performance collection point is sorted to obtain the descendant performance collection point and leaf node with the greatest correlation; according to the descendant performance collection point and leaf node with the greatest correlation, the performance anomaly propagation path is obtained.

[0010] Furthermore, in each process in the distributed system, the performance collection points are predefined, including:

[0011] In each process in the distributed system, a performance collection point is preset, and the performance collection point includes an interface for receiving network information and an interface for writing data to a disk.

[0012] Furthermore, the collecting of the performance indicators of each performance collection point and obtaining the change rate of the performance indicators of the performance collection points with the time interval according to the performance indicators of the performance collection points include:

[0013] The performance indicators of the performance collection points include iops and delay, and the rising and falling trends of the iops and delay of each collection point over time are obtained.

[0014] Furthermore, the performance variation trend of each descendant performance collection point of the target performance collection point and the fitting target curve to obtain the correlation between the target performance collection point and each descendant performance collection point includes:

[0015] From the target performance collection point downward, the performance change trend of each descendant node of the target performance collection point within the time period is fitted with the change trend of the target performance collection point. By comparing whether the positive and negative change trends at the same time point are consistent, the correlation between the two collection points is obtained.

[0016] Further, the relevance between the target performance collection point and each descendant performance collection point is sorted according to the obtained relevance between the target performance collection point and each descendant performance collection point, so as to obtain the descendant node and leaf node with the greatest relevance, including:

[0017] The correlation between the target performance collection point and each descendant performance collection point is sorted to obtain the descendant performance collection point and leaf node with the greatest correlation, and the sub-module and performance problem causing performance jitter are located according to the descendant performance collection point and leaf node with the greatest correlation, and the propagation path of the performance problem is obtained according to the descendant performance collection point and leaf node with the greatest correlation.

[0018] The beneficial effects of the present invention are:

[0019] 1. Quickly and automatically locate the submodule that causes performance jitter;

[0020] 2. This method can flexibly adjust the parent-child collection point definitions according to different configurations of the distributed system, and is suitable for different hardware configurations and topological structures.

[0021] 3. The performance impact relationship between modules is presented clearly and directly in a graphical way, which reduces the quality requirements for personnel. It is suitable for use in testing and operation and maintenance systems, and improves product iteration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of a process for analyzing an automated performance anomaly of a distributed storage system;

[0023] Figure 2 It is a schematic diagram of the pre-buried collection point information and parent-child relationship. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0025] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0026] like Figure 1 As shown, a distributed storage system automatic performance anomaly analysis method includes the following steps:

[0027] Step 1: In each process in the distributed system, pre-set the performance collection point and enter the parent-child relationship of the performance collection point;

[0028] Step 2: collecting performance indicators of each performance collection point, and obtaining the change rate of the performance indicator of the performance collection point with the time interval according to the performance indicator of the performance collection point;

[0029] Step 3: When an abnormality or jitter occurs at a performance collection point within a set time period, the performance collection point is used as a target performance collection point, and the performance curve of the target performance collection point is used as a target curve. The target curve is fitted with the performance change trend of each descendant performance collection point of the target performance collection point to obtain the correlation between the target performance collection point and each descendant performance collection point;

[0030] Step 4: According to the obtained correlation between the target performance collection point and each descendant performance collection point, the correlation between the benchmark performance collection point and each descendant performance collection point is sorted to obtain the descendant performance collection point and leaf node with the greatest correlation; according to the descendant performance collection point and leaf node with the greatest correlation, the performance anomaly propagation path is obtained.

[0031] The performance collection points are predefined in each process in the distributed system, including:

[0032] In each process in the distributed system, a performance collection point is preset, and the performance collection point includes an interface for receiving network information and an interface for writing data to a disk.

[0033] The collecting of the performance indicators of each performance collection point and obtaining the change rate of the performance indicators of the performance collection points with the time interval according to the performance indicators of the performance collection points include:

[0034] The performance indicators of the performance collection points include iops and delay, and the rising and falling trends of the iops and delay of each collection point over time are obtained.

[0035] The change rate is simply described as: (next-pre) / pre. If the change rate is positive, the value is increasing and the trend is increasing. The delay gradually increases, indicating that the delay has an upward trend.

[0036] The performance change trend of each descendant performance collection point of the target performance collection point and the fitting target curve is obtained to obtain the correlation between the target performance collection point and each descendant performance collection point, including:

[0037] From the target performance collection point downward, the performance change trend of each descendant node of the target performance collection point within the time period is fitted with the change trend of the target performance collection point. By comparing whether the positive and negative change trends at the same time point are consistent, the correlation between the two collection points is obtained.

[0038] Simply put, it is to see whether the shapes of the two curves are similar. That is, see which child performance point curve (preferably a leaf node) is similar to the shape of the target curve. When the child is rising, the parent is also rising, and when the child is falling, the parent is also falling. Then the relationship between the two is relatively large. Converted into mathematical form, it is whether the values ​​of the rate of change at the same time point are close. The closer the rate of change at more time points in a period of time, the greater the correlation.

[0039] The method of sorting the correlation between the target performance collection point and each descendant performance collection point according to the obtained correlation between the target performance collection point and each descendant performance collection point to obtain the descendant node and leaf node with the greatest correlation includes:

[0040] The correlation between the target performance collection point and each descendant performance collection point is sorted to obtain the descendant performance collection point and leaf node with the greatest correlation, and the sub-module and performance problem causing performance jitter are located according to the descendant performance collection point and leaf node with the greatest correlation, and the propagation path of the performance problem is obtained according to the descendant performance collection point and leaf node with the greatest correlation.

[0041] Specifically, in the first embodiment, this solution can automatically analyze the main factor module causing performance jitter among many modules in the distributed system. Developers only need to analyze this module specifically, saving a lot of analysis and positioning time.

[0042] In each process of the distributed system, data collection points are predefined. For example, at the interface for receiving network information and the interface for writing data to the disk, the latency and number of each IO are collected at these interfaces. The collected information is periodically reported to a time series database. In this way, some monitoring information of each disk and each node in the cluster is gathered in the database. And this monitoring information is recorded in order of time points. With the help of graphical tools, the performance curve of each statistical point can be seen. In this way, the value size and changes of each statistical point can be observed.

[0043] 1. Enter the parent-child relationship of performance collection points

[0044] The performance collection points embedded in the process have a parent-child relationship. In a strongly consistent distributed system, the IO processing time is returned to the client only after the master replica receives the processing return results of all slave replicas. Therefore, the performance of the master replica is composed of the processing performance of the slave replica and the network processing performance.

[0045] In a single copy, processing performance is composed of metadata processing performance and data processing performance. And data processing performance is composed of disk writing performance and software processing performance.

[0046] Therefore, each collection point of the cluster performance is a tree structure starting from the client, such as Figure 2 As shown in Figure 1, there is a parent-child relationship between each collection point. To locate the root cause of the performance problem, it is necessary to find the leaf node that causes the performance change.

[0047] These embedded collection point information and parent-child relationships need to be entered in advance into the analysis system.

[0048] 2. Calculate the performance change rate of each collection point

[0049] The performance of each collection point includes two elements: iops and latency. When the system latency increases, the client's iops will decrease. When the client increases the iops it sends, the system latency will increase. When the iops sent by the client decreases, the system latency will decrease. Each performance collection point will reflect the following characteristics: the iops and latency of the parent and child nodes should be positively correlated, while the iops and latency of a single node are inversely proportional. Calculate the rising and falling trends of the iops and latency of each collection point over time. The time interval can be modified dynamically as needed. The minimum supported time interval is 1s. Further reducing the time interval to improve accuracy will put pressure on the collection system and cause data distortion.

[0050] 3. Fit the target curve to the performance change trend of each descendant performance collection point

[0051] When a performance point of interest jitters within a period of time, the performance curve of the performance collection point is specified as the target of this analysis in the analysis system. When the node iops remains unchanged or decreases, the child node iops increases, indicating that background tasks are generated on the path between the two nodes. When the node latency increases and the iops decreases, the child node latency also increases and the iops decreases, indicating that this child node is the cause of the parent node's performance jitter. The analysis system fits the performance change trend of each descendant node in the performance relationship tree from the target point node downward to its change trend during the time period. The fitting method is to see whether the positive and negative trends of the change trends at the same time point are consistent. The more time points the positive and negative match (may be the same or opposite), the greater the correlation between the two collection points.

[0052] 4. List the collection points that are closest to the changing trend.

[0053] The target performance point calculated in the previous step and the correlation between each descendant performance point are sorted to obtain the descendant nodes and leaf nodes with the greatest correlation. These collection points are the main factors and propagation paths that cause the target performance jitter. Then the analyst can analyze the performance data of the problem node. Generally speaking, there are two factors. One is that the leaf node closest to the hardware has a problem, so it can be quickly located that it is a problem with a certain hardware module. The other is that a certain node has a sudden background task, so it can be quickly located that it is caused by a specific background task.

[0054] Embodiment 2, in each process of the distributed storage system, we pre-set multiple performance collection points, which cover the main functional modules and key interfaces of the system. Among them, the interface for receiving network information and the interface for writing data to disk are two points that we pay special attention to. The interface for receiving network information is responsible for processing the task of receiving data from the network, while the interface for writing data to disk is responsible for writing data to disk. When setting these performance collection points, we also clarified the parent-child relationship between them, that is, the interface for receiving network information is the parent node of the interface for writing data to disk. This means that if the performance of the interface for receiving network information is abnormal, it is likely to affect the performance of the interface for writing data to disk.

[0055] We began to collect performance indicators of the interface for receiving network information and the interface for writing data to disk. These indicators include iops (the number of input / output operations per second) and latency (the time required for data processing). Through real-time monitoring, we obtained data on the changes of these two performance indicators over time. In order to analyze the performance change trend more intuitively, we calculated the rate of change of these two performance indicators over time intervals. For example, based on the rising or falling trend of iops and latency. If the rate of change is positive, it means that this performance indicator is rising; if the rate of change is negative, it means that this performance indicator is falling.

[0056] After a period of monitoring, we found that the receiving network information interface had anomalies or jitters within the set time period. This means that the performance of the interface was unstable during this period. In order to find out the cause of the performance anomaly, we marked the receiving network information interface as the target performance collection point and began to fit its performance curve with the performance change trend of the descendant performance collection point (that is, the interface for writing data to the disk). We determine the correlation between the two collection points by comparing whether the positive and negative trends at the same time point are consistent. Specifically, if the performance of the receiving network information interface is increasing, the performance of the writing data to the disk interface is also increasing; similarly, if the performance of the receiving network information interface is decreasing, the performance of the writing data to the disk interface is also decreasing, then we can assume that the two collection points are highly correlated.

[0057] After fitting the performance change trend of the target curve and the descendant performance collection points, we sorted the correlation between the receiving network information interface and the writing data to disk interface. It turned out that the writing data to disk interface had the highest correlation. This means that the performance anomaly of the receiving network information interface is likely caused by the performance problem of the writing data to disk interface. To further confirm this, we checked the performance data of the writing data to disk interface and found that its iops and latency indicators did show abnormal fluctuations. Therefore, we determined that the writing data to disk interface is the submodule that causes performance jitter, and obtained the propagation path of the performance problem: receiving network information interface -> writing data to disk interface.

[0058] Through this example, we can see the specific process of performance anomaly analysis in a distributed storage system. By presetting performance collection points, collecting performance indicators, fitting performance change trends and analyzing correlations, we can effectively find out the causes of performance anomalies and determine the propagation path of performance problems.

Claims

1. A distributed storage system automatic performance anomaly analysis method, characterized in that: The steps include: Step 1: In each process in the distributed system, pre-set the performance collection point and enter the parent-child relationship of the performance collection point; Step 2: collecting performance indicators of each performance collection point, and obtaining the change rate of the performance indicator of the performance collection point with the time interval according to the performance indicator of the performance collection point; Step 3: When an abnormality or jitter occurs at a performance collection point within a set time period, the performance collection point is used as a target performance collection point, and the performance curve of the target performance collection point is used as a target curve. The target curve is fitted with the performance change trend of each descendant performance collection point of the target performance collection point to obtain the correlation between the target performance collection point and each descendant performance collection point; Step 4: According to the obtained correlation between the target performance collection point and each descendant performance collection point, the correlation between the target performance collection point and each descendant performance collection point is sorted to obtain the descendant performance collection point and leaf node with the greatest correlation; according to the descendant performance collection point and leaf node with the greatest correlation, the performance anomaly propagation path is obtained; The collecting of the performance indicators of each performance collection point and obtaining the change rate of the performance indicators of the performance collection points with the time interval according to the performance indicators of the performance collection points include: The performance indicators of the performance collection points include iops and latency, and the rising and falling trends of iops and latency of each collection point over time are obtained; The performance change trend of each descendant performance collection point of the target performance collection point and the fitting target curve is obtained to obtain the correlation between the target performance collection point and each descendant performance collection point, including: From the target performance collection point downward, the performance change trend of each descendant node of the target performance collection point within the time period is fitted with the change trend of the target performance collection point. By comparing whether the positive and negative change trends at the same time point are consistent, the correlation between the two collection points is obtained.

2. The method for automatic performance anomaly analysis of a distributed storage system according to claim 1, characterized in that: The performance collection points are pre-set in each process in the distributed system, including: In each process in the distributed system, a performance collection point is preset, and the performance collection point includes an interface for receiving network information and an interface for writing data to a disk.

3. The method for automatic performance anomaly analysis of a distributed storage system according to claim 1, characterized in that: The method of sorting the correlation between the target performance collection point and each descendant performance collection point according to the obtained correlation between the target performance collection point and each descendant performance collection point to obtain the descendant performance collection point and leaf node with the greatest correlation includes: The correlation between the target performance collection point and each descendant performance collection point is sorted to obtain the descendant performance collection point and leaf node with the greatest correlation, and the sub-module and performance problem causing performance jitter are located according to the descendant performance collection point and leaf node with the greatest correlation, and the propagation path of the performance problem is obtained according to the descendant performance collection point and leaf node with the greatest correlation.

Images