A multi-connection clustering method for data center clusters

Abstract

The application discloses a multi-connection clustering method for a data center cluster, characterized in that N monitoring data groups are randomly selected from a monitoring system of the data center cluster application to form a data set, and normalization is performed; initial clustering is performed on the normalized data set to obtain K clusters; the average value of all monitoring data in each cluster is obtained, and the average value is taken as a new cluster center; each monitoring data in the normalized data set is assigned to the cluster where the new cluster center with the nearest Euclidean distance is located, and K clusters are obtained; the above steps are repeated until a stop condition is met; each cluster is numbered and recorded; the above steps are repeated, so that each monitoring data obtains L cluster numbers, and time obtained according to the L cluster numbers of each monitoring data is aggregated in a first-order sequence into a multi-tuple; monitoring data corresponding to the same multi-tuple is classified into the same class, Knew classes are obtained, and clustering is completed; and the advantage is that accurate and actual clustering results can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data center cluster application, and particularly relates to a multi-connection clustering method for data center cluster. BACKGROUND

[0002] In the era of big data, the scale of data center cluster is expanding, and the complexity of its application is increasing. The traditional data processing and analysis method is not competent for the monitoring data of massive applications, especially in the classification of large-scale applications, there are many challenges and limitations. At present, by using clustering algorithm to analyze the monitoring data of data center cluster application, the effective classification and organization of massive application load can be realized, so as to find the monitoring data with similar load patterns, optimize the cluster resource allocation mode, and improve the cluster scheduling management efficiency. It can also be used for anomaly detection to identify monitoring data with abnormal resource consumption and perform targeted performance optimization, thereby improving the scheduling efficiency and resource utilization of data center cluster application.

[0003] The K-means algorithm is a relatively classic clustering algorithm, the core idea of which is to divide the data set into K clusters through an iterative process, so that the distance between each data point and the center point (centroid) of its cluster is minimized, and each cluster has similar characteristics. Although the K-means algorithm is simple and easy to understand, it requires the user to specify the number of clusters K in advance at the beginning, and in actual applications, it is often difficult to determine the true cluster number of data in advance. Selecting an inappropriate K value may lead to inaccurate or unrealistic clustering results. In addition, the selection of initial values in the K-means algorithm directly affects the convergence and clustering results of the K-means algorithm. If the initial values are not properly selected, the algorithm may fall into a local optimal solution and fail to find a global optimal solution, resulting in unstable and inaccurate clustering results. The same data may get different clustering results under different initial conditions. In addition, the K-means algorithm assumes that the clusters are convex and similar in shape and size, and it performs poorly for clusters of complex shapes and different sizes. Complex-shaped clusters may be incorrectly split or merged, and small clusters may be ignored or misjudged as noise, while large clusters may be over-divided or contain unrelated data points, thus failing to accurately reflect the actual distribution of data. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a multi-connection clustering method for data center cluster, which can obtain accurate and realistic clustering results.

[0005] The technical solution adopted by the present application to solve the above technical problems is: a multi-connection clustering method for data center cluster, comprising the following steps:

[0006] Step 1, randomly select N monitoring data sets from the monitoring system of the data center cluster application to form a data set, and normalize the data set to obtain a normalized data set;

[0007] Step 2, perform initial clustering on the normalized data set to obtain K clusters;

[0008] Step 3, for the K clusters, obtain the average value of all monitoring data in each cluster, and take the average value as a new cluster center;

[0009] Step 4, obtain the Euclidean distance between each monitoring data in the normalized data set and the new cluster center, and according to the obtained Euclidean distance, assign each monitoring data in the normalized data set to the cluster where the new cluster center with the nearest Euclidean distance is located, to obtain K clusters;

[0010] Step 5, repeat steps 3 to 4 until the stop condition is met, and execute step 6;

[0011] Step 6, number each cluster obtained by step 5 to obtain and record the cluster number of each monitoring data;

[0012] Step 7, repeat steps 2 to 6 until the repetition number is L times, and each monitoring data obtains L cluster numbers, and the time according to the L cluster numbers of each monitoring data is aggregated in chronological order into a multi-tuple, the monitoring data corresponding to the same multi-tuple is classified into the same class, to obtain Knew classes, and the clustering of the data center cluster is completed.

[0013] Compared with the prior art, the advantages of the present application are that the cluster steps, i.e., steps ③ to ④, are repeatedly performed multiple times, effectively reducing the uncertainty of the clustering result caused by the randomness of the initial value selection, and improving the accuracy and stability of the data center cluster clustering; multiple repeated execution can ensure that the algorithm explores the data space from multiple angles and finally converges to a more stable cluster center, avoiding the problem of local optimal solution caused by single running. The multiple clustering results are aggregated into a tuple, and the K clusters obtained by initial clustering do not represent the final clustering result, but gradually increase with multiple running, solving the shortcoming of the existing K-means algorithm that needs to determine the initial value in advance, and obtaining an accurate and actual clustering result, enhancing the stability and usability of the data center cluster clustering result; at the same time, this method can automatically adjust the number of clusters according to the actual data distribution, more flexibly and accurately reflecting the real resource consumption mode of the data center. In addition, the constructed data set is clustered into Knew clusters, and the monitoring data in each cluster has similar resource consumption, which is convenient for subsequent application category management and anomaly detection in the data center cluster. By constructing a more stable and accurate clustering result, it is easier to identify clusters with abnormal behavior, which is crucial for the operation of the data center, because it helps to timely discover and solve potential problems, prevent system failure and security threats, and thus improve the overall system performance and security of the data center cluster.

[0014] Further, in step ①, each of the monitoring data includes multiple index data;

[0015] The index data includes CPU, memory, and disk IO.

[0016] The specific operation of normalizing the data set to obtain the normalized data set is: using the min-max normalization method to scale each index data in the data set to the range of 0-1 to obtain the normalized data set. Using the min-max normalization method can eliminate the influence of different dimensions and different value ranges on the calculation of the distance between data points, ensuring the comparability and fairness between different index data.

[0017] Further, the specific operation process of step ② is as follows:

[0018] Step ②-1, randomly selecting K monitoring data from the normalized data set as initial cluster centers;

[0019] Step ②-2, obtaining the Euclidean distance between each monitoring data in the normalized data set and the K initial cluster centers, respectively, and according to the obtained Euclidean distance, assigning each monitoring data in the normalized data set to the cluster where the initial cluster center with the nearest Euclidean distance is located, to obtain K clusters.

[0020] Further, K≥2.

[0021] Further, in step ⑤, the stop condition is that the number of repetitions reaches a preset number or the cluster assignment no longer changes.

[0022] Further, the preset number is 20.

[0023] Further, in step ⑦, L≥1; Knew≥K. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The overall flowchart of the present application is shown. DETAILED DESCRIPTION

[0025] The present application will be further described in detail below with reference to the embodiments of the drawings.

[0026] A multi-connection clustering method for a data center cluster, comprising the following steps:

[0027] Step ①: randomly selecting N monitoring data groups from a monitoring system of a data center cluster application to form a data set, and normalizing the data set to obtain a normalized data set; each monitoring data includes multiple index data; the index data includes CPU, memory, disk IO, and other system resource utilization indexes;

[0028] The specific operation of normalizing the data set to obtain the normalized data set is: using the minimum-maximum normalization method to scale each index data in the data set to the range of 0-1 to obtain the normalized data set;

[0029] Step ②: performing initial clustering on the normalized data set to obtain K clusters:

[0030] Step ②-1: randomly selecting K monitoring data from the normalized data set as initial cluster centers, K≥2;

[0031] Step ②-2: obtaining the Euclidean distance between each monitoring data in the normalized data set and the K initial cluster centers, respectively, and according to the obtained Euclidean distance, assigning each monitoring data in the normalized data set to the cluster where the initial cluster center with the nearest Euclidean distance is located, to obtain K clusters;

[0032] Step ③: for the K clusters, obtaining the average value of all monitoring data in each cluster, and taking the average value as a new cluster center;

[0033] Step ④: obtaining the Euclidean distance between each monitoring data in the normalized data set and the new cluster center, respectively, and according to the obtained Euclidean distance, assigning each monitoring data in the normalized data set to the cluster where the new cluster center with the nearest Euclidean distance is located, to obtain K clusters;

[0034] Step ⑤, repeat steps ③ to ④ until the stop condition is met, and perform step ⑥; the stop condition is that the number of repetitions reaches a preset number or the cluster assignment no longer changes (i.e., the average value calculated this time is the same as the last time); the preset number is 20 times;

[0035] Step ⑥, number each cluster obtained in step ⑤ to obtain and record the cluster number of each monitoring data; wherein, the clusters are numbered from 0 to K-1, for example: the cluster number of the first cluster is 0, and the cluster numbers of all monitoring data in the first cluster are 0;

[0036] Step ⑦, repeat steps ② to ⑥ until the number of repetitions is L (L≥1) times, each monitoring data obtains L cluster numbers, and the time obtained from the L cluster numbers of each monitoring data is aggregated in chronological order into a tuple, the monitoring data corresponding to the same tuple is classified into the same class, Knew (Knew≥K) classes are obtained, and the clustering of the data center cluster is completed.

[0037] Example: randomly select N=4 monitoring data from the monitoring system of the data center cluster to form a data set, which are monitoring data a, monitoring data b, monitoring data c and monitoring data d; set K=2 and L=3;

[0038] The L cluster numbers of monitoring data a are: 0, 1, 1; the L cluster numbers of monitoring data b are: 1, 0, 0; the L cluster numbers of monitoring data c are: 0, 1, 0; and the L cluster numbers of monitoring data d are: 1, 0, 0;

[0039] After aggregation, the tuple of monitoring data a is (0, 1, 1), the tuple of monitoring data b is (1, 0, 0), the tuple of monitoring data c is (0, 1, 0), and the tuple of monitoring data d is (1, 0, 0);

[0040] Therefore, monitoring data b and monitoring data d are classified into the same class, monitoring data a is one class, and monitoring data c is one class, obtaining Knew=3 classes.

Claims

1. A multi-connection clustering method for data center clusters, characterized in that... Includes the following steps: Step ①: Randomly select N monitoring data from the monitoring system of the data center cluster application to form a dataset, and normalize the dataset to obtain the normalized dataset. Step ②: Perform initial clustering on the normalized dataset to obtain K clusters; Step ③: For K clusters, obtain the average value of all monitoring data within each cluster, and use this average value as the new cluster center; Step 4: Obtain the Euclidean distance between each monitoring data point in the normalized dataset and the new cluster center. Based on the obtained Euclidean distance, assign each monitoring data point in the normalized dataset to the cluster containing the new cluster center with the closest Euclidean distance, resulting in K clusters. Step 5: Repeat steps 3 to 4 until the stopping condition is met, then proceed to step 6. Step 6: Number each cluster obtained in step 5, and record the cluster number for each monitoring data. Step ⑦: Repeat steps ② to ⑥ until the number of repetitions is L. Each monitoring data will have L cluster numbers. Based on the time obtained from the L cluster numbers of each monitoring data, aggregate them into a tuple in chronological order. Group the monitoring data with the same tuple into the same class to obtain Knew classes, thus completing the clustering of the data center cluster.

2. The multi-connection clustering method for data center clusters according to claim 1, characterized in that... In step ①, each monitoring data item includes multiple indicator data; The aforementioned metrics include CPU, memory, and disk I / O; The specific operation to normalize the dataset and obtain the normalized dataset is as follows: use the min-max normalization method to scale each indicator data in the dataset to the range of 0-1 to obtain the normalized dataset.

3. The multi-connection clustering method for data center clusters according to claim 1, characterized in that... The specific operation process of step ② is as follows: Step ②-1: Randomly select K monitoring data points from the normalized dataset as initial cluster centers; Step ②-2: Obtain the Euclidean distance between each monitoring data point in the normalized dataset and the K initial cluster centers. Based on the obtained Euclidean distances, assign each monitoring data point in the normalized dataset to the cluster containing the initial cluster center with the closest Euclidean distance, thus obtaining K clusters.

4. The multi-connection clustering method for data center clusters according to claim 3, characterized in that... K≥2。 5. The multi-connection clustering method for data center clusters according to claim 1, characterized in that... In step ⑤, the stopping condition is that the number of repetitions reaches a preset number or the cluster allocation no longer changes.

6. The multi-connection clustering method for data center clusters according to claim 5, characterized in that... The preset number of times is 20.

7. The multi-connection clustering method for data center clusters according to claim 1, characterized in that... In step ⑦, L≥1; Knew≥K.

Images