Running log data mining method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data mining method and device for a running log, equipment and a storage medium, and the method comprises the steps: obtaining a first data set corresponding to the running log of a server, the first data set comprising a plurality of to-be-mined sample data; determining each second data set for reflecting adjacent sample data corresponding to the sample data by utilizing a first distance between the sample data, a first average distance of the first data set and a preset adjacent sample selection rule; according to the data density of the second data set and the second average distance, target parameters used for reflecting density distribution characteristics of the second data set are determined; performing clustering initial center point selection processing by using each target parameter and a preset center selection rule, and determining an initial center point; and performing clustering processing based on the initial center point, a preset clustering algorithm and the first data set, and determining a target clustering result. The initial center point can be determined without enumeration, and the clustering efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data mining, and in particular to a data mining method and device, equipment and storage medium for operation logs. Background Art

[0002] Data mining is the process of analyzing and discovering valuable and meaningful information from a large amount of data, and can also predict future trends based on the data. It is an indispensable part of data knowledge discovery, which also includes steps such as data preprocessing and final data visualization. However, with the in-depth study of data mining, the two concepts of data mining and knowledge discovery are not much different. At present, data mining is still a relatively demanding subject area.

[0003] Log mining is the most important application in system development, and almost every company attaches great importance to this popular technology. Since it is relatively new, there is no official definition of this technology. People usually define it as the process of integrating and processing a large number of network background resources, and adding corresponding data mining algorithms to finally obtain the most valuable knowledge or information. There are many differences between log data and traditional data. For example, the format has certain logical rules compared to traditional data, and data requires more professional data models for data modeling and subsequent analysis and mining; more professional knowledge is needed to deal with a wide variety of log data; compared with the more mature traditional text mining, as a relatively new and highly concerned type of data mining, log mining is more developable and closer to the characteristics of the modern information age.

[0004] Through the above concepts, the characteristics and difficulties of log data mining can be basically summarized as follows:

[0005] (1) Massive growth of data

[0006] The Internet generates a large amount of data every day. For example, a website has a large number of users browsing the website every day, which generates a lot of log files. Secondly, the website itself also has a lot of text content.

[0007] (2) A wide variety of data

[0008] There are many file formats associated with logs, including text information directly crawled from web pages using crawlers, log files automatically generated by website servers, browsing path information pasted from data links, and data downloaded from the Internet such as videos and pictures.

[0009] (3) Frequent data updates

[0010] Due to the large user base, the website generates a large number of clicks every day, so the data changes dramatically every few days.

[0011] (4) Complex data format

[0012] Each data record is multi-dimensional data and contains a variety of information. For example, the data format of log records is very complex, and usually requires a specific program to prepare it before the algorithm runs. Therefore, after the preparation of 1TB of data, the data actually used for the subsequent algorithm may be less than 100MB.

[0013] There are many clustering methods currently, and they all have good effects in log mining related fields. However, there are still technical problems such as the difficulty in determining the initial cluster center and the general clustering accuracy and efficiency. For example, the clustering method based on the Canopy algorithm. The Canopy algorithm is a clustering algorithm based on a specific distance threshold. First, the threshold value of the Canopy algorithm is determined based on the operator's experience or multiple algorithm runs. These operations need to be performed on different data sets, which often consumes a lot of time and manpower when clustering; secondly, the initial vector points of the Canopy algorithm are random, so it takes a lot of time to enumerate and calculate.

[0014] Therefore, there is still room for further improvement in the current log mining system to alleviate the above-mentioned defects of clustering. Summary of the invention

[0015] The main purpose of the present invention is to provide a data mining method and device, equipment and storage medium for operation logs, which can solve the technical problems in the prior art that it is difficult to determine the initial cluster center, clustering accuracy and efficiency are generally poor.

[0016] To achieve the above object, the present invention provides a data mining method for operation logs in a first aspect, the method comprising:

[0017] Acquire a first data set corresponding to the operation log of the server, wherein the first data set includes a plurality of sample data to be mined, and the sample data is used to reflect the operation status of the server;

[0018] Determine a second data set corresponding to each of the sample data by using the first distance between the sample data, the first average distance of the first data set, and a preset adjacent sample selection rule, wherein the second data set is used to reflect adjacent sample data corresponding to the sample data;

[0019] determining a target parameter of each of the second data sets according to the data density of the second data sets and the second average distance, wherein the target parameter is used to reflect a density distribution feature of the second data sets;

[0020] Using each of the target parameters and the preset center selection rule to perform initial center point selection processing for clustering, and determine the initial center point;

[0021] Clustering processing is performed based on the initial center point, a preset clustering algorithm and the first data set to determine a target clustering result.

[0022] To achieve the above object, the second aspect of the present invention provides a data mining device for operation logs, the device comprising:

[0023] Data acquisition module: used to acquire a first data set corresponding to the operation log of the server, wherein the first data set includes a number of sample data to be mined, and the sample data is used to reflect the operation status of the server;

[0024] An adjacent set determination module is used to determine a second data set corresponding to each of the sample data by using the first distance between the sample data, the first average distance of the first data set and a preset adjacent sample selection rule, wherein the second data set is used to reflect adjacent sample data corresponding to the sample data;

[0025] A parameter determination module: used to determine a target parameter of each of the second data sets according to the data density of the second data sets and the second average distance, wherein the target parameter is used to reflect the density distribution characteristics of the second data sets;

[0026] Center determination module: used to select the initial center point of clustering by using each of the target parameters and the preset center selection rules to determine the initial center point;

[0027] Clustering processing module: used to perform clustering processing based on the initial center point, a preset clustering algorithm and the first data set to determine a target clustering result.

[0028] To achieve the above-mentioned purpose, the third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the method shown in the first aspect.

[0029] To achieve the above objectives, the fourth aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method shown in the first aspect.

[0030] The embodiments of the present invention have the following beneficial effects:

[0031] The present invention provides a data mining method for operation logs, the method comprising: obtaining a first data set corresponding to the operation log of a server, the first data set comprising a number of sample data to be mined, the sample data being used to reflect the operation status of the server; using a first distance between the sample data, a first average distance of the first data set and a preset adjacent sample selection rule to determine a second data set corresponding to each sample data, the second data set being used to reflect adjacent sample data corresponding to the sample data; determining target parameters of each second data set according to the data density of the second data set and the second average distance, the target parameters being used to reflect the density distribution characteristics of the second data set; using each target parameter and a preset center selection rule to perform initial center point selection processing for clustering, and determining the initial center point; performing clustering processing based on the initial center point, a preset clustering algorithm and the first data set, and determining the target clustering result. Through the above-mentioned method, the initial center point is determined by using the target parameter reflecting the density distribution characteristics of the second data set, and the initial center point can be determined without enumeration, thereby improving clustering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] in:

[0034] Figure 1 This is a flow chart of a data mining method for an operation log in an embodiment of the present invention;

[0035] Figure 2 This is a structural block diagram of a data mining device for operation logs in an embodiment of the present invention;

[0036] Figure 3 4 is a structural block diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1 , Figure 1FIG. 1 is a flow chart of a data mining method for a running log in an embodiment of the present invention. Figure 1 The method shown can be applied to both a terminal and a server. The terminal can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server can be implemented as an independent server or a server cluster composed of multiple servers. This embodiment is described by applying to a terminal as an example. Figure 1 The method shown comprises the following steps:

[0039] 101. Obtain a first data set corresponding to an operation log of a server, where the first data set includes a number of sample data to be mined, and the sample data is used to reflect the operation status of the server;

[0040] It should be noted that the server can be a distributed server cluster or an independent server, and this application does not limit this. Taking a server cluster as an example, each server can be regarded as a service node, and then a first data set N used to reflect the operation status of the server can be obtained as a mining object for data mining of the operation log, wherein the first data set includes a number of sample data, and the sample data can be operation log data, and the operation log data can include multidimensional attribute data of at least 9 attributes such as service node name, timestamp, 5-minute average load, CPU usage, memory usage, disk read rate, disk write rate, download bandwidth, upload bandwidth, etc., and then the sample data R to be mined can be the above-mentioned operation log data, or it can be the data after cleaning the above-mentioned operation log data, which is not limited here.

[0041] Exemplarily, step 101 includes the following steps A01 to A03:

[0042] A01. Obtain multi-dimensional original operation log data of the server;

[0043] A02. Clean the original operation log data using a preset multi-dimensional data cleaning rule to obtain cleaned operation log data;

[0044] Taking the server as the server of the electricity consumption information collection system as an example, the above-mentioned original operation log data can be the original electricity consumption information collection system operation log data, wherein the electricity consumption information collection system operation log data includes at least 9 attributes such as service node name, timestamp, 5-minute average load, CPU usage, memory usage, disk read rate, disk write rate, download bandwidth, and upload bandwidth.

[0045] Furthermore, a multi-dimensional data cleaning rule may be pre-set, and the operation log data of the power consumption information collection system is cleaned through step A02 to obtain the cleaned operation log data, wherein the multi-dimensional data cleaning rule may be as shown in Table 1 below:

[0046] Table 1

[0047]

[0048] A03. Arrange the cleaned operation log data according to preset data arrangement rules to obtain a number of sample data to be mined.

[0049] Furthermore, after obtaining the cleaned operation log data, data can be sorted according to the preset data sorting rules to obtain a number of sample data R to be mined. Specifically, the rules for sorting the cleaned data can be to use the service node name and timestamp as the data record I, and the 5-minute average load, CPU usage, memory usage, disk read rate, disk write rate, download bandwidth, upload bandwidth and other 7 attributes as the data group D. Then each sample data R corresponding to the data record I can be expressed as:

[0050] R=D I , where D = {d1, d2, d3, d4, d5, d6, d7}, I = {i1, i2} (1)

[0051] Among them, i1 represents the service node name, i2 represents the timestamp, d1 represents the 5-minute load, d2 represents the CPU usage, d3 represents the memory usage, d4 represents the disk read rate, d5 represents the disk write rate, d6 represents the download bandwidth, and d7 represents the upload bandwidth.

[0052] The first data set N includes a number of sample data R to be mined. The first data set can be regarded as a sample data set. Assuming that there are n sample data R in the cleaned sample data set N, N can be expressed as:

[0053] N={R1,R2,R3,…,R n} (2)

[0054] 102. Determine a second data set corresponding to each of the sample data by using the first distance between the sample data, the first average distance of the first data set, and a preset adjacent sample selection rule, where the second data set is used to reflect adjacent sample data corresponding to the sample data;

[0055] After obtaining the first data set, a second data set Q of each sample data can be selected from the first data set. The second data set can be regarded as an adjacent sample data set. The second data set Q is used to reflect the adjacent sample data corresponding to the sample data. The adjacent sample data set includes the adjacent sample data of the sample data. The adjacent sample data set can be obtained by the adjacent sample selection rule. The adjacent sample selection rule can be set based on the first average distance AvgDist(N). For example, to determine the sample data R i The second data set, the selection rule of adjacent samples can be based on the sample data R i The sample data within the radius of AvgDist(N) is taken as the sample data R. i The adjacent sample data.

[0056] The first distance is the distance between two sample data in the first data set. The distance between sample data can be the Euclidean distance. The Euclidean distance Dist(R i ,R j ) is defined as follows:

[0057]

[0058] Among them, R i ,R j are two sample data in the first data set N, d il and d jl The corresponding data groups It can be understood that formula (3) uses the attributes of the server in 7 dimensions as an example, so the maximum value of l is 7. In fact, the dimension l can be flexibly adjusted as needed, which can be more or less than 7, that is, the attributes can be flexibly expanded according to needs. The dimension l of 7 is only used as an example here without specific limitation.

[0059] Then the first average distance AvgDist(N) between sample data in the first data set N is defined as follows:

[0060]

[0061] In formula (4), n is the total number of sample data in the first data set N.

[0062] Sample data R i The adjacent sample data set Q is defined as follows:

[0063] Q={R m |R m ∈N,Dist(R i ,R m ) <AvgDist(N)} (5)

[0064] Define c(Q) as the number of elements in set Q, that is, sample data R i The number of sample data within the radius of AvgDist(N) as the center.

[0065] 103. Determine a target parameter of each of the second data sets according to the data density of the second data sets and the second average distance, where the target parameter is used to reflect density distribution characteristics of the second data sets;

[0066] 104. Performing a clustering initial center point selection process using each of the target parameters and a preset center selection rule to determine an initial center point;

[0067] Furthermore, after obtaining the second data set Q, the target parameter λ of the second data set of each sample data can be determined by the data density ρ(Q) and the second average distance AvgDist(Q) of the second data set Q. Q , target parameter λ Q Used to reflect the density distribution characteristics of the second data set.

[0068] Exemplarily, ρ(Q) is defined as the data density of the adjacent sample data set Q, then ρ(Q) can be expressed as:

[0069]

[0070] AvgDist(Q) is defined as the average distance between each sample data in the adjacent sample data set Q:

[0071]

[0072] Where R g ,R h is the sample data in the adjacent sample data set Q.

[0073] In a feasible implementation, the parameter may be the ratio of the data density to the second average distance, and the parameter λ Q The definition is as follows:

[0074]

[0075] It should be noted that, according to formula (8) Q From the definition, it can be seen that the smaller the AvgDist(Q) is and the larger the ρ(Q) is, the more densely distributed the sample data of the adjacent sample data set Q is. Therefore, λ i It can better reflect the density distribution characteristics of the adjacent sample data set Q. Therefore, we choose λ Q The largest sample data R i As the initial center point of the cluster.

[0076] Furthermore, when the target parameter is the ratio of the data density to the second average distance, the initial center point selection process of clustering is performed using each of the target parameters and the preset center selection rule to determine the initial center point, including steps B01 and B02:

[0077] B01. determining a target second data set corresponding to the maximum value in the ratio;

[0078] B02. Use the first sample data corresponding to the target second data set as the initial center point.

[0079] 105. Perform clustering processing based on the initial center point, a preset clustering algorithm and the first data set to determine a target clustering result.

[0080] After obtaining the initial center point of the cluster, clustering can be performed based on the center point and the first data set N, wherein the clustering algorithm can be the Canopy algorithm and / or the K-means algorithm. Taking the Canopy algorithm and the K-means algorithm as examples, the initial center point can be regarded as the initial clustering center of the Canopy algorithm. Clustering processing is performed based on the initial center point, the preset clustering algorithm and the first data set to determine the target clustering result.

[0081] It should be noted that in order to improve the clustering accuracy, the present invention targets two representative algorithms, K-means and Canopy, and combines the basic ideas of the two clustering algorithms to propose an improved data mining method to solve the technical problems that the existing server log mining algorithm is difficult to determine the initial clustering center, and the clustering accuracy and efficiency are generally.

[0082] Among them, the K-means algorithm has two unavoidable shortcomings:

[0083] 1) The change of k value has a great impact on the clustering results, and enumerating multiple k values ​​for algorithm calculations and comparing the results will consume too much time;

[0084] 2) The initial input of K-means also requires the center vector point of the k initial data vector point set, which is also uncertain, and different initial center vector point sets will also cause a large change in the final calculation results;

[0085] From the common Canopy algorithm, we can know that due to the low time complexity of the algorithm, it can provide the center vector point set and k value of the initial data vector point set for the K-means algorithm. However, the algorithm itself also has some defects that have a significant impact on the results:

[0086] First, the threshold value is determined based on operator experience or multiple algorithm runs, and these operations need to be performed for different data sets, which consumes a lot of time and manpower;

[0087] Secondly, the initial vector points of the Canopy algorithm are random, so it takes a lot of time to enumerate and calculate.

[0088] Therefore, the present invention designs an optimization algorithm based on the Canopy concept to solve the above two defects, and also provides a traceable clustering initialization center vector point set and k value for the subsequent parallelization algorithm.

[0089] Specifically, the clustering algorithm of this embodiment includes the Canopy algorithm and the K-means algorithm, and the clustering process based on the initial center point, the preset clustering algorithm and the first data set is performed to determine the target clustering result, including steps C01 and C02:

[0090] C01. Perform Canopy clustering using the initial center point and the first data set to obtain a first clustering result, where the first clustering result includes a plurality of clusters and cluster centers corresponding to each of the clusters;

[0091] Specifically, performing Canopy clustering using the initial center point and the first data set to obtain a first clustering result includes:

[0092] D01, taking the first average distance AvgDist(N) as an initial threshold value T1;

[0093] D02, traverse the first data set N, determine the initial center point R i A second distance between each sample data in the first data set N;

[0094] D03: If there is second sample data whose second distance is less than the initial threshold value T1, then the second sample data is added to the cluster with the initial center point as the cluster center. The second sample data is deleted from the first data set N to obtain an updated first data set N; the initial center point is added to the preset center point set Z K In the above example, we get the updated center point set Z K ;

[0095] D04, calculate the relationship between each sample data in the updated first data set N and the updated center point set Z K The third distance between the center points in ;

[0096] D05. Use the third sample data corresponding to the maximum value of the minimum third distances corresponding to each of the center points as the next cluster center, and based on the third sample data and the updated first data set, return to execute D02 to traverse the first data set and determine the second distance between the initial center point and each sample data in the first data set until the first data set is empty, thereby obtaining a plurality of clusters and the cluster centers corresponding to each of the clusters.

[0097] Assume that the sample data R i As the initial center point, AvgDist(N) is set to the initial threshold value T1 through step D01. The distance Dist(R) between the center point and other points is calculated by using formula (3) through step D02. i ,R j ), by comparing Dist(R i ,R j ) and T1, and the points smaller than T1 are classified into the group with R in step D03. i The center point is in the Canopy set, and it is deleted from the sample data set N and put into the set C Ri It can be understood that a cluster can be regarded as a set, and the clustering result includes each cluster of the first data set, so the cluster can also be regarded as a subset of the first data set.

[0098] After completing the preliminary clustering based on the initial center point, the “min-max principle” is used to select the next center point. Assuming that m center points have been calculated, the Euclidean distance between each sample data in N and all center points is calculated in step D04. At this time, the Euclidean distance between any sample data in N and Z K The Euclidean distance between any two center points can also be calculated based on the calculation method of formula (3), which will not be described in detail here. In step D05, the minimum distance is selected from these distances, all minimum distances are formed into a set, and the maximum value (that is, the maximum value in the minimum third distance) is selected from them, and the corresponding sample data is used as the next Canopy center point, which is specifically expressed as follows:

[0099]

[0100] Among them, DistCollect(x) is the minimum value of the distance between the sample data in N and all Canopy center points, Dist min (m+1) is the maximum value among all DistCollect(x) obtained, and its corresponding sample data is used as the center point of the next Canopy group.

[0101] According to the above-mentioned Canopy optimization method based on the "min-max principle", there is no need to set the value of threshold T2, and the next Canopy group center point can be directly obtained.

[0102] Repeat steps D02 to D05 until the number of data in the sample data set N is 0, and output the Canopy set and its center point.

[0103] In a feasible implementation, in order to improve the accuracy of the mining sample data, denoising processing may also be performed, wherein Canopy clustering is performed using the initial center point and the first data set to obtain a first clustering result, and then the following is further included:

[0104] E01. Determine the sample data volume of each cluster;

[0105] E02, if the sample data volume is less than a preset data volume threshold, determining that the cluster is a noise set;

[0106] E03. Eliminate the noise set from the first clustering result to obtain an updated first clustering result.

[0107] The sample data volume is the total number of sample data in the cluster, and the data volume threshold can be calculated by the average value of the data volume. And standard deviation σ, for example, the data volume threshold can be

[0108] For example, the average number of sample data in the set And the standard deviation σ is expressed as follows:

[0109]

[0110] Where n is the total number of sample data, p is the number of Canopy sets (clustering clusters), is the average number of sample data contained in each Canopy set. i is the number of sample data contained in the i-th Canopy set, and σ is the standard deviation of the number of sample data contained in each Canopy set.

[0111] The sample data volume is less than The set is regarded as a noise data set that is not universal, and the elimination of the noise data set is regarded as a denoising process.

[0112] C02. Perform K-means clustering on the first data set by taking the total number of the clusters as the K value and the cluster center as the initial centroid to obtain a second clustering result, wherein the target clustering result includes the second clustering result.

[0113] It can be understood that after denoising, the number of Canopy sets after denoising is used as the K value, and the center point of the Canopy set is used as the initial centroid to perform K-means clustering on the sample data set N to obtain the optimized clustering result A1, which is the second clustering result mentioned above.

[0114] Furthermore, based on the clustering result A1, all sample data R′ in each cluster can be clustered again according to the process from 102 to 105, using the service node name i1 of the data record I as an identifier and the timestamp i2 as a data item. The optimized clustering result A2 of the operation status of each service node in each cluster of A1 according to time division is obtained, indicating which service nodes have similar performance in similar time periods. The target clustering result can also include the clustering result A2. In addition, in addition to data mining for clustering sample data, data mining for performance evaluation of sample data can also be performed based on the clustering results, that is, clustering results A1 and A2 can also be used to evaluate the performance of service nodes and guide performance optimization, so as to make in-depth mining of the server's operation log.

[0115] In a feasible implementation, the method further comprises step F01:

[0116] F01. Perform performance evaluation on the server based on the target clustering result and preset server performance evaluation rules to determine a performance evaluation result of the server, where the performance evaluation result is used to reflect the operating status of the server.

[0117] It should be noted that the server performance evaluation rules include at least performance indicator algorithms such as service node performance-specific evaluation and service node overall performance evaluation.

[0118] Exemplary, (1) Special evaluation of service node performance:

[0119] Define σ ij is any sample data set N in the clustering result A1 i The standard deviation of the data in the jth dimension, where N i It can be regarded as the i-th cluster, then:

[0120]

[0121] Among them, c i N i The number of sample data in d jn N i The j-th dimension data of the n-th sample data. N i The average value of the j-th dimension data of all sample data in .

[0122] Definitions σ is the standard deviation σ ij The score is:

[0123]

[0124] From the definition, we can see that when σ ij The more hours σ The larger the value, the greater the ij =0 means the standard deviation is 0 and the data is stable. σ The maximum value is 1.

[0125] definition N i The median of the j-th dimension of all sample data in When N i Most of the internal service nodes do not involve the j-th dimension data during operation and do not participate in the evaluation.

[0126] Definitions ij N i The score of the j-th dimension data is:

[0127]

[0128] From the definition, we can see that when s σ When the value is larger, the data standard deviation is smaller, indicating that the service performance is more stable; when The smaller the value, the smaller the resource usage of the service. ij The larger the value, the higher the score, indicating N i The performance of the j-th dimension of internal services is excellent.

[0129] By comparing the scores of the same dimension of each cluster in A1, we can compare the performance advantages of different clusters. By analyzing the advantages, we can identify the performance pattern of the server and then perform special optimization on the server with low scores in a single dimension.

[0130] (2) Overall performance evaluation of service nodes:

[0131] Taking the electricity consumption information collection system as an example, the weights are divided according to the importance of resources occupied by the electricity consumption information collection system. The division results are shown in Table 2 below:

[0132] Table 2 Resource weight division table of electricity consumption information collection system

[0133]

[0134] The scores of each dimension are graded based on experience. The results are shown in Table 3 below:

[0135] Table 3 Score intervals for each dimension

[0136]

[0137] definition N i The overall score is:

[0138]

[0139] Among them, ω ij N i The resource weight value corresponding to the j-th dimension data; e j N i The score s of the j-th dimension data ij The corresponding score in the interval.

[0140] From the definition, it can be seen that when The larger the N i The higher the overall score, the better the performance of the service nodes in the cluster. Combined with the A2 clustering results, the performance of a service node in a certain period of time can be obtained, and other service nodes with the same performance in the same period of time can be quickly found. According to the analysis results, the performance of the service node can be optimized by means such as exchanging the operating time period of the service node with poor performance.

[0141] The clustering results of the present invention can help discover abnormal patterns of system servers. For example, if the performance indicators of some servers are significantly different from those of most servers, this may indicate potential hardware problems or configuration errors; the clustering results can be used to assist in elastic capacity planning and predictive operation and maintenance. Cluster analysis can reveal long-term trends in server performance, assist in predicting when resources need to be expanded to cope with increased load, and help predict future performance issues and hardware failures, so that maintenance and upgrades can be performed in advance. It can also help operation and maintenance personnel understand the performance of different servers, thereby conducting cost-benefit analysis and evaluating the efficiency of server resource use, so as to strike a balance between cost and performance, reduce unnecessary waste of resources, and optimize resource investment.

[0142] Furthermore, the present invention corrects the operation log data of the power consumption information collection system, uses the processed data as a sample set, clusters it using an improved Canopy algorithm, and performs K-means clustering with the clustering division result corresponding to the cluster number as the target, thereby determining the operation log similarity index. A system operation time-space distribution portrait is generated based on the index and the target clustering division result.

[0143] The present invention uses the Canopy-K-means algorithm to cluster the sample set, which can improve the overall efficiency of the clustering algorithm and solve the problem that the initial cluster center is difficult to determine. The optimal number of clusters is determined by the value of each clustering effectiveness index, which can effectively improve the clustering accuracy.

[0144] The present invention provides a data mining method for operation logs, the method comprising: obtaining a first data set corresponding to the operation log of a server, the first data set comprising a number of sample data to be mined, the sample data being used to reflect the operation status of the server; using a first distance between the sample data, a first average distance of the first data set and a preset adjacent sample selection rule to determine a second data set corresponding to each sample data, the second data set being used to reflect adjacent sample data corresponding to the sample data; determining target parameters of each second data set according to the data density of the second data set and the second average distance, the target parameters being used to reflect the density distribution characteristics of the second data set; using each target parameter and a preset center selection rule to perform initial center point selection processing for clustering, and determining the initial center point; performing clustering processing based on the initial center point, a preset clustering algorithm and the first data set, and determining the target clustering result. Through the above-mentioned method, the initial center point is determined by using the target parameter reflecting the density distribution characteristics of the second data set, and the initial center point can be determined without enumeration, thereby improving clustering efficiency.

[0145] See also Figure 2 , Figure 2 FIG. 1 is a structural block diagram of a data mining device for operation logs in an embodiment of the present invention. Figure 2 The device shown comprises:

[0146] Data acquisition module 201: used to acquire a first data set corresponding to the operation log of the server, wherein the first data set includes a number of sample data to be mined, and the sample data is used to reflect the operation status of the server;

[0147] The adjacent sample determination module 202 is used to determine the second data set corresponding to each of the sample data by using the first distance between the sample data, the first average distance of the first data set and the preset adjacent sample selection rule, wherein the second data set is used to reflect the adjacent sample data corresponding to the sample data;

[0148] Parameter determination module 203: used to determine target parameters of each second data set according to the data density of the second data set and the second average distance, wherein the target parameters are used to reflect the density distribution characteristics of the second data set;

[0149] Center determination module 204: used to select the initial center point of clustering by using each of the target parameters and the preset center selection rule to determine the initial center point;

[0150] Clustering processing module 205: used to perform clustering processing based on the initial center point, a preset clustering algorithm and the first data set to determine a target clustering result.

[0151] In a feasible implementation, the device further includes:

[0152] Denoising module: used to determine the sample data volume of each cluster; if the sample data volume is less than a preset data volume threshold, determine the cluster as a noise set; remove the noise set from the first clustering result to obtain an updated first clustering result.

[0153] In a feasible implementation, the device further includes:

[0154] Performance evaluation module: used to perform performance evaluation on the server based on the target clustering result and the preset server performance evaluation, and determine the performance evaluation result of the server, and the performance evaluation result is used to reflect the operating status of the server.

[0155] Understandably, Figure 2 The contents of each module in the device are Figure 1 The contents of each step in the method are similar, and are not described here to avoid repetition. For details, please refer to Figure 1 The content of each step in the method.

[0156] The present invention provides a data mining device for operation logs, the device comprising: a data acquisition module: used to acquire a first data set corresponding to the operation log of a server, the first data set comprising a plurality of sample data to be mined, the sample data being used to reflect the operation status of the server; an adjacent set determination module: used to determine a second data set corresponding to each sample data by using a first distance between sample data, a first average distance of the first data set and a preset adjacent sample selection rule, the second data set being used to reflect adjacent sample data corresponding to the sample data; a parameter determination module: used to determine a target parameter of each second data set according to the data density of the second data set and the second average distance, the target parameter being used to reflect the density distribution characteristics of the second data set; a center determination module: used to perform initial center point selection processing for clustering by using each target parameter and a preset center selection rule, and determine the initial center point; a cluster processing module: used to perform cluster processing based on the initial center point, a preset clustering algorithm and the first data set, and determine the target clustering result. Through the above-mentioned method, the initial center point is determined by using the target parameter reflecting the density distribution characteristics of the second data set, and the initial center point can be determined without enumeration, thereby improving the clustering efficiency.

[0157] Figure 3FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 3 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the above method. Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0158] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps: Figure 1 Steps shown.

[0159] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor performs the following steps: Figure 1 Steps shown.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data mining method for operation logs, characterized in that: The method comprises: Acquire a first data set corresponding to the operation log of the server, wherein the first data set includes a plurality of sample data to be mined, and the sample data is used to reflect the operation status of the server; Determine a second data set corresponding to each of the sample data by using the first distance between the sample data, the first average distance of the first data set, and a preset adjacent sample selection rule, wherein the second data set is used to reflect adjacent sample data corresponding to the sample data; determining a target parameter of each of the second data sets according to the data density of the second data sets and the second average distance, wherein the target parameter is used to reflect a density distribution feature of the second data sets; Using each of the target parameters and the preset center selection rule to perform initial center point selection processing for clustering, and determine the initial center point; Clustering processing is performed based on the initial center point, a preset clustering algorithm and the first data set to determine a target clustering result.

2. The method according to claim 1, characterized in that: The target parameter is the ratio of the data density to the second average distance, and the initial center point selection process of clustering is performed using each of the target parameters and a preset center selection rule to determine the initial center point, including: Determine a target second data set corresponding to a maximum value among the ratios; The first sample data corresponding to the target second data set is used as the initial center point.

3. The method according to claim 1, characterized in that: The clustering algorithm includes a Canopy algorithm and a K-means algorithm, and the clustering process is performed based on the initial center point, the preset clustering algorithm and the first data set to determine the target clustering result, including: Performing Canopy clustering using the initial center point and the first data set to obtain a first clustering result, wherein the first clustering result includes a plurality of clusters and cluster centers corresponding to each of the clusters; K-means clustering is performed on the first data set using the total number of the clusters as the K value and the cluster center as the initial centroid to obtain a second clustering result, and the target clustering result includes the second clustering result.

4. The method according to claim 2, characterized in that: The using the initial center point and the first data set to perform Canopy clustering to obtain a first clustering result includes: Using the first average distance as an initial threshold value; Traversing the first data set, and determining a second distance between the initial center point and each sample data in the first data set; If there is second sample data whose second distance is less than the initial threshold value, the second sample data is added to the cluster with the initial center point as the cluster center, and the second sample data is deleted from the first data set to obtain an updated first data set; the initial center point is added to the preset center point set to obtain an updated center point set; Calculating a third distance between each sample data in the updated first data set and each center point in the updated center point set; The third sample data corresponding to the maximum value of the minimum third distances corresponding to each of the center points is used as the next cluster center, and based on the third sample data and the updated first data set, the step of traversing the first data set to determine the second distance between the initial center point and each of the sample data in the first data set is returned to be executed until the first data set is empty, thereby obtaining a plurality of clusters and the cluster centers corresponding to each of the clusters.

5. The method according to claim 3 or 4, characterized in that: The method further comprises: performing Canopy clustering using the initial center point and the first data set to obtain a first clustering result; and then: Determine the amount of sample data for each cluster; If the sample data volume is less than a preset data volume threshold, determining that the cluster is a noise set; The noise set is removed from the first clustering result to obtain an updated first clustering result.

6. The method according to claim 1, characterized in that: The step of obtaining a first data set corresponding to the operation log of the server includes: Obtaining multi-dimensional original operation log data of the server; Using a preset multi-dimensional data cleaning rule to clean the original operation log data, to obtain cleaned operation log data; The cleaned operation log data is sorted according to preset data sorting rules to obtain a number of sample data to be mined.

7. The method according to claim 1, characterized in that: After the method, it also includes: Based on the target clustering result and the preset server performance evaluation, a performance evaluation is performed on the server to determine a performance evaluation result of the server, and the performance evaluation result is used to reflect the operating status of the server.

8. A data mining device for operation logs, characterized in that: The device comprises: Data acquisition module: used to acquire a first data set corresponding to the operation log of the server, wherein the first data set includes a number of sample data to be mined, and the sample data is used to reflect the operation status of the server; An adjacent set determination module: used to determine a second data set corresponding to each of the sample data by using the first distance between the sample data, the first average distance of the first data set and a preset adjacent sample selection rule, wherein the second data set is used to reflect adjacent sample data corresponding to the sample data; A parameter determination module: used to determine a target parameter of each of the second data sets according to the data density of the second data sets and the second average distance, wherein the target parameter is used to reflect the density distribution characteristics of the second data sets; Center determination module: used to select the initial center point of clustering by using each of the target parameters and the preset center selection rules to determine the initial center point; Clustering processing module: used to perform clustering processing based on the initial center point, a preset clustering algorithm and the first data set to determine a target clustering result.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images