Initialization kmeans log classification method based on distributed split fusion

Abstract

The application provides a kind of initialization kmeans log classification method based on distributed split fusion, it is related to big data field.The method comprises: obtaining N log sample sets and the corresponding copy of the log sample set;Determine K centers in each log sample set;And the cluster division is carried out to the log sample in the copy of the log sample set, obtain the K clusters of the copy;Obtain the error function of each cluster in each copy, according to the error function of the cluster, the cluster is split processed;Get the cluster after splitting processing, form cluster set;According to the cosine distance between the cluster center of each cluster in cluster set, each cluster is fused until the distance of the cluster center of all clusters meets the preset condition;Obtain the cluster center in the cluster set as initial centroid, and carry out Kmeans clustering.The method improves the clustering speed and effect of kmeans clustering algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a distributed splitting and fusing based initialization kmeans log classification method, device and equipment and storage medium. BACKGROUND

[0002] In the process of using an application program, errors may occur in the application program due to the personal habits of users, and these errors are recorded in the log of the application program. When analyzing the error problems occurring in the process of using the application program by each user, logs of the same error type need to be divided into a group, and then the possible error methods of the habits of the users are excavated in a classified manner, so as to give targeted suggestions for similar problems in the future.

[0003] At present, the existing technology usually adopts a k-means clustering algorithm (kmeans clustering algorithm for short) for text clustering. In the initialization stage, one step is to divide data into K groups, then randomly select an object in each group as the initial clustering center of the group, and then calculate the distance between each object and each seed clustering center, and assign each object to the nearest clustering center.

[0004] However, the existing K-means clustering algorithm randomly extracts an initial centroid in the initialization stage, which may result in slow clustering convergence speed and poor clustering effect. SUMMARY

[0005] The present application provides a distributed splitting and fusing based initialization kmeans log classification method to solve the problem of slow clustering speed and poor effect of the existing kmeans clustering algorithm.

[0006] In a first aspect, the present application provides a distributed splitting and fusing based initialization kmeans log classification method, which comprises the following steps:

[0007] Obtain N log sample sets, back up each log sample set to obtain a copy corresponding to the log sample set, wherein the log sample set includes at least one log sample, and N is a positive integer;

[0008] Screen and delete the log samples in each log sample set to determine K centers in each log sample set, wherein K is a positive integer;

[0009] Determine K centers in the copy of each log sample set and K clusters of the copy according to the K centers in each log sample set;

[0010] obtain an error function of each cluster in each copy, and perform splitting processing on the cluster according to the error function of the cluster;

[0011] obtain the clusters after the splitting processing, and form a cluster set;

[0012] fuse the clusters according to the cosine distance between the cluster centers of the clusters in the cluster set until the distance between the cluster centers of all the clusters satisfies a preset condition;

[0013] obtain the cluster centers in the cluster set as initial centroids, and perform Kmeans clustering.

[0014] In a second aspect, the present application provides an initialization kmeans log classification device based on distributed splitting and fusing, comprising:

[0015] an obtaining module, configured to obtain N log sample sets, back up each log sample set to obtain a copy corresponding to the log sample set, wherein the log sample set comprises at least one log sample, and N is a positive integer;

[0016] a center determining module, configured to filter and delete log samples in each log sample set, and determine K centers in each log sample set, wherein K is a positive integer;

[0017] a cluster dividing module, configured to determine K centers in the copy of the log sample set and K clusters of the copy according to the K centers in each log sample set;

[0018] a cluster splitting module, configured to obtain an error function of each cluster in each copy, and perform splitting processing on the cluster according to the error function of the cluster;

[0019] a cluster set module, configured to obtain the clusters after the splitting processing, and form a cluster set;

[0020] a cluster fusing module, configured to fuse the clusters according to the cosine distance between the cluster centers of the clusters in the cluster set until the distance between the cluster centers of all the clusters satisfies a preset condition;

[0021] a clustering module, configured to obtain the cluster centers in the cluster set as initial centroids, and perform Kmeans clustering.

[0022] In a third aspect, the present application provides an electronic device, comprising a processor and a memory connected with the processor in communication; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory to realize the method as described above.

[0023] In a fourth aspect, the present application provides a computer readable storage medium, wherein computer execution instructions are stored in the computer readable storage medium, and the computer execution instructions are used to implement the method as described above when executed by a processor.

[0024] In a fifth aspect, the present application provides a computer program product, which is used to implement the method as described above when executed by a processor.

[0025] The initialization kmeans log classification method based on distributed splitting and fusing provided by the present application can distribute and process the massive log samples by dividing the log samples into N different log sample sets, reduce the initialization time, and perform splitting and fusing processing on the clusters in the log sample set, so that the initial centroid can be more accurately selected, the improvement of the initialization stage of the kmeans clustering algorithm is realized, the clustering effect and the convergence speed can be improved, and the time spent in clustering can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] Figure 1 The clustering effect diagram provided for the embodiments of the present application;

[0028] Figure 2 The flowchart of the initialization kmeans log classification method based on distributed splitting and fusing provided for the embodiments of the present application;

[0029] Figure 3 The effect diagram of the cluster division of the log samples provided for the embodiments of the present application;

[0030] Figure 4 The overall flowchart of the initialization kmeans log classification method based on distributed splitting and fusing provided for the embodiments of the present application;

[0031] Figure 5 The structure diagram of the initialization kmeans log classification device based on distributed splitting and fusing provided for the embodiments of the present application;

[0032] Figure 6 The structure diagram of the electronic device provided for the embodiments of the present application.

[0033] The specific embodiments of the present application have been shown in the above-described drawings, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0034] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements throughout the description. The following exemplary embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0035] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0036] It should be noted that the initialization kmeans log classification method, device and equipment based on distributed split fusion and storage medium provided by the present application can be used in the field of big data technology, and can also be used in any field other than the field of big data technology. The application field of the initialization kmeans log classification method, device and equipment based on distributed split fusion and storage medium provided by the present application is not limited.

[0037] K-means clustering algorithm: a clustering algorithm that can be used for text clustering, which has a step in the initialization stage, that is, to divide the data into K groups, then randomly select an object in each group as the initial cluster center of the group, then calculate the distance between each object and each seed cluster center, and assign each object to the cluster center closest to it.

[0038] The K-means clustering algorithm has the following disadvantages: (1) the selection of the centroid in the initialization stage has randomness, and the initial centroid is too close or scattered, which is not uniform enough, which will lead to slow convergence speed of the algorithm. (2) The initial centroid selected to the discrete point or noise data is not conducive to the convergence of subsequent clustering, resulting in unsatisfactory clustering effect of the algorithm. (3) The larger the data, the more obvious the defect, and the initialization time is very long.

[0039] To address the aforementioned issues, this application provides a method, apparatus, device, and storage medium for initializing k-means log classification based on distributed split-fusion. It improves the initialization method for massive log samples by processing them in a distributed and parallel manner, reducing initialization time. Furthermore, it improves the initialization phase of the k-means algorithm by proposing a split-fusion method for selecting initial centroids. This ensures that the samples are divided into several clusters with significantly different similarities and sufficient compactness. The final selected cluster centers, used as initial centroids, are sufficiently evenly distributed, facilitating subsequent clustering processes and improving the quality of clustering results.

[0040] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0041] For example, Figure 1 Clustering effect diagrams provided for embodiments of this application, such as Figure 1 As shown, taking 20 sample logs (R1 to R20) as an example, R1 to R8 are clustered into one cluster, and R9 to R20 are clustered into another. If there is an outlier that does not belong to either of the above two clusters, then the sample log represented by that outlier can be identified as an anomalous log. Here, "log" can be a term from the computer science field; application platforms generate logs during operation; each log line records a description of the date, time, user, and action.

[0042] Figure 2 This is a flowchart illustrating the initial k-means log classification method based on distributed splitting and fusion provided in this application embodiment. This method can be applied to text clustering scenarios, clustering massive amounts of historical error logs into multiple groups. After obtaining initial centroids from the large amount of input log data using this algorithm, K-means clustering is then applied to quickly obtain K log groups with good clustering results. By analyzing and organizing the error logs of each group, incorrect user habits during application usage can be identified, facilitating the provision of corresponding solutions. Figure 2 As shown, the method may specifically include the following steps: Step 201, obtain N log sample sets, back up each log sample set to obtain a copy of the log sample set, wherein the log sample set includes at least one log sample, and N is a positive integer.

[0043] In the embodiment, the log samples included in each log sample set can be different. For example, the log samples included in the log sample set R1 are R11, R12 and R13, and the log samples included in the log sample set R2 are R21, R22 and R23.

[0044] At least one log sample is included in each log sample set. Before the subsequent steps, a corresponding copy of each log sample set can be obtained by copying, which is completely the same as the log sample set. The log sample set can be changed during subsequent processing. The copy can be kept intact. For example, when the log sample set is filtered and deleted, K centers are determined. Based on the K centers, the K centers in the copy are found. Subsequently, the copy can be directly split and fused without restoring the log sample set.

[0045] In the embodiment, the massive error logs generated by the application program can be randomly divided into different sets, thereby forming the N log sample sets in the embodiment.

[0046] In step S202, the log samples in each log sample set are filtered and deleted, and K centers in each log sample set are determined, K being a positive integer.

[0047] In the embodiment, there can be multiple log samples in each log sample set. Based on the log samples, a center position can be calculated. The log samples close to or at the center position can become the center. The log samples in the log sample set can be regionally filtered, for example, divided into regions P1, P2 and P3. Thus, there is a center in region P1, a center in region P2, and a center in region P3.

[0048] In step S203, according to the K centers in each log sample set, K centers in the copy of the log sample set and K clusters of the copy are determined.

[0049] In the embodiment, each log sample set has K centers (each center can be understood as a log sample). In addition, there can be log samples other than the centers in the log sample set. These log samples can be clustered based on the K centers to obtain K clusters.

[0050] For example, when the log sample set Z1 has 1 center (for example, the log sample Z11 is the center), if the log sample Z14 and the log sample Z15 are near the center Z11, the log samples Z14, Z11 and Z15 are divided into the same cluster.

[0051] For example, Figure 3 The log sample diagram provided by the embodiment of the present application is shown in the following figure: Figure 3 As shown in the figure, by digitizing the log samples, each log sample is replaced by a coordinate point. R11 and R12 are selected as the centers. The log sample R13 is near the center R12, and the log samples R13 and R12 are divided into the first cluster. The log samples R14 and R15 are near the center R11, and the log samples R14, R11 and R15 are divided into the second cluster.

[0052] In step S204, the error function of each cluster in each replica is obtained, and the cluster is split according to the error function of the cluster.

[0053] In the embodiment, errors may occur when the clusters are divided. For example, some log samples should be divided into the first cluster, but are actually divided into the second cluster. Therefore, the error function of each cluster needs to be calculated. If the error function of the cluster is too large, the cluster needs to be split. Or the log samples in a cluster are too scattered, resulting in a too large error function. At this time, the cluster needs to be split. Splitting means that a cluster is split into two clusters.

[0054] In step S205, the clusters after the splitting are obtained to form a cluster set.

[0055] In the embodiment, some clusters may be split into two clusters due to a too large error function, and some clusters may not be split. Some clusters after splitting may be split again. By counting all these clusters, a cluster set is formed. It can be understood that the cluster set can include multiple clusters, and the number of clusters is greater than or equal to K.

[0056] In step S206, the clusters are fused according to the cosine distance between the cluster centers of the clusters in the cluster set until the distance between the cluster centers of all clusters meets a preset condition.

[0057] In the embodiment, each cluster has a log sample at the center, which is the cluster center. Each log sample can be converted into a vector form. Similarly, the cluster center of the cluster can also be converted into a vector form. Then, the cosine distance between the cluster centers is calculated through the vector form of the cluster center. For example, the above-mentioned Figure 3Taking the cluster center R11 and the cluster center R12 as an example, the closeness between the cluster centers can be represented by the cosine distance.

[0058] If the cosine distance between the cluster center R11 and the cluster center R12 is less than a certain set value, the first cluster and the second cluster can be fused, that is, the log samples contained in the first cluster and the second cluster are divided into a new cluster, and the cluster center of the new cluster is re-determined, and the original cluster center R11 and the cluster center R12 are no longer used as the cluster center.

[0059] In this embodiment, the preset condition can be a cosine distance threshold. For example, when the cosine distance between the cluster centers of all clusters is greater than the cosine distance threshold, the clusters are no longer fused.

[0060] In step S207, the cluster centers in the cluster set are obtained as initial centroids, and Kmeans clustering is performed.

[0061] In this embodiment, the K-means clustering algorithm has been introduced above. In the beginning, K objects are randomly selected as initial cluster centers. In this embodiment, the selected cluster centers (i.e., initial centroids) are used to replace the step of randomly selecting K objects in the K-means clustering algorithm, so that the convergence speed of the algorithm can be improved and the clustering effect can be improved.

[0062] In this embodiment, the massive log samples are divided into N different log sample sets, and the log samples are processed in a distributed and parallel manner to reduce the initialization time and select the initial centroids, thereby improving the initialization stage of the k-means clustering algorithm, improving the clustering effect and convergence speed, and reducing the time spent on clustering.

[0063] In some embodiments, the above step S201 can be implemented by the following steps: obtaining an initial log set, the initial log set including at least one log sample; and dividing the initial log set to obtain N log sample sets.

[0064] In this embodiment, the massive log samples generated by user error behaviors can be used as an initial log set, and the initial log set is divided into N initial log sets, that is, dataset1, dataset2, …, and datasetN. Each log sample set can be copied as a copy.

[0065] The embodiment of the application can realize distributed processing by dividing the initial log set into N log sample sets, simultaneously processing multiple log sample sets, thereby quickly selecting the initial centroid for the Kmeans clustering algorithm, and reducing the initialization time.

[0066] In some embodiments, the above step S202 can be implemented by the following steps: randomly selecting a first sample Cn1 from the log sample set, deleting first log samples in the log sample set with a cosine distance less than a first preset threshold from the first sample Cn1; obtaining all the deleted first log samples, and determining the center of all the deleted first log samples as a first center; randomly selecting a second sample Cn2 from the log sample set, deleting second log samples in the log sample set with a cosine distance less than the first preset threshold from the second sample Cn2; obtaining all the deleted second log samples, and determining the center of all the deleted second log samples as a second center; randomly selecting a Kth sample Cnk from the log sample set, deleting Kth log samples in the log sample set with a cosine distance less than the first preset threshold from the Kth sample Cnk; obtaining all the deleted Kth log samples, and determining the center of all the deleted Kth log samples as a Kth center.

[0067] In the embodiment, multiple MAP functions can be set, each of which can process a log sample set. The log sample set input into a MAP function is denoted as dataset1 (dataset1 is taken as an example, and the rest dataseti are processed according to the idea described below), and the copy is dataset_bak1. A first sample Cn1 is randomly selected from dataset1, first log samples in dataset1 with a cosine distance less than a first preset threshold T1 from the first sample Cn1 are deleted, and the center of the first log samples is denoted as a first center C1. Then a second sample Cn2 is randomly selected from dataset1, second log samples in dataset1 with a cosine distance less than the first preset threshold T1 from the second sample Cn2 are deleted, and the center of the second log samples is denoted as a second center C2. In this way, a Kth center Ck is obtained.

[0068] The embodiment of the application can improve the processing speed and shorten the processing time by setting multiple MAP functions, each of which processes each log sample set. Meanwhile, the K centers can be accurately obtained by continuously screening the log samples in the log sample set according to the cosine distance, thereby ensuring the accuracy of the subsequent initial centroid selection and improving the clustering effect of Kmeans.

[0069] In some embodiments, the step S203 can be implemented by the following steps: determining K centers in the copy of the log sample set according to the K centers in each log sample set; calculating the cosine distance between each log sample in the copy of the log sample set and each center in the copy; dividing the log sample with the closest cosine distance to the Kth center and the Kth center in the copy into the same cluster to obtain K clusters of the copy.

[0070] In the embodiment, since the copy is a backup of the log sample set, after the K centers are determined in the log sample set, the corresponding centers can be found in the copy. For example, the log sample set includes centers C1 and C2, and the corresponding centers C1 and C2 can be found in the copy. The log sample and the center can be divided into the same cluster according to the cosine distance between each log sample and each center.

[0071] For example, the log sample R13 and the center R12 form a cluster, and the log sample R14 and the log sample R15 and the center R11 form a cluster. Figure 3

[0072] The embodiment of the application can divide the log sample into the corresponding cluster according to the cosine distance, can avoid the log sample far from the center from being divided into the same cluster with other centers, and can effectively improve the accuracy of clustering and further improve the accuracy of initial centroid selection.

[0073] In some embodiments, the step S204 can be implemented by the following steps: calculating the sum of the square value of the cosine distance between each log sample and the cluster center in each cluster to obtain the error function of the cluster; comparing the error function of the cluster with the first preset threshold; if the error function of the cluster is greater than the first preset threshold, splitting the cluster to obtain two clusters after splitting; and if the error function of the cluster is less than or equal to the first preset threshold, keeping the cluster.

[0074] In the embodiment, the error function of each cluster can be calculated by the following formula:

[0075]

[0076] In the formula, Xi represents the i th log sample of the cluster A.

[0077] ​In the embodiment, if the error function of a certain cluster is greater than a preset first threshold T1, the cluster is split. Splitting means splitting a cluster into two clusters, and the log samples in the two split clusters are the log samples contained in the cluster. The two split clusters each have a cluster center.

[0078] The embodiment calculates the error function of each cluster. When the error function is large, it indicates that the log samples in the cluster are distributed more dispersedly and are not concentrated enough. At this time, splitting the cluster into two clusters can further ensure that the log samples in each cluster are more concentrated, and further improve the accuracy of clustering.

[0079] Further, on the basis of the above embodiment, in some other embodiments, the step "splitting the cluster to obtain two split clusters" in the above embodiment can be implemented by the following steps: calculating the density of each log sample in the cluster, and obtaining the log sample with the maximum density in the cluster; calculating the cosine distance between the log sample with the maximum density and other log samples in the cluster except the log sample with the maximum density, and obtaining the log sample with the farthest distance; and splitting the other log samples in the cluster into two different clusters according to the cosine distances between the other log samples in the cluster and the log sample with the maximum density and the log sample with the farthest distance.

[0080] In the embodiment, the splitting method is as follows: (1) calculating the density of all log samples in the cluster; (2) finding the log sample with the maximum density C1 in the cluster, and then calculating the cosine distance between the log sample with the maximum density C1 and other log samples in the cluster to find the log sample with the farthest distance C2. Then, the log sample with the maximum density C1 and the log sample with the farthest distance C2 can be taken as the cluster centers of the two clusters respectively, the cosine distances between all other log samples in the cluster to be split and the log sample with the maximum density C1 and the log sample with the farthest distance C2 are calculated, and the log samples are divided into the corresponding clusters based on the principle of the nearest cosine distance.

[0081] For example, the cluster to be split includes a log sample RD1. The cosine distance between the log sample RD1 and the log sample with the maximum density C1 is D1, and the cosine distance between the log sample RD1 and the log sample with the farthest distance C2 is D2, and D1 is less than D2. Therefore, the log sample RD1 and the log sample with the maximum density C1 are divided into one cluster.

[0082] The embodiment finds the log sample with the maximum density and the log sample with the farthest distance, and splits the cluster to be split by taking the log sample with the maximum density and the log sample with the farthest distance as the cluster centers of the two new clusters. This can make the log samples in the split clusters more concentrated, and improve the accuracy of splitting.

[0083] Further, on the basis of the above-mentioned embodiments, in some other embodiments, the step of "calculating the density of each log sample in the cluster" in the above-mentioned embodiments can be implemented through the following steps: obtaining a preset threshold Kn; obtaining a first set composed of Kn log samples closest to the target log sample in terms of cosine distance; obtaining a second set composed of Kn log samples closest to each log sample in the first set in terms of cosine distance; obtaining the union set of the first set and the second set as the second-order Kn neighbor of the target log sample; calculating the sum of the cosine distances between each log sample in the second-order Kn neighbor and the target log sample; and comparing the sum with Kn to obtain the density of the target log sample.

[0084] In the embodiment, the density of each log sample is calculated in the following manner:

[0085] Density of log sample X = Kn / (sum of cosine distances between the second-order Kn neighbor of X and X)

[0086] In the above formula, Kn is a preset threshold, the second-order Kn neighbor of X refers to a set composed of Kn neighbor samples closest to the log sample X (in terms of cosine distance), a set composed of Kn neighbor samples closest to each sample point in the set neighbour1 is neighbour2, and the union set of the set neighbour1 and the set neighbour2 is neighbour_total, i.e., the second-order Kn neighbor of X.

[0087] The embodiment of the application finds the log sample with the maximum density by calculating the density of each log sample, and then finds the log sample farthest away from the log sample with the maximum density based on the log sample with the maximum density, so as to ensure that the log samples in the cluster obtained by splitting are more concentrated, and further improve the splitting and accuracy of the cluster.

[0088] In some embodiments, the step S206 can be implemented through the following steps: comparing the cosine distance between the cluster centers of each cluster with a preset second threshold; and if the cosine distance between the cluster center of the first cluster and the cluster center of the second cluster is less than the preset second threshold, merging the first cluster and the second cluster into one cluster.

[0089] In the embodiment, the cluster merging can be implemented through a Reduce function. The idea of the Reduce function is as follows:

[0090] (1) Input all the clusters output by the above-mentioned Map functions, and denote the input as a set list_cu, wherein the elements in the set list_cu are clusters. Denote the number of clusters in the set list_cu as N_total.

[0091] (2) Calculate the cosine distance of the cluster centers of each cluster in list_cu two by two, and if the cosine distance of the cluster centers of two clusters is less than a preset second threshold T2, fuse the two clusters.

[0092] The fusion method is as follows:

[0093] 1.1, the cluster Cnew after the fusion of the cluster CA and the cluster CB is the center of all samples of the two clusters CA and CB;

[0094] 1.2, delete the two clusters that have been fused from the set list_cu, and put the new cluster after the fusion into the set list_cu. At this time, the number of clusters in the set list_cu is N_total-1;

[0095] 1.3, according to the above principle, calculate the cosine distance of the clusters in list_cu two by two again to find out the two clusters whose cosine distance is less than the preset threshold T2, and then delete the two clusters from list_cu, and then put the new cluster after the fusion of the two clusters into list_cu. Repeat this process until the distance between the clusters in list_cu meets the preset condition.

[0096] Finally, the Reduce function outputs all the cluster centers in list_cu, and takes the cluster centers in list_cu as the initial centroids to perform kmeans clustering.

[0097] Further, in other embodiments, the preset condition can be that the distance between the cluster centers of all clusters in the cluster set is greater than a preset second threshold T2.

[0098] The embodiments of the present application can avoid too few log samples covered by the cluster by further fusing the clusters with close distances, improve the clustering effect, and more accurately find the initial centroids, so that the effect of kmeans clustering based on the initial centroids is better.

[0099] Exemplarily, Figure 4 The overall flowchart of the initialization kmeans log classification method based on distributed splitting and fusion provided by the embodiments of the present application includes the following Map function processing stage and Reduce function processing stage. As shown in the figure, Figure 4 The Map function processing stage includes steps S4011, S4012,..., S401i (i takes a value of 1 to N). The Reduce function processing stage includes step S402.

[0100] The embodiment of the application improves the initialization mode of a mass of log samples, processes the log samples in a distributed and parallel manner, and reduces the initialization time. Meanwhile, the initialization stage of the kmeans algorithm is improved. In the initialization process, a splitting and fusing initialization cluster center selection method is proposed to ensure that the samples are divided into several clusters that are obviously different in similarity and are compact enough. The finally selected cluster centers are uniform and dispersed enough, which is beneficial to the subsequent clustering process and improves the quality of the clustering result.

[0101] The following is an apparatus embodiment of the application, which can be used to execute the method embodiments of the application. For details not disclosed in the apparatus embodiments of the application, refer to the method embodiments of the application.

[0102] Figure 5 The structure diagram of the initialization kmeans log classification device based on distributed splitting and fusing provided by the embodiment of the application is shown in FIG. 5. As shown in FIG. 5, the log classification device 500 includes an acquisition module 510, a center determination module 520, a cluster division module 530, a cluster splitting module 540, a cluster set module 550, a cluster fusing module 560, and a clustering module 570. Figure 5

[0103] The acquisition module 510 is configured to acquire N log sample sets, back up each log sample set to obtain a copy corresponding to the log sample set, and include at least one log sample in the log sample set. N is a positive integer. The center determination module 520 is configured to filter and delete the log samples in each log sample set, and determine K centers in each log sample set. K is a positive integer. The cluster division module 530 is configured to determine K centers in the copy of the log sample set and K clusters of the copy according to the K centers in each log sample set. The cluster splitting module 540 is configured to obtain an error function of each cluster in each copy, and split the cluster according to the error function of the cluster. The cluster set module 550 is configured to obtain the clusters after splitting to form a cluster set. The cluster fusing module 560 is configured to fuse the clusters according to the cosine distance between the cluster centers of the clusters in the cluster set until the distance between the cluster centers of all the clusters satisfies a preset condition. The clustering module 570 is configured to obtain the cluster centers in the cluster set as initial centroids, and perform Kmeans clustering.

[0104] Optionally, the acquisition module can be specifically configured to acquire an initial log set, and include at least one log sample in the initial log set; and divide the initial log set into N log sample sets.

[0105] ​Optionally, the center determining module can be specifically configured to: randomly select a first sample Cn1 from the set of log samples, delete first log samples in the set of log samples that have a cosine distance less than a first preset threshold from the first sample Cn1; obtain all the deleted first log samples, and determine a center of all the deleted first log samples as a first center; randomly select a second sample Cn2 from the set of log samples, delete second log samples in the set of log samples that have a cosine distance less than the first preset threshold from the second sample Cn2; obtain all the deleted second log samples, and determine a center of all the deleted second log samples as a second center; randomly select a Kth sample Cnk from the set of log samples, delete Kth log samples in the set of log samples that have a cosine distance less than the first preset threshold from the Kth sample Cnk; obtain all the deleted Kth log samples, and determine a center of all the deleted Kth log samples as a Kth center.

[0106] Optionally, the cluster dividing module can be specifically configured to: determine K centers in a copy of the set of log samples according to the K centers in each set of log samples; calculate a cosine distance between each log sample in the copy of the set of log samples and each center in the copy; divide a log sample in the copy that has a closest cosine distance from a Kth center and the Kth center into a same cluster, to obtain K clusters of the copy.

[0107] Optionally, the cluster splitting module can be specifically configured to: calculate a square value of a cosine distance between each log sample in each cluster and a center of the cluster and sum the square values, to obtain an error function of the cluster; compare the error function of the cluster with a first preset threshold; if the error function of the cluster is greater than the first preset threshold, split the cluster to obtain two clusters after splitting; if the error function of the cluster is less than or equal to the first preset threshold, retain the cluster.

[0108] Optionally, the cluster splitting module can be specifically configured to: calculate a density of each log sample in the cluster, obtain a log sample with a maximum density in the cluster; calculate a cosine distance between the log sample with the maximum density and other log samples in the cluster except the log sample with the maximum density, and obtain a log sample with a farthest distance; divide other log samples in the cluster into two different clusters according to cosine distances between the other log samples and the log sample with the maximum density and the log sample with the farthest distance.

[0109] Optionally, the cluster splitting module can be specifically configured to: acquire a preset threshold Kn; acquire a first set composed of Kn log samples closest to the target log sample in terms of cosine distance; acquire a second set composed of Kn log samples closest to each log sample in the first set in terms of cosine distance; acquire a union set of the first set and the second set as a second-order Kn neighbor of the target log sample; calculate and sum cosine distances between each log sample in the second-order Kn neighbor and the target log sample; and compare the Kn with the sum to obtain a density of the target log sample.

[0110] Optionally, the cluster fusion module can be specifically configured to: compare cosine distances between cluster centers of respective clusters with a preset second threshold; and if a cosine distance between a cluster center of a first cluster and a cluster center of a second cluster is less than the preset second threshold, fuse the first cluster and the second cluster into one cluster.

[0111] Optionally, the preset condition is that distances between cluster centers of all clusters in the cluster set are greater than the preset second threshold.

[0112] The apparatus provided by the embodiments of the present application can be used to execute the method in the above embodiments, and has similar implementation principles and technical effects, which will not be repeated here.

[0113] It should be noted that the division of each module of the above apparatus is only a logical function division, and all or part of the modules can be integrated into one physical entity, or can be physically separated. The modules can all be implemented in the form of software invoked by a processing element; or all be implemented in the form of hardware; or part of the modules are implemented in the form of software invoked by a processing element, and part of the modules are implemented in the form of hardware. For example, the acquisition module can be a separately established processing element, or can be integrated in a chip of the above apparatus, and in addition, the acquisition module can be stored in the form of program code in a memory of the above apparatus, and the functions of the acquisition module are invoked and executed by a processing element of the above apparatus. The implementation of other modules is similar. In addition, all or part of the modules can be integrated together, or can be independently implemented. The processing element herein can be an integrated circuit having a signal processing capability. In the implementation process, each step of the above method or each module can be completed by an integrated logic circuit of hardware or an instruction in the form of software in the processing element.

[0114] Figure 6 The structural schematic diagram of the electronic device provided by the embodiments of the present application is shown in FIG. 1. Figure 6As shown, the electronic device 600 includes at least one processor 601, a memory 602, a bus 603, and a communication interface 604. The processor, the communication interface, and the memory can communicate with each other through the bus. The communication interface is configured to communicate with other devices. The communication interface includes a communication interface configured to perform data transmission, and a display interface or an operation interface configured to perform human-computer interaction. The processor is configured to execute computer-executable instructions stored in the memory, and can perform the related steps in the methods described in the above embodiments.

[0115] The processor can be a central processing unit, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the electronic device can be the same type of processor, such as one or more CPUs; or different types of processors, such as one or more CPUs and one or more ASICs.

[0116] The memory is configured to store computer-executable instructions. The memory can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0117] The embodiments also provide a computer-readable storage medium, and the computer-readable storage medium stores computer instructions. When at least one processor of the electronic device executes the computer instructions, the electronic device executes the methods provided in the various embodiments described above.

[0118] The embodiments also provide a computer program product, and the computer program product includes computer instructions stored in a readable storage medium. At least one processor of the electronic device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to cause the electronic device to implement the methods provided in the various embodiments described above.

[0119] In the present application, "at least one" means one or more, "multiple" means two or more. "And / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship; in the formula, the character " / ", represents that the associated objects before and after are in a "division" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c, can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0120] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the serial number of the above processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0121] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the application are indicated by the following claims.

[0122] It should be understood that the present application is not limited to the precise construction that has been described and shown in the accompanying drawings, and that various modifications and changes can be affected therein without departing from the scope of the application. The scope of the application should only be limited by the appended claims.

Claims

1. A distributed split-and-merge based initialization kmeans log classification method, characterized in that, The method comprises the following steps: obtaining N log sample sets, and performing backup on each log sample set to obtain a copy corresponding to the log sample set, wherein the log sample set comprises at least one log sample, and N is a positive integer; performing screening and deletion on the log samples in each log sample set to determine K centers in each log sample set, wherein K is a positive integer; determining K centers in the copy of each log sample set and K clusters in the copy according to the K centers in each log sample set; obtaining an error function of each cluster in each copy, and performing splitting processing on the cluster according to the error function of the cluster; the splitting processing on the cluster according to the error function of the cluster comprises: calculating the square value of the cosine distance between each log sample in each cluster and the center of the cluster and summing up to obtain the error function of the cluster; if the error function of the cluster is greater than a first preset threshold, calculating the density of each log sample in the cluster, obtaining a log sample with the maximum density in the cluster; calculating the cosine distance between the log sample with the maximum density and other log samples in the cluster except the log sample with the maximum density, obtaining a log sample with the farthest distance; and splitting the other log samples in the cluster into two different clusters according to the cosine distance between the other log samples in the cluster and the log sample with the maximum density and the log sample with the farthest distance; obtaining the clusters after the splitting processing to form a cluster set; fusing the clusters in the cluster set according to the cosine distance between the cluster centers of the clusters until the distance between the cluster centers of all the clusters satisfies a preset condition; obtaining the cluster centers in the cluster set as initial centroids to perform Kmeans clustering.

2. The method of claim 1, wherein, the obtaining of the N log sample sets comprises: obtaining an initial log set, wherein the initial log set comprises at least one log sample; dividing the initial log set into N log sample sets.

3. The method of claim 1, wherein, the screening and deletion of the log samples in each log sample set to determine the K centers in each log sample set comprises: randomly selecting a first sample Cn1 from the log sample set, and deleting a first log sample in the log sample set with a cosine distance smaller than a first preset threshold from the first sample Cn1; obtaining all the deleted first log samples, and determining the center of all the deleted first log samples as a first center; randomly selecting a second sample Cn2 from the log sample set, and deleting a second log sample in the log sample set with a cosine distance smaller than the first preset threshold from the second sample Cn2; obtaining all the deleted second log samples, and determining the center of all the deleted second log samples as a second center; randomly selecting a Kth sample Cnk from the log sample set, and deleting a Kth log sample in the log sample set with a cosine distance smaller than the first preset threshold from the Kth sample Cnk; obtaining all the deleted Kth log samples, and determining the center of all the deleted Kth log samples as a Kth center.

4. The method of claim 1, wherein, the determination of the K centers in the copy of each log sample set and the K clusters in the copy according to the K centers in each log sample set comprises: determine K centers in the copy of the log sample set according to the K centers in each log sample set; calculate cosine distances between each log sample in the copy of the log sample set and each center in the copy; divide the Kth center and the log sample with the closest cosine distance to the Kth center in the copy into the same cluster to obtain K clusters of the copy.

5. The method of claim 1, wherein, Further comprising: if the error function of the cluster is less than or equal to the first preset threshold, the cluster is retained.

6. The method of claim 1, wherein, The calculation of the density of each log sample in the cluster comprises: obtaining a preset threshold Kn; obtaining a first set composed of Kn log samples with the closest cosine distance to the target log sample in the cluster; obtaining a second set composed of Kn log samples with the closest cosine distance to each log sample in the first set; obtaining the union set of the first set and the second set as the second-order Kn nearest neighbors of the target log sample; calculating the cosine distances between each log sample in the second-order Kn nearest neighbors and the target log sample and summing them up; comparing the sum value obtained by summing up with Kn to obtain the density of the target log sample.

7. The method of claim 1, wherein, The fusion of each cluster in the cluster set according to the cosine distances between the cluster centers of each cluster in the cluster set comprises: comparing the cosine distances between the cluster centers of each cluster with a preset second threshold; if the cosine distance between the cluster center of the first cluster and the cluster center of the second cluster is less than the preset second threshold, the first cluster and the second cluster are fused into one cluster.

8. The method of claim 7, wherein, The preset condition is that the distances between the cluster centers of all clusters in the cluster set are greater than the preset second threshold.

9. An initialization kmeans log classification device based on distributed split-merge, characterized in that, The initialization kmeans log classification device based on distributed splitting and fusion is used to execute the initialization kmeans log classification method based on distributed splitting and fusion of any one of claims 1-8, comprising: an acquisition module, configured to acquire N log sample sets, back up each log sample set to obtain a copy corresponding to the log sample set, and each log sample set comprises at least one log sample, and N is a positive integer; a center determination module, configured to filter and delete log samples in each log sample set to determine K centers in each log sample set, and K is a positive integer; a cluster division module, configured to determine K centers in the copy of each log sample set and K clusters of the copy according to the K centers in each log sample set; a cluster splitting module, configured to acquire an error function of each cluster in each copy, and split the cluster according to the error function of the cluster; a cluster set module, configured to acquire the clusters after splitting to form a cluster set; a cluster fusion module, configured to fuse each cluster in the cluster set according to the cosine distances between the cluster centers of each cluster in the cluster set until the distances between the cluster centers of all clusters satisfy a preset condition; a clustering module, configured to acquire the cluster centers in the cluster set as initial centroids to perform Kmeans clustering.

10. An electronic device, comprising: comprise: a processor, and a memory connected with the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 8.

12. A computer program product, characterised in that, The computer program product, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images