Log classification method and device based on distributed centroid screening and hierarchical clustering

Abstract

The application provides a log classification method and device based on distributed centroid screening and hierarchical clustering, and relates to the field of big data. The method comprises the following steps: obtaining N log sample sets and corresponding copies, and determining K centers in each log sample set; performing cluster division on the log samples in the copy of the log sample set according to the K centers in the log sample set, to obtain K clusters of the copy and the cluster center of each cluster; determining a high-quality cluster and the cluster center of the high-quality cluster in all clusters of the copy according to the number of log samples contained in each cluster in the copy; grouping the cluster centers of the high-quality clusters of all copies to form a high-quality point set; fusing each high-quality point in the high-quality point set according to the cosine distance of each high-quality point, to obtain a target high-quality point set; and taking the high-quality points in the target high-quality point set as initial centroids to perform Kmeans clustering. The scheme improves the convergence speed and clustering effect of the clustering algorithm.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a log classification method and apparatus based on distributed centroid filtering and hierarchical clustering. Background Technology

[0002] The purpose of log clustering is to find logs with high similarity from a massive log collection. Errors encountered by users during application use are recorded through logs. When analyzing errors, logs with the same error type need to be grouped together, and then the possible incorrect methods of user usage habits can be explored in a categorized manner to provide suggestions for similar methods in the future.

[0003] Currently, existing technologies typically use the k-means clustering algorithm (kmeans clustering algorithm) for text clustering. In the initialization phase, K objects need to be randomly selected as initial cluster centers. Then, the distance between each object and each seed cluster center is calculated, and each object is assigned to the cluster center closest to it.

[0004] However, existing kmeans clustering algorithms randomly select initial centroids during the initialization phase. This can lead to the selected centroids being too scattered or concentrated, and not particularly uniform, resulting in slow clustering convergence and poor clustering performance. Summary of the Invention

[0005] This application provides a log classification method and apparatus based on distributed centroid filtering and hierarchical clustering to solve the problems of slow convergence speed and poor clustering effect of current clustering algorithms.

[0006] Firstly, this application provides a log classification method based on distributed centroid filtering and hierarchical clustering, including:

[0007] Obtain N log sample sets and replicas, and determine K centers in each log sample set. The replicas are the same as the log samples in the corresponding log sample sets. N and K are positive integers, and the value of K is different in different log sample sets.

[0008] Based on the K centers in each log sample set, determine the K clusters in the replicas of that log sample set and the cluster center of each cluster. The value of K is different in different replicas.

[0009] Based on the number of log samples contained in each cluster of the replica, the high-quality cluster and the cluster center of the high-quality cluster are determined among all clusters of the replica;

[0010] The cluster centers of all high-quality clusters in all replicas are combined into a set of high-quality points;

[0011] Based on the cosine distance between each high-quality point in the set of high-quality points, the high-quality points are fused to obtain the target set of high-quality points.

[0012] Use the high-quality points in the target high-quality point set as the initial centroids and perform K-means clustering.

[0013] Secondly, this application provides a log classification device based on distributed centroid filtering and hierarchical clustering, comprising:

[0014] The acquisition module is used to acquire N log sample sets and replicas, and determine K centers in each log sample set. The replicas are the same as the log samples in the corresponding log sample sets. N and K are positive integers, and the value of K is different in different log sample sets.

[0015] The cluster partitioning module is used to determine the K clusters in the replicas of each log sample set and the cluster center of each cluster based on the K centers in each log sample set. The value of K is different in different replicas.

[0016] The high-quality cluster determination module is used to determine the high-quality clusters and the cluster centers of the high-quality clusters among all clusters in the replica based on the number of log samples contained in each cluster in the replica.

[0017] The High-Quality Point Set module is used to form a high-quality point set from the cluster centers of all replicas' high-quality clusters.

[0018] The target point set module is used to merge the high-quality points according to the cosine distance between each high-quality point in the high-quality point set to obtain the target high-quality point set.

[0019] The clustering module is used to perform K-means clustering with the high-quality points in the target high-quality point set as the initial centroids.

[0020] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described above.

[0021] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described above.

[0022] Fifthly, this application provides a computer program product that, when executed by a processor, is used to implement the method described above.

[0023] The log classification method and apparatus based on distributed centroid screening and hierarchical clustering provided in this application improve the initialization method of massive logs by distributing and parallelizing it, thereby reducing the initialization time. At the same time, it determines the high-quality clusters and the cluster centers of the high-quality clusters from the replicas and merges them to obtain the target high-quality point set. The high-quality points are selected from this set as the initial centroids, which can quickly obtain more uniformly distributed initial centroids based on the cluster centers. This avoids randomly selecting the initial centroids during the initialization stage and improves the current clustering effect. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] Figure 1 This is a schematic diagram illustrating the clustering effect of the Kmeans clustering algorithm provided in the embodiments of this application;

[0026] Figure 2 A flowchart illustrating the log classification method based on distributed centroid filtering and hierarchical clustering provided in this application embodiment;

[0027] Figure 3 This is a schematic diagram of log sample cluster partitioning provided in an embodiment of this application;

[0028] Figure 4 A flowchart illustrating the overall process of the log classification method based on distributed centroid filtering and hierarchical clustering provided in this application embodiment;

[0029] Figure 5 A schematic diagram of the structure of a log classification device based on distributed centroid filtering and hierarchical clustering provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0031] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0034] It should be noted that the log classification method and apparatus based on distributed centroid screening and hierarchical clustering provided in this application can be used in the field of big data, or in any field other than big data. The application field of the log classification method and apparatus based on distributed centroid screening and hierarchical clustering provided in this application is not limited.

[0035] K-means clustering algorithm: It is an iterative clustering analysis algorithm. Its steps are as follows: first, divide the data into K groups, then randomly select K objects as initial cluster centers, then calculate the distance between each object and each seed cluster center, and assign each object to the cluster center closest to it.

[0036] Text clustering is about finding similar texts, which is very meaningful for data mining. The traditional K-means clustering algorithm has the following disadvantages: (1) The selection of centroids in the initialization stage is random. The initial centroids are too close or scattered and not uniform enough, which will lead to a slow convergence speed of the algorithm. (2) Selecting discrete points or noisy data as the initial centroids is not conducive to the convergence of subsequent clustering, resulting in an unsatisfactory clustering effect. (3) The larger the data, the more obvious this defect becomes, and the initialization time is very long.

[0037] To address the problems of slow convergence and unsatisfactory clustering results when clustering large amounts of text, caused by the random selection of K objects as initial cluster centers in the initialization phase of existing K-means clustering algorithms, this application provides a log classification method and apparatus based on distributed centroid selection and hierarchical clustering. The method improves the initialization of massive logs by implementing distributed parallelization, reducing initialization time. Simultaneously, the initialization phase of the K-means algorithm is improved. During initialization, each map function performs exploratory clustering on the sample log set using different K values ​​in parallel. Density selection of the clusters results in relatively compact clusters. Then, a reduce function is used in the reduce phase to merge the compact clusters collected from the map function outputs in a hierarchical bottom-up manner. During fusion, the sample data of the cluster centers are considered, ensuring a relatively fair fusion process. Finally, K uniformly distributed initial centroids are obtained, avoiding the random selection of K objects as initial cluster centers in the initialization phase of the K-means clustering algorithm, thus improving the subsequent clustering performance of the K-means clustering algorithm.

[0038] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0039] For example, Figure 1 This is a schematic diagram illustrating the clustering effect of the K-means clustering algorithm provided in the embodiments of this application, as shown below. Figure 1 As shown, taking eight sample logs (Y1 to Y8) as an example, Y1 to Y5 are clustered into one cluster, and Y6 to Y8 are clustered into another. This clustering algorithm can be applied to text clustering scenarios, clustering massive amounts of historical error logs into multiple groups. After obtaining high-quality centroids from a large amount of input log data, this algorithm can quickly and effectively obtain K log groups. By organizing the error logs from each group, we can identify users' incorrect habits and corresponding solutions when using our application, forming solution assets that provide reference ideas for subsequent maintenance personnel when analyzing similar problems.

[0040] Figure 2 This is a flowchart illustrating the log classification method based on distributed centroid filtering and hierarchical clustering provided in an embodiment of this application. This method can be applied to text clustering scenarios, such as... Figure 2 As shown, the method may specifically include the following steps:

[0041] Step S201: Obtain N log sample sets and replicas, determine K centers in each log sample set, and ensure that the replicas are the same as the log samples in the corresponding log sample sets. N and K are positive integers, and the value of K is different in different log sample sets.

[0042] In this embodiment, the log samples contained in each log sample set may be different. For example, log sample set J1 includes log samples J11, J12, and J13, while log sample set J2 includes log samples J21, J22, and J23.

[0043] The number of log sample sets can be two or more. Before proceeding to the next step, each log sample set can be copied to obtain a corresponding copy, which is exactly the same as the log sample set. When processing the log sample set later, the log sample set may change, such as deleting log samples from the log sample set, resulting in a decrease in the number of log samples in the log sample set. In this way, by retaining the copy and not processing the copy, the changes in the log sample set can be observed through the copy, which is convenient for targeted operations on the copy. The specific operations will be explained in detail later.

[0044] In this embodiment, for a massive number of log samples, these samples can be categorized into different sets. Specifically, these log samples can be randomly divided into different log sample sets, thus forming N log sample sets as shown in this embodiment. In other embodiments, N log sample sets can be constructed first, and the number of log samples that each set can hold can be determined in advance. Then, the massive number of log samples generated by the application can be randomly assigned to each log sample set, ensuring that each set is saturated with log samples.

[0045] In this embodiment, since each log sample set may contain multiple log samples, there will be a log sample that is at the center of all log samples, and this log sample can be called the center. The log samples in the log sample set can be filtered by region, for example, divided into region Q1, region Q2, and region Q3. Thus, there is a center in region Q1, a center in region Q2, and a center in region Q3.

[0046] Step S202: Based on the K centers in each log sample set, determine the K clusters in the replicas of the log sample set and the cluster center of each cluster. The value of K is different in different replicas.

[0047] In this embodiment, the log sample set has K centers (each center can be understood as a log sample). In addition, there may be other log samples in the log sample set. These log samples can be clustered based on the K centers to obtain K clusters.

[0048] For example, taking a log sample set J1 with two centers (e.g., log samples J11 and J12 are both centers), when there are log samples J13, J14, and J15 in the replica corresponding to log sample set J1, if log sample J13 is near center J12, then log sample J13 and center J12 are assigned to the first cluster. If log sample J14 and log sample J15 are near center J11, then log samples J14, J11, and J15 are assigned to the second cluster. In this way, the replica has two clusters (i.e., the first cluster and the second cluster).

[0049] For example, Figure 3 This is a schematic diagram of log sample cluster partitioning provided in an embodiment of this application, as shown below. Figure 3 As shown, by digitizing the log samples and replacing each log sample with a coordinate point, Y11 and Y12 can be selected as the center. Log sample Y13 is near the center Y12, so log sample Y13 and the center Y12 are assigned to the first cluster. Log samples Y14 and Y15 are near the center Y11, so log samples Y14, Y11, and Y15 are assigned to the second cluster.

[0050] Step S203: Based on the number of log samples contained in each cluster of the replica, determine the high-quality cluster and the cluster center of the high-quality cluster among all clusters of the replica.

[0051] In this embodiment, the number of log samples contained in each cluster can be calculated. When the number of log samples exceeds a certain threshold, the cluster can be determined to be a high-quality cluster, and the high-quality cluster has a cluster center.

[0052] For example, in other implementations, clusters with more than a certain number of log samples can be found first, and then the density of the cluster center can be checked. If the density is greater than a certain density threshold, the cluster center is considered to be the cluster center of a high-quality cluster, that is, the cluster is also regarded as a high-quality cluster.

[0053] The density of a cluster center can be characterized by the number of log samples around that cluster center; the more log samples there are, the greater the density of the cluster center.

[0054] Step S204: Combine the cluster centers of all replicas of high-quality clusters into a set of high-quality points.

[0055] In this embodiment, the number of log sample sets can be N, and the corresponding number of replicas can also be N. Each replica contains a high-quality cluster and the cluster center of that high-quality cluster. The log samples of the cluster centers of the high-quality clusters are selected to form a set of high-quality points.

[0056] Step S205: Based on the cosine distance between each high-quality point in the high-quality point set, merge the high-quality points to obtain the target high-quality point set.

[0057] In this embodiment, a high-quality point is the cluster center of a high-quality cluster. The cosine distance between each high-quality point can be calculated. When the cosine distance is less than a certain distance threshold, the two high-quality points can be merged into a new high-quality point. In this way, the cluster center in the set of high-quality points is reduced by one. By continuously merging in this way, when the number of cluster centers in the set of high-quality points is reduced to a certain value, the target set of high-quality points can be obtained.

[0058] Step S206: Use the high-quality points in the target high-quality point set as the initial centroids and perform K-means clustering.

[0059] In this embodiment, all high-quality points in the target high-quality point set can be used as the initial centroid, or several can be selected from them as the initial centroid.

[0060] This application's embodiments improve the initialization method for massive logs through distributed parallelization, reducing initialization time. Simultaneously, it identifies high-quality clusters and their cluster centers from the replicas, merges them to obtain a target set of high-quality points, and selects these high-quality points as initial centroids. This results in a more evenly distributed initial centroid distribution, improving the quality of clustering.

[0061] In some embodiments, the above method further includes the following steps: calculating the value of Kn based on the number N of the log sample set; and determining the number of centers in each log sample set and the number of clusters in each replica based on the value of Kn.

[0062] In this embodiment, when performing cluster partitioning on the log samples in each replica, the number of clusters that each replica should be partitioned into can be configured. For example, replica 1 can be configured to partition the log samples into 10 clusters, and replica 2 can be configured to partition the log samples into 11 clusters. In this embodiment, the number of log sample sets (dataset) is N. The square root of N is denoted as Kn. Then, based on the value of Kn, the number of centers in each log sample set and the number of clusters that each replica should be partitioned into are further determined.

[0063] The number of centers in the log sample set can be the same as the number of clusters required to be divided for the replicas corresponding to that log sample set.

[0064] Furthermore, in other implementations, multiple graph functions (i.e., Map functions) can be configured, each Map function processing one replica and clustering the log samples into different clusters to achieve cluster partitioning. Specifically, this can be achieved through the following steps: obtaining the total number S of pre-configured graph functions and the replica processed by each graph function; each graph function is used to determine the high-quality cluster and its cluster center among all clusters of the replica it processes; and determining the number of clusters in the replica processed by each graph function based on the values ​​of the total number S and Kn.

[0065] In this embodiment, each graph function can process a replica, which can cluster the log samples in the replica. For example, a replica should be divided into 2 clusters. The replica includes log sample A1, log sample A2 and log sample A3. The graph function can cluster log sample A1 and log sample A2 into one cluster and log sample A3 into another cluster.

[0066] Furthermore, in other implementations, the number of clusters that each replica should be divided into can be determined by the following formula:

[0067] Si = Kn - ([S / 2] + 1 - i)

[0068] In the above formula, Si is the number of clusters in the replica corresponding to the i-th graph function, [S / 2] is the total number of configured graph functions S / 2 rounded down, and i is a positive integer.

[0069] In this embodiment, let the set Klist = {Kn-5, Kn-4, Kn-3, Kn-2, Kn-1, Kn, Kn+1, Kn+2, Kn+3, Kn+4, Kn+5}. Eleven map functions are set up, each requiring K values ​​for the number of clusters to be clustered: Kn-5, Kn-4, Kn-3, Kn-2, Kn-1, Kn, Kn+1, Kn+2, Kn+3, Kn+4, Kn+5. The log sample set input to each map function is denoted as dataset. For example, the log sample set input to the first map function is dataset1, the second map function is dataset2, and so on, until the log sample set input to the i-th map function is dataseti.

[0070] In this embodiment, if the number of clusters K that the map function needs to cluster is Kn-5, then the number of centers of the log sample set input to the map function is Kn-5, and the number of clusters in the corresponding replicas is Kn-5.

[0071] This application embodiment achieves more compact clusters by using different K values ​​to perform exploratory clustering on the sample log set in parallel during the initialization process for each map function. This results in more concentrated log samples within the clusters and improves the quality of subsequent initial centroid selection.

[0072] In some embodiments, step S201 can be implemented by the following steps: backing up each log sample set to obtain a copy of each log sample set; randomly selecting a first sample C1 from the log sample set and deleting the first log sample in the log sample set whose cosine distance to the first sample C1 is less than a first preset threshold; obtaining the center of the set composed of the deleted first log samples as the first center; randomly selecting a second sample C2 from the log sample set and deleting the second log sample in the log sample set whose cosine distance to the second sample C2 is less than the first preset threshold; obtaining the center of the deleted second log sample as the second center; randomly selecting a Kth sample Ck from the log sample set and deleting the Kth log sample in the log sample set whose cosine distance to the Kth sample Ck is less than the first preset threshold; obtaining the center of the set composed of the deleted Kth log samples as the Kth center.

[0073] In this embodiment, the i-th log sample set dataseti is backed up to obtain a copy of the log sample set dataseti, which is dataset_baki.

[0074] In this process, a first sample C1 is randomly selected from dataset1. Log samples from dataset1 whose cosine distance to C1 is less than a first preset threshold T1 are deleted (there are usually multiple first log samples). The center of the set formed by these first log samples is denoted as the first center. Then, a second sample C2 is randomly selected from dataset1. Log samples from dataset1 whose cosine distance to C2 is less than the first preset threshold T1 are deleted (there are usually multiple second log samples). The center of the set formed by these second log samples is denoted as the second center. This process is repeated until the Kth center is found. The set of K centers is backed up as klist_bak1.

[0075] This application's embodiments improve processing speed and shorten processing time by processing each log sample set separately. Simultaneously, by continuously filtering log samples in the log sample set using cosine distance, it is possible to accurately obtain K centroid samples, ensuring the accuracy of subsequent initial centroid selection and improving the K-means clustering effect.

[0076] In some embodiments, step S202 can be implemented by the following steps: determining K centers in a replica of the log sample set based on K centers in each log sample set; calculating the cosine distance between each log sample in the replica and each center; assigning the log sample in the replica closest to the Kth center and the Kth center to the same cluster to obtain K clusters; and determining the log sample in the replica that is located within the center range of the cluster as the cluster center.

[0077] In this embodiment, the process can be handled using the idea of ​​the map function: The idea of ​​the map function is as follows:

[0078] (1) Let the copy of dataset be dataset_bak,

[0079] (2) Randomly select a log sample C1 from the dataset, delete log samples in the dataset whose cosine distance to log sample C1 is less than the threshold T1, and denote the center of the set formed by these log samples as the first center. Then randomly select a log sample C2 from the dataset, delete log samples in the dataset whose cosine distance to log sample C2 is less than the threshold T1, and denote the center of the set formed by these log samples as the second center. Continue in this manner until the Kth center is found. (K is a value taken from Klist, K is Kn-5 for map1 function, K is Kn-4 for map2 function, ..., Kn+5 for map11 function). Then divide the samples in dataset_bak into K clusters according to their cosine distance, finally obtaining K clusters.

[0080] Taking replica F1 as an example, assuming replica F1 has two centers (e.g., a first center and a second center), and replica F1 also has three log samples F11 and F12, with F11's distance to the first center being DF11 and its distance to the second center being DF12, and F12's distance to the first center being DF21 and its distance to the second center being DF22: If DF11 is less than DF12, then F11 and the first center are assigned to the same cluster, and then log samples within the center range of that cluster are selected as the center of that cluster. If DF21 is greater than DF22, then F12 and the second center are assigned to another cluster, and then log samples within the center range of that cluster are selected as the center of that cluster.

[0081] This application embodiment divides log samples into corresponding clusters based on the cosine distance, which can prevent log samples that are far from a certain center from being divided into the same cluster as other centers, effectively improving the accuracy of clustering and further improving the accuracy of initial centroid selection.

[0082] In some embodiments, step S203 can be implemented by the following steps: comparing the number of log samples contained in each cluster of the replica with a first preset threshold; if the number of log samples contained in the cluster is greater than the first preset threshold, then obtaining the density of the cluster center; when the density of the cluster center is greater than the density threshold, the cluster is determined to be a high-quality cluster.

[0083] In this embodiment, K clusters in the replica are examined, and the number of log samples in each cluster is calculated. If the number of log samples is less than a preset first threshold, the cluster is considered an outlier and is ignored. For clusters where the number of log samples is greater than the preset first threshold, the density of the cluster center is checked. If the density of the cluster center is greater than a density threshold T_good, the cluster center is considered a high-quality point. Finally, the map function outputs all high-quality points and the number of samples in their respective clusters.

[0084] Furthermore, based on the above embodiments, in some embodiments, the above-mentioned acquisition of the density of the cluster center can be achieved by the following steps: obtaining the number of log samples in the cluster whose cosine distance from the cluster center is less than a second preset threshold, and using this as the density of the cluster center.

[0085] In this embodiment, the center of cluster X can be denoted as Cx, and the density of Cx is the number n of samples in the cluster whose cosine distance from Cx is less than the threshold T3.

[0086] This application embodiment filters the clusters by density, and identifies cluster centers with a density greater than a density threshold as high-quality points. This results in relatively compact high-quality clusters, where log samples are more concentrated and the clustering effect is better, thus enabling the selection of higher-quality initial centroids.

[0087] In some embodiments, step S205 can be implemented through the following steps: obtaining the first and second high-quality points with the closest cosine distance among all high-quality points in the high-quality point set; merging the first and second high-quality points into a new high-quality point; determining whether the number of high-quality points in the current high-quality point set is a preset number; if the number of high-quality points in the current high-quality point set is a preset number, then using the current high-quality point set as the target high-quality point set; if the number of high-quality points in the current high-quality point set is not a preset number, then continuing to merge the high-quality points in the current high-quality point set.

[0088] In this embodiment, the process can be performed using the Reduce function. The idea behind the Reduce function is as follows:

[0089] (1) Input the number of log samples of the 11 high-quality points output by the map function and their corresponding clusters. The set of all high-quality points is denoted as list_good, and the number of high-quality points in the set list_good is denoted as N_total.

[0090] (2) Calculate the cosine distance between each pair of high-quality points in list_good, find the two high-quality points with the closest cosine distance, and merge these two high-quality points;

[0091] (3) Delete the two high-quality points that have been merged from the set list_good, and put the merged new high-quality points into the set list_good. At this time, the number of high-quality points in the set list_good is N_total-1.

[0092] Based on the above principle, the two best points with the closest cosine distance are calculated pairwise from list_good again, and these two best points are deleted from list_good. Then, the new point formed by merging these two points is put into list_good. This process is repeated until the number of best points in list_good is the set number of K_good points. Finally, the reduce function outputs K_good best points.

[0093] When merging two high-quality points (e.g., the first high-quality point and the second high-quality point) into a new high-quality point, the merging method is as follows: obtain the number of log samples contained in the cluster corresponding to the first high-quality point; obtain the number of log samples contained in the cluster corresponding to the second high-quality point; and obtain the new high-quality point by merging the first high-quality point, the second high-quality point, the number of log samples contained in the cluster corresponding to the first high-quality point, and the number of log samples contained in the cluster corresponding to the second high-quality point.

[0094] In this embodiment, it is assumed that the first high-quality point Cy1 has Ny1 log samples in its corresponding cluster, and the second high-quality point Cy2 has Ny1 log samples in its corresponding cluster. The new high-quality point after fusion is Cnew = (Cy1*Ny1+Cy2*Ny2) / (Ny1+Ny2).

[0095] This application embodiment fuses the compact clusters collected from the map function output in a hierarchical bottom-up manner during the reduce phase, and considers the sample data of the cluster to which the cluster center belongs during the fusion, so as to perform the fusion relatively fairly and finally obtain K uniformly dispersed initial centroids, thereby improving the quality of clustering.

[0096] For example, Figure 4The overall flowchart of the log classification method based on distributed high-quality centroid screening and hierarchical clustering provided in this application embodiment includes the following Map function processing stage and Reduce function processing stage. Multiple Map functions can be configured, and each Map function performs the same processing on a log sample set simultaneously. Figure 4 As shown, the Map function processing stage includes steps S4011, S4012, ..., S401i (i takes values ​​from 1 to N). The Reduce function processing stage includes step S402.

[0097] This application improves the initialization method for massive logs by implementing distributed parallelization, reducing initialization time. It also improves the initialization phase of the k-means algorithm. During initialization, each map function performs exploratory clustering on the sample log set using different K values ​​in parallel. The clusters are then density-filtered to obtain relatively compact clusters. In the reduce phase, the compact clusters from the map function outputs are merged in a hierarchical bottom-up manner, considering the sample data of the cluster centers to ensure a relatively fair fusion. This results in K uniformly distributed initial centroids, improving the quality of clustering.

[0098] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0099] Figure 5 A schematic diagram of the log classification device based on distributed centroid filtering and hierarchical clustering provided in the embodiments of this application is shown below. Figure 5 As shown, the log classification device 500 includes an acquisition module 510, a cluster division module 520, a high-quality cluster determination module 530, a high-quality point set module 540, a target point set module 550, and a clustering module 560.

[0100] The acquisition module 510 acquires N log sample sets and replicas, determining K centroids in each log sample set. Replicas contain the same log samples as their corresponding log sample sets. N and K are positive integers, and K takes different values ​​in different log sample sets. The cluster partitioning module 520 determines K clusters and their cluster centers in the replicas of each log sample set based on the K centroids. K takes different values ​​in different replicas. The high-quality cluster determination module 530 determines high-quality clusters and their cluster centers in all clusters of a replica based on the number of log samples contained in each cluster. The high-quality point set module 540 forms a high-quality point set from the cluster centers of all high-quality clusters in all replicas. The target point set module 550 merges the high-quality points in the high-quality point set based on the cosine distance between them to obtain a target high-quality point set. The clustering module 560 uses the high-quality points in the target high-quality point set as initial centroids for K-means clustering.

[0101] Optionally, it also includes a quantity determination module, which is used to calculate the value of Kn based on the number N of the log sample set; and to determine the number of centers in each log sample set and the number of clusters in each replica based on the value of Kn.

[0102] Optionally, the quantity determination module can be used to: obtain the total number S of pre-configured graph functions and the replicas processed by each graph function, and each graph function is used to determine the high-quality cluster and the cluster center of the high-quality cluster among all the clusters of the replicas processed by the graph function; and determine the number of clusters in the replicas processed by each graph function according to the values ​​of the total number S and Kn.

[0103] Optionally, the quantity determination module can be used to: calculate the number of clusters that each replica should be divided into:

[0104] Si = Kn - ([S / 2] + 1 - i)

[0105] In the above formula, Si is the number of clusters in the replica corresponding to the i-th graph function, [S / 2] is the total number of configured graph functions S / 2 rounded down, and i is a positive integer.

[0106] Optionally, the acquisition module can be specifically used for: backing up each log sample set to obtain a copy of each log sample set; randomly selecting a first sample C1 from the log sample set and deleting the first log sample in the log sample set whose cosine distance to the first sample C1 is less than a first preset threshold; obtaining the center of the set composed of the deleted first log samples as the first center; randomly selecting a second sample C2 from the log sample set and deleting the second log sample in the log sample set whose cosine distance to the second sample C2 is less than the first preset threshold; obtaining the center of the set composed of the deleted second log samples as the second center; randomly selecting a Kth sample Ck from the log sample set and deleting the Kth log sample in the log sample set whose cosine distance to the Kth sample Ck is less than the first preset threshold; obtaining the center of the set composed of the deleted Kth log samples as the Kth center.

[0107] Optionally, the cluster partitioning module can be used to: determine K centers in the replica of each log sample set based on K centers in each log sample set; calculate the cosine distance between each log sample in the replica and each center; partition the log sample in the replica that has the closest cosine distance to the Kth center and the Kth center into the same cluster to obtain K clusters; and determine the log samples in the replica that are located within the center range of the cluster as the cluster center.

[0108] Optionally, the high-quality cluster determination module can be used to: compare the number of log samples contained in each cluster in the replica with a first preset threshold; if the number of log samples contained in the cluster is greater than the first preset threshold, then obtain the density of the cluster center; when the density of the cluster center is greater than the density threshold, determine the cluster as a high-quality cluster.

[0109] Optionally, the high-quality cluster determination module can be used to: obtain the number of log samples in the cluster whose cosine distance from the cluster center is less than a second preset threshold, and use this number as the density of the cluster center.

[0110] Optionally, the target point set module can be used to: obtain the first and second high-quality points with the closest cosine distance among all high-quality points in the high-quality point set; merge the first and second high-quality points into a new high-quality point; determine whether the number of high-quality points in the current high-quality point set is a preset number; if the number of high-quality points in the current high-quality point set is a preset number, then use the current high-quality point set as the target high-quality point set; if the number of high-quality points in the current high-quality point set is not a preset number, then continue to merge the high-quality points in the current high-quality point set.

[0111] Optionally, the target point set module can be used to: obtain the number of log samples contained in the cluster corresponding to the first high-quality point; obtain the number of log samples contained in the cluster corresponding to the second high-quality point; and merge the first high-quality point, the second high-quality point, the number of log samples contained in the cluster corresponding to the first high-quality point, and the number of log samples contained in the cluster corresponding to the second high-quality point to obtain a new high-quality point.

[0112] The apparatus provided in this application embodiment can be used to execute the methods in the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0113] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the acquisition module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0114] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 includes at least one processor 601, a memory 602, a bus 603, and a communication interface 604. The processor, communication interface, and memory communicate with each other via the bus. The communication interface is used to communicate with other devices. This communication interface includes a communication interface for data transmission and a display interface or operation interface for human-computer interaction. The processor executes computer instructions stored in the memory, specifically performing the relevant steps in the methods described in the above embodiments.

[0115] The processor may be a central processing unit, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0116] Memory is used to store instructions executed by a computer. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0117] This embodiment also provides a computer-readable storage medium storing computer instructions, which, when executed by at least one processor of an electronic device, enable the electronic device to perform the methods provided in the various embodiments described above.

[0118] This embodiment also provides a computer program product including computer instructions stored in a readable storage medium. At least one processor of an electronic device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to cause the electronic device to perform the methods provided in the various embodiments described above.

[0119] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0120] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0121] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0122] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A log classification method based on distributed centroid filtering and hierarchical clustering, characterized in that, include: Obtain N log sample sets and replicas, and determine K centers in each log sample set. The replicas are identical to the log samples in the corresponding log sample sets. N and K are positive integers, and the value of K is different in different log sample sets. The log sample sets are used to filter centers, and the replicas are used for subsequent cluster partitioning. Based on the K centers in each log sample set, determine the K clusters in the replicas of that log sample set and the cluster center of each cluster. The values ​​of the K clusters in different replicas are different. Based on the number of log samples contained in each cluster of the replica, the high-quality cluster and the cluster center of the high-quality cluster are determined among all clusters of the replica; The cluster centers of all high-quality clusters in all replicas are combined into a set of high-quality points; Based on the cosine distance between each high-quality point in the set of high-quality points, the high-quality points are fused to obtain the target set of high-quality points. Use the high-quality points in the target high-quality point set as the initial centroids and perform K-means clustering; The step of fusing the high-quality points in the high-quality point set based on the cosine distance of each high-quality point to obtain the target high-quality point set includes: Obtain the first and second high-quality points in the set of high-quality points that have the closest cosine distance to each other; merge the first and second high-quality points into a new high-quality point; determine whether the number of high-quality points in the current set of high-quality points is a preset number; if the number of high-quality points in the current set of high-quality points is a preset number, then use the current set of high-quality points as the target set of high-quality points; if the number of high-quality points in the current set of high-quality points is not a preset number, then continue to merge the high-quality points in the current set of high-quality points.

2. The method according to claim 1, characterized in that, Also includes: The value of Kn is calculated based on the number N of the log sample set; Based on the value of Kn, determine the number of centers in each log sample set and the number of clusters in each replica.

3. The method according to claim 2, characterized in that, Determine the number of clusters in the replicas of each log sample, including: Obtain the total number S of pre-configured graph functions and the copy of each graph function to be processed. Each graph function is used to determine the high-quality cluster and the cluster center of the high-quality cluster among all the clusters of the copy of the graph function to be processed. Based on the total quantity S and the value of Kn, the number of clusters in the replicas processed by each graph function is determined.

4. The method according to claim 3, characterized in that, The step of determining the number of clusters in the replicas processed by each graph function based on the total number S and the value of Kn includes: Si = Kn - ([S / 2] + 1 - i) In the above formula, Si is the number of clusters in the replica corresponding to the i-th graph function, [S / 2] is the total number of configured graph functions S / 2 rounded down, and i is a positive integer.

5. The method according to claim 1, characterized in that, The step of determining the high-quality cluster and the cluster center of the high-quality cluster among all clusters in the replica based on the number of log samples contained in each cluster in the replica includes: Compare the number of log samples contained in each cluster of the replica with a first preset threshold; If the number of log samples contained in the cluster is greater than the first preset threshold, then the density of the cluster center is obtained; When the density at the cluster center of a cluster is greater than a density threshold, the cluster is determined to be a high-quality cluster.

6. The method according to claim 5, characterized in that, Obtaining the density of the cluster center includes: The number of log samples in the cluster whose cosine distance from the cluster center is less than a second preset threshold is obtained and used as the density of the cluster center.

7. The method according to claim 1, characterized in that, The step of merging the first high-quality point and the second high-quality point into a new high-quality point includes: Get the number of log samples contained in the cluster corresponding to the first high-quality point; Get the number of log samples contained in the cluster corresponding to the second-highest quality point; A new high-quality point is obtained by fusing the first high-quality point, the second high-quality point, the number of log samples contained in the cluster corresponding to the first high-quality point, and the number of log samples contained in the cluster corresponding to the second high-quality point.

8. A log classification device based on distributed centroid filtering and hierarchical clustering, characterized in that, include: The acquisition module is used to acquire N log sample sets and replicas, and determine K centers in each log sample set. The replicas are the same as the log samples in the corresponding log sample sets. N and K are positive integers, and the value of K is different in different log sample sets. The log sample sets are used to filter centers, and the replicas are used for subsequent cluster partitioning. The cluster partitioning module is used to determine the K clusters in the replicas of each log sample set and the cluster center of each cluster based on the K centers in each log sample set. The value of K is different in different replicas. The high-quality cluster determination module is used to determine the high-quality clusters and the cluster centers of the high-quality clusters among all clusters in the replica based on the number of log samples contained in each cluster in the replica. The High-Quality Point Set module is used to form a high-quality point set from the cluster centers of all replicas' high-quality clusters. The target point set module is used to merge the high-quality points in the high-quality point set according to the cosine distance between each high-quality point in the high-quality point set to obtain the target high-quality point set. The clustering module is used to perform K-means clustering with the high-quality points in the target high-quality point set as the initial centroids. The target point set module can be specifically used to: obtain the first and second high-quality points with the closest cosine distance among all high-quality points in the high-quality point set; merge the first and second high-quality points into a new high-quality point; determine whether the number of high-quality points in the current high-quality point set is a preset number; if the number of high-quality points in the current high-quality point set is a preset number, then use the current high-quality point set as the target high-quality point set; if the number of high-quality points in the current high-quality point set is not a preset number, then continue to merge the high-quality points in the current high-quality point set.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, When executed by a processor, the computer program product is used to implement the method as described in any one of claims 1 to 7.

Images