Kmeans log classification method and device based on fusion of cut-off distance

Abstract

The application provides a kmeans log classification method and device based on fusion of cut-off distance, and relates to the field of big data.The method comprises the following steps: selecting log samples with cosine distance within a first distance range from a center from a log sample set to form a first sample set, and selecting a first log sample from the first sample set according to sample density and adding the first log sample to a center set; selecting log samples with cosine distance within a Kth distance range from the first log sample from the log sample set to form a Kth sample set according to the value of K, and selecting a Kth log sample from the Kth sample set according to sample density and adding the Kth log sample to the center set, wherein the value of K is [1, K] in turn; and fusing the log samples in the center set until the number of the log samples in the center set is K, so as to serve as initial centroids of kmeans clustering.The scheme improves the clustering effect of the Kmeans algorithm.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method and apparatus for k-means log classification based on distance fusion. Background Technology

[0002] During application use, user errors can easily lead to program errors. These errors are typically logged, and analyzing these logs can help identify solutions. However, based on the different types of errors, error logs can often be categorized. Therefore, it's necessary to cluster the massive amounts of error logs, grouping logs of the same type together for easier analysis.

[0003] Currently, the existing technology mainly uses the K-means clustering algorithm. This clustering algorithm randomly selects initial centroids during the initialization phase, then calculates the distance between each object and the initial centroid, and assigns each object to the cluster center closest to it.

[0004] However, this method of randomly selecting initial centroids leads to poor final clustering results. Summary of the Invention

[0005] This application provides a method and apparatus for k-means log classification based on gap distance fusion, in order to solve the problem of poor k-means clustering performance.

[0006] Firstly, this application provides a k-means log classification method based on gap distance fusion, including:

[0007] Obtain a log sample set, the center of the log sample set, the average distance between the log samples in the log sample set, and the sample density of each log sample in the log sample set, wherein the log sample set includes at least two log samples;

[0008] Log samples with a cosine distance to the center within a first distance range are selected from the log sample set to form a first sample set. Based on the sample density, a first log sample is selected from the first sample set and added to the center set.

[0009] Based on the value of k, log samples whose cosine distance to the first log sample is within the k-th distance range are selected from the log sample set to form the k-th sample set. Based on the sample density, the k-th log sample is selected from the k-th sample set and added to the central set. The values ​​of k are [1, k], where k is an integer greater than 1.

[0010] Merge the log samples in the central set until the number of log samples in the central set is K;

[0011] K log samples from the merged centroid set are used as initial centroids for kmean clustering.

[0012] Secondly, this application provides a k-means log classification device based on gap distance fusion, comprising:

[0013] The acquisition module is used to acquire a log sample set, the center of the log sample set, the average distance between the log samples in the log sample set, and the sample density of each log sample in the log sample set, wherein the log sample set includes at least two log samples;

[0014] The sample selection module is used to select log samples from the log sample set whose cosine distance to the center is within a first distance range to form a first sample set, and select a first log sample from the first sample set according to the sample density and add it to the center set;

[0015] The sample cyclic selection module is used to select log samples from the log sample set whose cosine distance to the first log sample is within the k-th distance range according to the value of k, to form the K-th sample set, and select the k-th log sample from the k-th sample set according to the sample density, and add it to the central set. The value of k is [1, k], where k is an integer greater than 1.

[0016] The sample fusion module is used to fuse log samples in the central set until the number of log samples in the central set is K.

[0017] The clustering module is used to perform kmeans clustering on the K log samples in the merged centroid set as the initial centroids.

[0018] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the above-described method.

[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described above.

[0020] Fifthly, this application provides a computer program product that, when executed by a processor, is used to implement the above-described method.

[0021] The k-means log classification method and apparatus based on distance fusion provided in this application selects log samples into a centroid set by utilizing the pre-defined distances and distance ranges between log samples. Then, the log samples in the centroid set are fused until the number of log samples in the centroid set is the number of initial centroids required by the k-means clustering algorithm. Finally, the log samples in the centroid set are used as the initial centroids, which can improve the selection quality of the initial centroids and further improve the clustering effect of the k-means clustering algorithm. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 A schematic diagram of the K-means clustering process provided in the embodiments of this application;

[0024] Figure 2 A flowchart illustrating the k-means log classification method based on gap distance fusion provided in this application embodiment;

[0025] Figure 3 A schematic diagram of a log sample provided in an embodiment of this application;

[0026] Figure 4 A flowchart of the k-means log classification method based on gap distance fusion provided in this application embodiment;

[0027] Figure 5 A schematic diagram of the structure of the k-means log classification device based on gap distance fusion provided in the embodiments of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0029] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] It should be noted that the k-means log classification method and apparatus based on gap distance fusion provided in this application can be used in the field of big data technology, or in any field other than big data technology. The application field of the k-means log classification method and apparatus based on gap distance fusion provided in this application is not limited.

[0033] K-means clustering algorithm: It is a clustering algorithm that can be used for text clustering. In the initialization phase, there is a step to divide the data into K groups, and then randomly select an object in each group as the initial cluster center of that group. Then, the distance between each object and each seed cluster center is calculated, and each object is assigned to the cluster center that is closest to it.

[0034] The K-means clustering algorithm currently has the following drawbacks: (1) The selection of centroids in the initialization stage is random. The initial centroids are too close or scattered and not uniform enough, which will lead to slow convergence of the algorithm. (2) Selecting discrete points or noisy data as initial centroids is not conducive to the convergence of subsequent clustering, resulting in unsatisfactory clustering effect. (3) The larger the data, the more obvious this defect becomes, and the initialization time is very long.

[0035] To address the aforementioned issues, this application provides a k-means log classification method and apparatus based on segment distance fusion. By utilizing a method based on segment distance fusion to select the initial centroids required by the k-means clustering algorithm in the initial stage, the quality of the selected initial centroids can be improved, thereby further enhancing the clustering effect of the k-means clustering algorithm, ensuring that error logs of the same type can be clustered into the same group, and improving the accuracy of log classification.

[0036] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0037] For example, Figure 1This is a flowchart illustrating the K-means clustering process provided in an embodiment of this application, as shown below. Figure 1 As shown, taking 10 sample logs (L1 to L10) as an example, these 10 sample logs can be divided into 3 groups. Three initial centroids, L1, L2, and L3, are selected. Then, based on these three initial centroids, the remaining log samples are clustered into 3 clusters. Here, "log" can be a term from the computer science field; application platforms generate logs during operation. Each log line records the date, time, user, and description of the relevant operations.

[0038] Figure 2 This is a flowchart illustrating the k-means log classification method based on gap distance fusion provided in this application embodiment. This method can be applied to text clustering scenarios, clustering massive amounts of historical error logs into multiple groups (i.e., different types of error logs are clustered into different groups). After obtaining initial centroids from the large amount of input log data using this algorithm, K-means clustering is then used for further clustering, quickly yielding K log groups with good clustering results. By analyzing and organizing the error logs in each group, incorrect user habits during application usage can be identified, facilitating the provision of corresponding solutions. Figure 2 As shown, the method may specifically include the following steps: Step S201, obtaining the log sample set, the center of the log sample set, the average distance between the log samples in the log sample set, and the sample density of each log sample in the log sample set, wherein the log sample set includes at least two log samples.

[0039] In this embodiment, the log samples included in the log sample set can be error logs generated by the application. Typically, the number of error logs generated by applications is very large, and when the log samples included in the log sample set are all error logs, the number of log samples in the log sample set will also be very large. Different error logs can be categorized into different types (this is because different user errors result in different types of error logs).

[0040] In this embodiment, log samples in the log sample set can be represented by coordinate points in a coordinate system. These coordinate points provide a visual representation of the log sample distribution, such as the distance between two log samples (representing their similarity; generally, log samples that are farther apart are considered to belong to two different types of error logs, while those that are closer are considered to belong to the same type of error logs). Furthermore, the coordinate points can be used to locate the center of the log sample set. The average distance between the log sample sets can be considered the average distance between all coordinate points.

[0041] In this embodiment, when log samples are converted into coordinate points, the sample density of a log sample can be understood as the number of other coordinate points around that coordinate point. The more other coordinate points around that coordinate point, the greater the sample density of the log sample corresponding to that coordinate point. Similarly, the fewer other coordinate points around that coordinate point, the smaller the sample density of the log sample corresponding to that coordinate point.

[0042] Step S202: Select log samples from the log sample set whose cosine distance from the center is within a first distance range to form a first sample set, and select a first log sample from the first sample set according to the sample density and add it to the center set.

[0043] In this embodiment, cosine distance, also known as cosine similarity, is a measure of the difference between two individuals using the cosine of the angle between two vectors in a vector space. Specifically, log samples can be converted into vector form in a spatial coordinate system. By converting the log samples into vector form, the cosine distance between two log samples can be calculated.

[0044] The first distance range can be configured according to actual conditions, and includes a lower limit and an upper limit. In other embodiments, the first distance range is positively correlated with the average distance of the samples; the larger the average distance of the samples, the larger both the lower and upper limits of the first distance range become. By configuring the first distance range, log samples within this range can be filtered out, ensuring the quality of the subsequent selection of initial centroids.

[0045] In this embodiment, when selecting the first log sample, in order to ensure that the first log sample is representative and improve the quality of subsequent initial centroid selection, the center sample of the first sample set can be selected as the first log sample. The center set may include multiple first log samples.

[0046] Step S203: Based on the value of k, select log samples from the log sample set whose cosine distance to the first log sample is within the k-th distance range to form the k-th sample set. Based on the sample density, select the k-th log sample from the k-th sample set and add it to the central set. The values ​​of k are [1, k], where k is an integer greater than 1.

[0047] In this embodiment, step S204 needs to be executed multiple times based on the value of k, where k can be [1, k]. When k is 1, a first sample set is formed, and a first log sample is selected from the first sample set and added to the central set; when k is 2, a second sample set is formed, and a second log sample is selected from the second sample set and added to the central set; ...; when k is k, a k-th sample set is formed, and a k-th log sample is selected from the k-th sample set and added to the central set. Therefore, the central set will contain multiple log samples.

[0048] Step S204: Merge the log samples in the central set until the number of log samples in the central set is K. In this embodiment, two log samples can be selected from the central set and then merged into a new log sample. This will reduce the number of log samples in the central set by 1. After N merging operations, the number of log samples in the central set will decrease by N, eventually reaching K.

[0049] It should be noted that the total number of log samples added to the central set is usually greater than or equal to K, meaning that the number of log samples in the central set will never be less than K.

[0050] Step S205: Use the K log samples in the merged centroid set as the initial centroids and perform kmeans clustering.

[0051] In this embodiment, the k-means clustering algorithm has been described above. It initially selects K objects randomly as the initial cluster centers. In this embodiment, the selected cluster centers (i.e., the initial centroids) are used as the K objects instead of the k-means clustering algorithm randomly selecting K objects. This can improve the convergence speed of the algorithm and improve the clustering effect.

[0052] This application embodiment filters log samples into a centroid set by utilizing the pre-defined distance and distance range between log samples. Then, it merges the log samples in the centroid set until the number of log samples in the centroid set is the number of initial centroids required by the K-means clustering algorithm. Finally, the log samples in the centroid set are used as the initial centroids, which can improve the selection quality of the initial centroids and further improve the clustering effect of the k-means clustering algorithm.

[0053] In some embodiments, the step S201 above, "obtaining the center of the log sample", can be implemented by the following steps: calculating the cosine distance between each log sample in the log sample set, selecting the two log samples with the largest cosine distance; calculating the center of the two log samples with the largest cosine distance, and using them as the center of the log sample set.

[0054] In this embodiment, the log sample set can be represented as a dataset. The cosine distance between each pair of log samples in the dataset is calculated, and the two log samples with the greatest cosine distance are denoted as X1 and X2. The center of X1 and X2, denoted as Cen, is calculated as the center of the sample set dataset. The centers of log samples X1 and X2 can be calculated by vectorizing them.

[0055] This embodiment selects the two log samples with the largest cosine distance and calculates the center of these two log samples as the center of the log sample set, which is more representative and can improve the accuracy of the subsequent selection of the first log sample, thereby further improving the selection quality of the initial centroid.

[0056] In some embodiments, the step S201 above, "obtaining the average distance of the log sample set", can be implemented by the following steps: calculating the cosine distance between each log sample in the log sample set and summing them to obtain the cumulative sum; dividing the cumulative sum by the square of the total number of log samples in the log sample set to obtain the average distance of the samples.

[0057] In this embodiment, there exists a cosine distance between each log sample in the log sample set. For example, taking a log sample set including log samples A1, A2, and A3, the cosine distance between A1 and A2 is disA1, the cosine distance between A1 and A3 is disA2, and the cosine distance between A2 and A3 is disA3. Then, the sum of the cosine distances = disA1 + disA2 + disA3. The average distance of the samples = the sum of the cosine distances / 3.

[0058] This application embodiment obtains the average distance between log sample sets, and calculates the sample density of each log sample based on the average distance. Based on the sample density, the Kth log sample with a higher sample density can be selected from the Kth sample set and added to the centroid set, thereby improving the quality of log samples in the centroid set and further improving the quality of the selected initial centroid.

[0059] In some embodiments, the step S201 above, "obtaining the sample density of each log sample in the log sample set", can be implemented by the following steps: calculating the total number of log samples in the log sample set whose cosine distance to the log sample is less than the average distance of the samples, and using this as the sample density of the log sample set.

[0060] In this embodiment, after representing log samples by coordinate points, the sample density of a log sample can refer to the number of coordinate points surrounding the log sample. The more coordinate points there are, the greater the sample density of the log sample. The average sample distance is used as the criterion; if there exists an adjacent log sample whose cosine distance to the log sample is less than the average sample distance, the sample density increases by 1.

[0061] For example, Figure 3 A schematic diagram of a log sample provided in an embodiment of this application, as shown below. Figure 3 As shown, log sample A1 has two adjacent log samples, and log sample A2 has three adjacent log samples. The sample density of log sample A2 is greater than that of log sample A1.

[0062] This application embodiment obtains the sample density of each log sample. Based on the sample density, the number of log samples around each log sample can be intuitively understood. This allows for the selection of higher quality log samples into the centroid set, thereby further improving the selection quality of the initial centroid.

[0063] In some embodiments, the above method may further determine the first distance range by the following steps: obtaining a first coefficient threshold and a first constant threshold; calculating the first distance range based on the first coefficient threshold, the first constant threshold and the average distance of the samples.

[0064] In this embodiment, the first coefficient threshold is a, the first constant threshold is b, and the average distance of the samples is R. a and b can be preset thresholds, so the first distance range is [a...]. Rb, a R+b].

[0065] Furthermore, in other embodiments, determining the Kth distance range can be achieved through the following steps: obtaining a second coefficient threshold, a second constant threshold, and the number of initial centroids to be selected; calculating the Kth distance range based on the second coefficient threshold m, the second constant threshold n, and the number of initial centroids to be selected.

[0066] In this embodiment, the number of initial centroids to be selected is K, the second coefficient threshold is m, and the second constant threshold is n. m and n can be set thresholds. Then the range of the Kth distance is [mR / kn, mR / k+n], where the value of K is [1, K].

[0067] For example, when K is 1, the first distance range is [mR-n, mR+n], when K is 2, the second distance range is [mR / 2-n, mR / 2+n], when K is 3, the third distance range is [mR / 3-n, mR / 3+n], ..., when K is K, the Kth distance range is [mR / kn, mR / k+n].

[0068] This application embodiment sets multiple different distance ranges. Based on each distance range, log samples of better quality can be selected into the central set each time log samples are selected, thereby further improving the selection quality of the initial centroid when obtaining the initial centroid from the central set.

[0069] In some embodiments, the step S202 above, "selecting a first log sample from a first sample set", can be implemented by the following steps: obtaining the sample density of each log sample in the first sample set; selecting the log sample with the highest sample density in the first sample set as the first log sample.

[0070] In this embodiment, the log sample set is represented by dataset. The cosine distance between the log samples in dataset and the center is calculated. Log samples whose cosine distance from the center is within the range of [aR-b, aR+b], i.e., the first distance range (where a and b are set thresholds), are selected and grouped into a first sample set dataG. Then, the sample density of the log samples in the first sample set dataG is calculated, and the log sample C1 with the highest density is found and added to the center set as the first log sample.

[0071] Furthermore, in other embodiments, the step S203 above, "selecting the kth log sample from the kth sample set", can be implemented through the following steps: obtaining the sample density of each log sample in the kth sample set; selecting the log sample with the largest sample density value in the kth sample set as the k1st log sample; selecting log samples in the kth sample set whose cosine distance to the k1st log sample is greater than the average distance of the samples, forming a log sample set, and selecting the log sample with the largest density from this log sample set as the k2st log sample; and using the k1st log sample and the k2st log sample as the kth log sample.

[0072] In this embodiment, the cosine distance between each log sample in the log sample set dataset and the first log sample C1 is calculated. Samples whose cosine distance to the first log sample C1 is within the range [mR / kn, mR / k+n] are selected (where m and n are set thresholds, and k is the number of centroids to be selected; in this case, k is 1, i.e., the range is [mR-n, mR+n]). These log samples are then grouped into a log sample set dataG1. Next, the log sample C11 with the highest density is selected from the log sample set dataG1, and then the sample with the second highest density and whose cosine distance to log sample C11 is greater than the average distance R of the samples is selected and denoted as log sample C12. Log sample C11 is the k1th log sample, and log sample C12 is the k2th log sample. Log samples C11 and C12 are then added to the centroid set.

[0073] This application embodiment selects qualified log samples from the log sample set and adds them to the central set, which can ensure the quality of the log samples in the central set, thereby further improving the selection quality of the initial centroid when determining the initial centroid from the central set.

[0074] In some embodiments, step S204 can be implemented by the following steps: calculating the cosine distance of each log sample in the central set and merging the two log samples with the closest cosine distance to obtain a new log sample; determining whether the number of log samples in the central set is K; if the number of log samples in the central set Cens is not K, then continuing to calculate the cosine distance of each log sample in the central set and merging the two log samples with the closest cosine distance to obtain a new log sample.

[0075] In this embodiment, log samples in the central set Cens are fused as follows: Assuming there are q log samples in the central set Cens, the cosine distance between each pair of log samples in the central set Cens is calculated, and the two log samples with the closest cosine distance are fused. The fusion method is as follows:

[0076] Log sample x and log sample y are merged into log sample new, and log sample new = (log sample x) / y. Density of log sample x + density of log sample y The density of log sample y is (density of log sample y) / (density of log sample x + density of log sample y), and at this time the density of log sample new is (density of log sample x + density of log sample y) / 2.

[0077] At this time, the number of log samples in the central set Cens is reduced to q - 1. Then, calculate the cosine distances between the log samples in the central set Cen pairwise, and find the two closest log samples for fusion until the number of samples in the central set Cen reaches the preset K. At this time, the K log samples in the central set Cens are the calculated initial centroids.

[0078] In the embodiment of the present application, by continuously fusing the log samples in the central set, new log samples are obtained. Finally, the K log samples obtained by fusion are used as the K initial centroids, which can ensure the quality of these initial centroids and improve the effect of the Kmeans clustering algorithm, thereby further realizing the accuracy of error log classification.

[0079] Exemplarily, Table 1 below shows the code logic of the kmeans log classification method based on partition distance fusion provided by the embodiment of the present application:

[0080]

[0081] Table 1

[0082] Exemplarily, Figure 4 is the overall flowchart of the kmeans log classification method based on partition distance fusion provided by the embodiment of the present application. As Figure 4 shown, it includes the following steps: Step S401, calculate the cosine distances between the log samples pairwise, and find the two log samples with the farthest distance, denoted as X1 and X2. Step S402, calculate the center of X1 and X2, denoted as Cen, as the center of the log sample set dataset. Step S403, calculate the sum of the cosine distances between the log samples in dataset pairwise divided by the square of the total number of log samples in dataset, as the sample average distance R. Step S404, calculate the density of each log sample in dataset. Step S405, calculate the cosine distance between the log samples in dataset and the center Cen, and select the log samples whose cosine distance from the center Cen is within [a R - b, a R + b]. These log samples form the set dataG. Step S406, for the log samples in the set dataG, select the log sample with the largest density, denoted as C1, add C1 to the set Cens, and set i = 0. Step S407, determine whether i < K. Step S4081, when i is less than K, i = i + 1. Step S4082, when i is greater than or equal to K, fuse the log samples in the set Cens until there are K. Step S409, calculate the cosine distance between the log samples in dataset and C1, and select the log samples whose cosine distance from C1 is within [m R / k - n, m The log sample is R / k+n], where k takes the value i, i.e., the range is [m]. R / in, m R / i+n], these log samples are grouped into a set dataGi, the log sample with the highest density is selected from dataGi Ci1, and then the log sample with the second highest density and a cosine distance greater than R from Ci1 is selected from dataGi and denoted as Ci2. Ci1 and Ci2 are added to the set Cens.

[0083] This application improves the initialization phase of the kmenes algorithm by proposing a method based on gap distance fusion to disperse the selection of centroids. This accelerates the initialization convergence speed and improves the quality of initial centroid selection. The resulting clustering quality is better, leading to more accurate classification of log data.

[0084] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0085] Figure 5 A schematic diagram of the structure of the k-means log classification device based on gap distance fusion provided in the embodiments of this application is shown below. Figure 5 As shown, the log classification device 500 includes an acquisition module 510, a sample selection module 520, a sample cyclic selection module 530, a sample fusion module 540, and a clustering module 550.

[0086] The acquisition module 510 is used to acquire the log sample set, the center of the log sample set, the average distance between the log samples in the log sample set, and the sample density of each log sample in the log sample set. The log sample set includes at least two log samples. The sample selection module 520 is used to select log samples from the log sample set whose cosine distance to the center is within a first distance range to form a first sample set. Based on the sample density, the first log sample is selected from the first sample set and added to the center set. The sample iterative selection module 530 is used to select log samples from the log sample set whose cosine distance to the first log sample is within a K distance range to form a Kth sample set. Based on the sample density, the Kth log sample is selected from the Kth sample set and added to the center set. The value of K is [1, K], where K is an integer greater than 1. The sample fusion module 540 is used to fuse the log samples in the center set until the number of log samples in the center set is K. The clustering module 550 is used to use the K log samples in the fused center set as initial centroids and perform k-means clustering.

[0087] Optionally, the acquisition module can be used to: calculate the cosine distance between each log sample in the log sample set, and select the two log samples with the largest cosine distance; calculate the center of the two log samples with the largest cosine distance, and use them as the center of the log sample set.

[0088] Optionally, the acquisition module can be used to: calculate and sum the cosine distances between each log sample in the log sample set to obtain the cumulative sum; divide the cumulative sum by the square of the total number of log samples in the log sample set to obtain the average distance between the samples.

[0089] Optionally, the acquisition module can be used to: calculate the total number of log samples in the log sample set whose cosine distance to the log sample is less than the average distance of the samples, and use this as the sample density of the log sample.

[0090] Optionally, it also includes a first distance range calculation module, used to obtain a first coefficient threshold and a first constant threshold; and to calculate the first distance range based on the first coefficient threshold, the first constant threshold and the average distance of the samples.

[0091] Optionally, a second distance range calculation module is also included, used to obtain a second coefficient threshold, a second constant threshold, and the number of initial centroids to be selected; and to calculate the Kth distance range based on the second coefficient threshold m, the second constant threshold n, and the number of initial centroids to be selected.

[0092] Optionally, the sample selection module can be used to: obtain the sample density of each log sample in the first sample set; and select the log sample with the highest sample density in the first sample set as the first log sample.

[0093] Optionally, the sample cyclic selection module can be used to: obtain the sample density of each log sample in the Kth sample set; select the log sample with the largest sample density value in the Kth sample set as the K1st log sample; select log samples in the Kth sample set whose cosine distance to the K1st log sample is greater than the average distance of the samples, form a log sample set, and select the log sample with the largest density from the log sample set as the K2st log sample; and use the K1st log sample and the K2st log sample as the Kth log sample.

[0094] Optionally, the sample fusion module can be used to: calculate the cosine distance of each log sample in the central set, and fuse the two log samples with the closest cosine distance to obtain a new log sample; determine whether the number of log samples in the central set is K; if the number of log samples in the central set Cens is not K, then continue to calculate the cosine distance of each log sample in the central set, and fuse the two log samples with the closest cosine distance to obtain a new log sample.

[0095] The apparatus provided in this application embodiment can be used to execute the methods in the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0096] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the acquisition module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0097] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 includes at least one processor 601, a memory 602, a bus 603, and a communication interface 604. The processor, communication interface, and memory communicate with each other via the bus. The communication interface is used to communicate with other devices. This communication interface includes a communication interface for data transmission and a display interface or operation interface for human-computer interaction. The processor executes computer instructions stored in the memory, specifically performing the relevant steps in the methods described in the above embodiments.

[0098] The processor may be a central processing unit, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0099] Memory is used to store instructions executed by a computer. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0100] This embodiment also provides a computer-readable storage medium storing computer instructions, which, when executed by at least one processor of an electronic device, enable the electronic device to perform the methods provided in the various embodiments described above.

[0101] This embodiment also provides a computer program product including computer instructions stored in a readable storage medium. At least one processor of an electronic device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to cause the electronic device to perform the methods provided in the various embodiments described above.

[0102] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0103] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0105] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A k-means log classification method based on gap distance fusion, characterized in that, include: Obtain a log sample set, the center of the log sample set, the average distance between the log samples in the log sample set, and the sample density of each log sample in the log sample set, wherein the log sample set includes at least two log samples; Log samples with a cosine distance to the center within a first distance range are selected from the log sample set to form a first sample set. Based on the sample density, a first log sample is selected from the first sample set and added to the center set. Based on the value of k, log samples whose cosine distance to the first log sample is within the k-th distance range are selected from the log sample set to form the k-th sample set. Based on the sample density, the k-th log sample is selected from the k-th sample set and added to the central set. The values ​​of k are [1, k], where k is an integer greater than 1. Merge the log samples in the central set until the number of log samples in the central set is K; K log samples from the merged centroid set are used as initial centroids for kmean clustering.

2. The method according to claim 1, characterized in that, Obtaining the center of the log sample set includes: Calculate the cosine distance between each log sample in the log sample set, and select the two log samples with the largest cosine distance. Calculate the centers of the two log samples with the largest cosine distance, and use them as the centers of the log sample set.

3. The method according to claim 1, characterized in that, Obtaining the average distance between the log sample set includes: Calculate the cosine distances between all log samples in the log sample set and sum them to obtain the cumulative sum; The average distance between the samples is obtained by dividing the sum by the square of the total number of log samples in the log sample set.

4. The method according to claim 1, characterized in that, Obtaining the sample density of each log sample in the log sample set includes: The total number of log samples in the log sample set whose cosine distance to the log sample is less than the average distance of the samples is calculated as the sample density of the log sample.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the first coefficient threshold and the first constant threshold; The first distance range is calculated based on the first coefficient threshold, the first constant threshold, and the average distance of the samples.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the second coefficient threshold, the second constant threshold, and the number of initial centroids to be selected; The Kth distance range is calculated based on the second coefficient threshold, the second constant threshold, and the number of initial centroids to be selected.

7. The method according to claim 1, characterized in that, The step of selecting the first log sample from the first sample set includes: Obtain the sample density of each log sample in the first sample set; The log sample with the highest sample density is selected from the first sample set and used as the first log sample.

8. The method according to claim 1, characterized in that, The step of selecting the k-th log sample from the k-th sample set includes: Obtain the sample density of each log sample in the k-th sample set; The log sample with the largest sample density value is selected from the k-th sample set and designated as the k1-th log sample. In the k-th sample set, log samples whose cosine distance to the k1-th log sample is greater than the average distance of the samples are selected to form a log sample set, and the log sample with the highest density is selected from the log sample set as the k2-th log sample. The k1th log sample and the k2th log sample are used as the kth log sample.

9. The method according to claim 1, characterized in that, The process of merging log samples in the central set until the number of log samples in the central set is K includes: Calculate the cosine distance between each log sample in the central set, and merge the two log samples with the closest cosine distance to obtain a new log sample; Determine whether the number of log samples in the central set is K; If the number of log samples in the central set Cens is not K, then continue to calculate the cosine distance of each log sample in the central set, and merge the two log samples with the closest cosine distance to obtain a new log sample.

10. A k-means log classification device based on gap distance fusion, characterized in that, include: The acquisition module is used to acquire a log sample set, the center of the log sample set, the average distance between the log samples in the log sample set, and the sample density of each log sample in the log sample set, wherein the log sample set includes at least two log samples; The sample selection module is used to select log samples from the log sample set whose cosine distance to the center is within a first distance range to form a first sample set, and select a first log sample from the first sample set according to the sample density and add it to the center set; The sample cyclic selection module is used to select log samples from the log sample set whose cosine distance to the first log sample is within the k-th distance range according to the value of k, to form the k-th sample set, and select the k-th log sample from the k-th sample set according to the sample density, and add it to the central set. The value of k is [1, k], where k is an integer greater than 1. The sample fusion module is used to fuse log samples in the central set until the number of log samples in the central set is K. The clustering module is used to perform kmeans clustering on the K log samples in the merged centroid set as the initial centroids.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, When executed by a processor, the computer program product is used to implement the method as described in any one of claims 1 to 9.

Images