Log text clustering method and apparatus, processor, and electronic device

Abstract

The application discloses a log text clustering method and device, a processor and electronic equipment. The method is applied to the technical field of big data. The method comprises the following steps: obtaining a candidate K value set; clustering log texts in a log text set according to N candidate K values to obtain N clustering results; determining a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set; and determining a target clustering result of the log text set according to the clustering result corresponding to the target K value. Through the application, the problem that the accuracy of the clustering result of the log text is relatively low due to the subjectivity and blindness of the preset value determined by human experience when the preset value determined by human experience is used as the number of the centroid to perform K-Means clustering on the log text in the related art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular to a log text clustering method and device, a processor and an electronic device. BACKGROUND

[0002] When the traditional K-Means algorithm clusters log texts, the number of classes of existing log data can be estimated based on human experience, that is, the K value corresponding to the K-Means algorithm is preset, and the K value is equal to the number of divided classes. For example, a plurality of classes of clustering results are set based on human experience, and then the log texts in the log text set are divided into the plurality of classes that have been set.

[0003] However, for a set of unknown log texts or when some log text data is added to an existing log text set, the staff may not clearly perceive the overall distribution of the data, resulting in the K value set based on human experience having a certain blindness and subjectivity, that is, the K value is not suitable for the log text set to be clustered, and thus the clustering result obtained according to the K value is inaccurate.

[0004] In the related art, when the K-Means algorithm is used to cluster log texts by taking a preset value determined based on human experience as the number of centroids, the preset value determined based on human experience has subjectivity and blindness, resulting in a relatively low accuracy of the clustering result of the log texts. Currently, no effective solution has been proposed. SUMMARY

[0005] The main purpose of the present application is to provide a log text clustering method, device, processor and electronic device to solve the problem that when the K-Means algorithm is used to cluster log texts by taking a preset value determined based on human experience as the number of centroids, the preset value determined based on human experience has subjectivity and blindness, resulting in a relatively low accuracy of the clustering result of the log texts.

[0006] To achieve the above purpose, according to one aspect of the present application, a log text clustering method is provided, which includes: obtaining a candidate K value set, wherein the candidate K value set includes N candidate K values, the candidate K value is a preset number of centroids, N is a positive integer, and K is an integer greater than 1; clustering log texts in a log text set according to N candidate K values to obtain N clustering results; determining a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set; and determining a target clustering result of the log text set according to the clustering result corresponding to the target K value.

[0007] Further, the acquiring the candidate K value set comprises: randomly selecting a preset number of log texts in the log text set to obtain a first data set; clustering the log texts in the first data set for multiple times to obtain multiple initial K values, and determining a first K value according to the multiple initial K values; determining a numerical selection range with the first K value as a center, and determining N candidate K values according to the numerical selection range; and generating the candidate K value set based on the N candidate K values.

[0008] Further, the clustering the log texts in the log text set according to the N candidate K values to obtain N clustering results comprises: for each candidate K value, determining K centroids corresponding to the candidate K value in the log text set; clustering the log texts in the log text set according to the K centroids to obtain a clustering result corresponding to the candidate K value; and synthesizing the clustering results corresponding to all the candidate K values to obtain the N clustering results.

[0009] Further, the determining the K centroids corresponding to each candidate K value in the log text set comprises: calculating the cosine distances between the log texts in the log text set to obtain a cosine distance set corresponding to each log text; calculating the number of cosine distances less than a preset threshold in the cosine distance set corresponding to each log text to obtain the density of each log text; and determining the K centroids corresponding to the candidate K value in the log text set according to the density of each log text.

[0010] Further, the determining the K centroids corresponding to each candidate K value in the log text set according to the density of each log text comprises: sorting the log texts in the log text set according to the density of each log text, and determining a first log text in the first order as a first centroid; determining a second log text according to the cosine distance between the first log text and the log texts in the log text set except the first log text, and determining the second log text as a second centroid; determining the target cosine distance of each log text according to the cosine distance between the first centroid and the log texts in the log text set and the cosine distance between the second centroid and the log texts in the log text set, and determining a next centroid according to the target cosine distance of each log text; and repeating the steps of determining the target cosine distance of each log text according to the cosine distance between the log texts in the log text set and the determined centroids, and determining a next centroid according to the target cosine distance of each log text until the Kth centroid is determined.

[0011] Further, the determining the target K value and the clustering result corresponding to the target K value according to the N clustering results and the candidate K value set comprises: determining K clusters corresponding to each clustering result according to the N clustering results, wherein each cluster comprises a plurality of log texts; for each clustering result, calculating cosine distances between cluster centers of each cluster corresponding to the clustering result to obtain first distances corresponding to the clustering result; for each clustering result, calculating cosine distances between cluster centers of each cluster corresponding to the clustering result and log texts in the cluster to obtain second distances corresponding to the clustering result; calculating according to the first distances and the second distances to obtain a clustering score corresponding to each clustering result; determining the target K value from the candidate K value set and the clustering result corresponding to the target K value from the candidate K value set according to the clustering score corresponding to each clustering result.

[0012] Further, after the target clustering result of the log text set is determined according to the clustering result corresponding to the target K value, the method further comprises: determining fault information of a target device according to the target clustering result, wherein the target device is a device corresponding to the log text set; and sending the fault information to a target object, wherein the target object optimizes the target device according to the fault information.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a clustering device of log texts is provided, which comprises: an acquisition unit configured to acquire a candidate K value set, wherein the candidate K value set comprises N candidate K values, the candidate K value is a preset number of centroids, N is a positive integer, and K is an integer greater than 1; a clustering unit configured to cluster log texts in a log text set according to N candidate K values to obtain N clustering results; a first determination unit configured to determine a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set; and a second determination unit configured to determine a target clustering result of the log text set according to the clustering result corresponding to the target K value.

[0014] Further, the acquisition unit comprises: a first determination sub-unit configured to randomly select a preset number of log texts in the log text set to obtain a first data set; a first clustering sub-unit configured to cluster log texts in the first data set multiple times to obtain a plurality of initial K values, and determine a first K value according to the plurality of initial K values; a second determination sub-unit configured to determine a numerical selection range with the first K value as a center, and determine N candidate K values according to the numerical selection range; and a generation sub-unit configured to generate the candidate K value set based on the N candidate K values.

[0015] Further, the clustering unit comprises: a third determining sub-unit, configured to determine K centroids corresponding to the candidate K value in the log text set for each candidate K value; a second clustering sub-unit, configured to cluster log texts in the log text set according to the K centroids to obtain a clustering result corresponding to the candidate K value; and a fourth determining sub-unit, configured to obtain the N clustering results by synthesizing the clustering results corresponding to all candidate K values.

[0016] Further, the third determining sub-unit comprises: a first calculating module, configured to calculate cosine distances between log texts in the log text set to obtain a cosine distance set corresponding to each log text; a second calculating module, configured to calculate a number of cosine distances less than a preset threshold in the cosine distance set corresponding to each log text to obtain a density of each log text; and a determining module, configured to determine K centroids corresponding to the candidate K value in the log text set according to the density of each log text.

[0017] Further, the determining module comprises: a sorting sub-module, configured to sort log texts in the log text set according to the density of each log text, and determine a first log text in the first order as a first centroid; a first determining sub-module, configured to determine a second log text according to cosine distances between the first log text and log texts in the log text set except the first log text, and determine the second log text as a second centroid; a second determining sub-module, configured to determine a target cosine distance of each log text according to cosine distances between the first centroid and log texts in the log text set and cosine distances between the second centroid and log texts in the log text set, and determine a next centroid according to the target cosine distance of each log text; and a processing sub-module, configured to repeat the steps of determining a target cosine distance of each log text according to cosine distances between log texts in the log text set and determined centroids, and determining a next centroid according to the target cosine distance of each log text, until the Kth centroid is determined.

[0018] Further, the first determining unit comprises: a fifth determining sub-unit, configured to determine K clusters corresponding to each clustering result according to the N clustering results, wherein each cluster comprises a plurality of log texts; a first calculating sub-unit, configured to calculate cosine distances between cluster centers of each cluster corresponding to each clustering result to obtain a first distance corresponding to the clustering result; a second calculating sub-unit, configured to calculate cosine distances between cluster centers of each cluster and log texts in the cluster corresponding to each clustering result to obtain a second distance corresponding to the clustering result; a third calculating sub-unit, configured to calculate the first distance and the second distance to obtain a clustering score corresponding to each clustering result; and a sixth determining sub-unit, configured to determine a target K value from the candidate K value set and determine a clustering result corresponding to the target K value from the candidate K value set according to the clustering score corresponding to each clustering result.

[0019] Further, the apparatus further comprises: a third determining unit, configured to determine fault information of a target device according to a target clustering result of the log text set after the target clustering result is determined according to the clustering result corresponding to the target K value, wherein the target device is a device corresponding to the log text set; and a sending unit, configured to send the fault information to a target object, wherein the target object optimizes the target device according to the fault information.

[0020] In order to achieve the above-mentioned purpose, according to an aspect of the present application, a processor is provided, and the processor is used to run a program, wherein when the program is running, the above-mentioned any one of the clustering method of log texts is executed.

[0021] In order to achieve the above-mentioned purpose, according to an aspect of the present application, an electronic device is provided, comprising one or more processors and a memory, the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned any one of the clustering method of log texts.

[0022] This application employs the following steps: obtaining a candidate K-value set, wherein the candidate K-value set includes N candidate K-values, the candidate K-values ​​being a preset number of centroids, N being a positive integer, and K being an integer greater than 1; clustering the log texts in the log text set based on the N candidate K-values ​​to obtain N clustering results; determining a target K-value and the corresponding clustering result based on the N clustering results and the candidate K-value set; and determining the target clustering result of the log text set based on the clustering result corresponding to the target K-value. This solves the problem in related technologies where, when using preset values ​​determined by human experience as the number of centroids for K-Means clustering of log texts, the accuracy of the log text clustering results is relatively low due to the subjectivity and blindness of the preset values ​​determined by human experience. Clustering is performed on each candidate K value in the candidate K-value set to obtain multiple clustering results. The target clustering result is determined from these multiple clustering results based on the cosine distance between log texts in the log text set. This method can adaptively adjust the number of centroids used by the K-Means clustering algorithm according to the log texts in the log text set, avoiding the problem of inaccurate clustering results caused by the subjectivity and blindness of human experience when setting the number of centroids. This achieves the effect of improving the clustering quality of the K-Means clustering algorithm, and further improves the clustering quality of the log text set. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 This is a flowchart of the log text clustering method provided in Embodiment 1 of this application;

[0025] Figure 2 This is a schematic diagram of an optional log text clustering method provided according to Embodiment 1 of this application;

[0026] Figure 3 This is a schematic diagram of a log text clustering device provided according to Embodiment 2 of this application;

[0027] Figure 4 This is a schematic diagram of an electronic device for clustering log text according to Embodiment 5 of this application. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the user information (including but not limited to user equipment information, user personal information, user information contained in log text, etc.) and data (including but not limited to data for analysis, stored data, displayed data, processed data, calculated data, input data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0030] In order to enable persons skilled in the art to better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should belong to the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment one

[0033] The present application will be described below in combination with preferred implementation steps, Figure 1 is a flow chart of the clustering method of log text provided according to the first embodiment of the present application, as Figure 1 shown, the method comprises the following steps:

[0034] Step S101, obtaining a candidate K value set, wherein the candidate K value set includes N candidate K values, the candidate K value is a preset number of centroids, N is a positive integer, and K is an integer greater than 1.

[0035] In the first embodiment, the candidate K value set is a set of numerical values, and the candidate K value set contains N candidate K values. The candidate K value represents the K value to be determined, wherein the K value represents the number of centroids used when the K-Means method is used for clustering. Specifically, the candidate K value set can be set to {5, 6, 7, 8, 9, 10, 11, 12}.

[0036] In step S102, the log texts in the log text set are clustered according to the N candidate K values, and N clustering results are obtained.

[0037] In the first embodiment, it is assumed that the candidate K value set includes {5, 6, 7, 8, 9, 10, 11, 12}, and the log texts in the log text set are clustered according to the candidate K value set, and eight clustering results can be obtained as shown in the following steps: first, the value 5 in the candidate K value set is taken as the preset number of centroids of K-Means clustering, and the first clustering result is obtained, wherein the first clustering result includes five clusters, i.e., five types of log texts; similarly, seven clustering results are obtained by clustering according to the other values in the candidate K value set except the value 5; and the eight clustering results are obtained by integrating the above clustering results.

[0038] It should be noted that before clustering the log texts in the log text set, the log texts of the target device need to be obtained, and the log texts are preprocessed to convert the log texts into word vectors to obtain the log text set. Specifically, the log texts in a preset time period (for example, 6 months, 1 year, etc.) in the target device containing the log texts are obtained; then, the obtained log texts are preprocessed (for example, word segmentation processing, stop word removal processing, etc.), and preprocessed log texts are obtained; secondly, the preprocessed log texts are input into the word2vec model for processing to obtain word vectors; and finally, the log text set is composed of the word vectors corresponding to the log texts.

[0039] In step S103, the target K value and the clustering result corresponding to the target K value are determined according to the N clustering results and the candidate K value set.

[0040] In the first embodiment, the clustering score of each clustering result in the N clustering results can be calculated, and the target K value and the clustering result corresponding to the target K value are determined according to the clustering score. When calculating the clustering score of the clustering result, the cosine distance between the log texts in different clusters in the clustering result can be calculated, or the cosine distance between the log texts in each cluster in the clustering result can also be calculated.

[0041] In step S104, the target clustering result of the log text set is determined according to the clustering result corresponding to the target K value.

[0042] In the first embodiment, the clustering result corresponding to the target K value can be directly determined as the target clustering result, or the target clustering result can be obtained by processing the clustering result corresponding to the target K value. For example, the word vector of the log text in the clustering result is restored to text data, the text data is segmented, the importance score of each word in the cluster to which the word belongs is calculated using the TF-IDF algorithm, and the word with the highest score is selected as the topic keyword of the cluster, and the target clustering result containing the topic keyword corresponding to each cluster is obtained.

[0043] In summary, the clustering method for log texts provided in the first embodiment of the present application includes the following steps: obtaining a candidate K value set, wherein the candidate K value set includes N candidate K values, the candidate K value is a preset number of centroids, N is a positive integer, and K is an integer greater than 1; clustering log texts in a log text set according to the N candidate K values to obtain N clustering results; determining a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set; and determining a target clustering result of the log text set according to the clustering result corresponding to the target K value. The problem of low accuracy of the clustering result of the log texts caused by the subjectivity and blindness of the preset value determined by human experience when the preset value is used as the number of centroids for K-Means clustering of the log texts in the related art is solved. By clustering each candidate K value in the candidate K value set to obtain multiple clustering results, and determining the target clustering result in the multiple clustering results according to the cosine distance between the log texts in the log text set, the number of centroids used by the K-Means clustering algorithm can be adaptively adjusted according to the log texts in the log text set, avoiding the problem of inaccurate clustering results caused by the subjectivity and blindness of human experience when setting the number of centroids. The clustering quality of the clustering result of the K-Means clustering algorithm is improved, and the clustering quality of the clustering result of the log text set is further improved.

[0044] Optionally, in the clustering method for log texts provided in the first embodiment of the present application, obtaining the candidate K value set includes: randomly selecting a preset number of log texts in the log text set to obtain a first data set; clustering the log texts in the first data set multiple times to obtain multiple initial K values, and determining a first K value according to the multiple initial K values; determining a value selection range with the first K value as the center, and determining N candidate K values according to the value selection range; and generating the candidate K value set based on the N candidate K values.

[0045] In the first embodiment, in order to determine the number of centroids in K-Means clustering, a K value (i.e., the first K value) can be determined, and then a candidate K value set can be determined according to the first K value. Specifically, in order to improve the clustering efficiency of the log text set, 15% of the log texts in the log text set (i.e., the preset number of log texts) can be randomly selected to obtain a first data set. Then, the elbow rule can be used to determine the first K value in the first data set (i.e., the log texts in the first data set are clustered multiple times to obtain multiple initial K values, and then a line chart with the K value as the horizontal axis and the sum of squares for error (SSE) as the vertical axis is drawn according to the SSE of the clustering result corresponding to each initial K value, and the K value corresponding to the inflection point in the line chart is determined as the first K value). Next, a numerical selection range is determined based on the first K value, and N candidate K values are determined. For example, if the first K value is equal to 10, the integers whose absolute difference with 10 is less than 5 can be used as candidate K values, i.e., {6, 7, 8, 9, 10, 11, 12, 13, 14}. Finally, the candidate K value set is composed of the N candidate K values.

[0046] The first K value is determined by the elbow rule, and then the candidate K value set is determined according to the first K value, which avoids the problem that the K value obtained by the elbow rule is inaccurate, and at the same time provides a suitable K value selection range (i.e., the candidate K value set) for the K-Means clustering algorithm, which is beneficial to adaptively determining the number of centroids used in clustering in the candidate K value set.

[0047] Optionally, in the log text clustering method provided in the first embodiment of the present application, the log texts in the log text set are clustered according to the N candidate K values to obtain N clustering results, including: for each candidate K value, determining K centroids corresponding to the candidate K value in the log text set; clustering the log texts in the log text set according to the K centroids to obtain a clustering result corresponding to the candidate K value; and integrating all the clustering results corresponding to the candidate K values to obtain N clustering results.

[0048] In the first embodiment, in order to obtain the clustering result corresponding to each candidate K value in the candidate K value set, the K centroids corresponding to the candidate K value can be determined according to the cosine distance between the log texts in the log text set; then, clustering is performed according to the K centroids corresponding to the candidate K value to obtain the clustering result corresponding to the candidate K value; and finally, all the clustering results corresponding to the candidate K values are integrated to obtain N clustering results.

[0049] Specifically, if the candidate K values include {9, 10, 11}, 9 centroids are determined according to the cosine distances between the log texts in the log text set, and the 9 centroids are clustered by using the K-Means algorithm to obtain clustering results corresponding to the 9 centroids (which include 9 clusters, and each cluster contains one centroid). Then, 10 centroids and 11 centroids are respectively determined according to the cosine distances between the log texts in the log text set, and the 10 centroids and the 11 centroids are respectively clustered by using the K-Means algorithm to obtain clustering results corresponding to the 10 centroids and clustering results corresponding to the 11 centroids. Finally, the three clustering results corresponding to the candidate K values {9, 10, 11} are determined according to the clustering results corresponding to the 9 centroids, the clustering results corresponding to the 10 centroids, and the clustering results corresponding to the 11 centroids.

[0050] By calculating the cosine distances between the log texts in the log text set and determining the clustering results corresponding to each candidate K value, the problem that the noise (i.e., the log text in the log text set that is far away from other log texts and is distributed relatively dispersedly) in the log text set is taken as a centroid for clustering by the traditional K-Means algorithm through a random selection method, thereby resulting in a poor clustering result, is avoided, the convergence speed of the clustering process is improved, and the effect of improving the clustering efficiency is achieved.

[0051] Optionally, in the log text clustering method provided in Embodiment One of the present application, for each candidate K value, the K centroids corresponding to the candidate K value are determined in the log text set by: calculating the cosine distances between the log texts in the log text set to obtain a cosine distance set corresponding to each log text; calculating the number of cosine distances less than a preset threshold in the cosine distance set corresponding to each log text to obtain the density of each log text; and determining the K centroids corresponding to the candidate K value in the log text set according to the density of each log text.

[0052] In Embodiment One, in order to avoid taking a discrete point as a clustering centroid, the cosine distances between the log texts in the log text set are calculated to obtain the density of each log text, and then the centroid for clustering is determined according to the density of each log text.

[0053] Specifically, cosine distances between log texts in the set of log texts are calculated to obtain cosine distances between each log text and other log texts, and a set of cosine distances of the log text is composed of the cosine distances between the log text and other log texts. Then, the number of cosine distances less than a preset threshold (for example, 0.9, 1.5, or the like) in the set of cosine distances of the log text is calculated to obtain the density of the log text. Finally, K log texts with the largest density in the set of log texts can be used as K centroids corresponding to a candidate K value; or K centroids corresponding to a candidate K value can be determined according to the density of the log text and the cosine distances between the log texts, for example, a log text with the largest density in the set of log texts is used as a first centroid corresponding to a candidate K value, a log text with the largest cosine distance from the first centroid is used as a second centroid, a log text with the largest cosine distance from the last centroid is repeatedly selected and determined as a next centroid until K centroids are obtained.

[0054] By calculating the density of each log text in the set of log texts, a centroid required for clustering can be determined in the set of log texts according to the density of each log text and the cosine distances between the log texts, so that the obtained centroid has a smaller cosine distance from other log texts in a local area, which to some extent solves the problem of local optimal solution in the clustering process, avoids the problem of using noise in the set of log texts as a centroid, and achieves the effect of improving the clustering quality of the clustering result.

[0055] Optionally, in the method for clustering log texts provided in Embodiment One of the present application, determining K centroids corresponding to a candidate K value in the set of log texts according to the density of each log text comprises: sorting log texts in the set of log texts according to the density of each log text, and determining a first log text in the first order as a first centroid; determining a second log text according to the cosine distance between the first log text and each log text in the set of log texts except the first log text, and determining the second log text as a second centroid; determining a target cosine distance of each log text according to the cosine distance between the first centroid and the log text in the set of log texts and the cosine distance between the second centroid and the log text in the set of log texts, and determining a next centroid according to the target cosine distance of each log text; and repeatedly performing the steps of determining a target cosine distance of each log text according to the cosine distance between the log text in the set of log texts and the determined centroid, and determining a next centroid according to the target cosine distance of each log text, until the Kth centroid is determined.

[0056] Specifically, the log text with the largest density in the set of log texts can be determined as the first centroid. Then, the log text with the largest cosine distance from the first centroid in the set of log texts is determined as the second centroid. Next, the cosine distance of each log text in the set of log texts from the first centroid and the cosine distance of the log text from the second centroid are calculated, and the cosine distance with the smaller value between the cosine distance of the log text from the first centroid and the cosine distance of the log text from the second centroid is taken as the target cosine distance of the log text. Then, the log text with the largest target cosine distance in the set of log texts is determined as the third centroid. Finally, the cosine distance of the log text in the set of log texts from the determined centroid is repeatedly calculated, the target cosine distance of the log text is determined, and the log text with the largest target cosine distance is determined as the next centroid until K centroids are obtained.

[0057] It should be noted that when the cosine distance of the log text in the set of log texts from the determined centroid is calculated, the cosine distance between the determined centroids is not calculated, and only the cosine distance of the log text in the set of log texts from the determined centroid is calculated.

[0058] By calculating the density of each log text in the set of log texts and the cosine distance between the log texts in the set of log texts, it is ensured that the obtained centroids have no noise, and the obtained centroids are more dispersed, the quality of the centroids used for clustering is improved, and the effect of improving the clustering quality of the clustering result is achieved.

[0059] Optionally, in the log text clustering method provided in Embodiment One of the present application, the target K value and the clustering result corresponding to the target K value are determined according to the N clustering results and the set of candidate K values, comprising: determining the K clusters corresponding to each clustering result according to the N clustering results, wherein each cluster includes a plurality of log texts; for each clustering result, calculating the cosine distance between the cluster centers of each cluster corresponding to the clustering result to obtain the first distance corresponding to the clustering result; for each clustering result, calculating the cosine distance between the cluster center of each cluster corresponding to the clustering result and the log text in the cluster to obtain the second distance corresponding to the clustering result; calculating the clustering score corresponding to each clustering result according to the first distance and the second distance; and determining the target K value from the set of candidate K values and the clustering result corresponding to the target K value from the set of candidate K values according to the clustering score corresponding to each clustering result.

[0060] Specifically, the K clusters corresponding to each clustering result are determined according to the N clustering results. For example, the candidate K value set includes {9, 10, 11}, when the candidate K value is 9, 9 clusters (each cluster includes multiple log texts) are determined; when the candidate K value is 10, 10 clusters are determined; and when the candidate K value is 11, 11 clusters are determined. Then, the first distance corresponding to each clustering result can be calculated according to Formula (1) as follows:

[0061]

[0062] wherein Dist_avg represents the first distance corresponding to a clustering result, K represents the candidate K value, C i represents the cluster center of the i-th cluster in the clustering result, C j represents the cluster center of the j-th cluster in the clustering result, and D(C i , C j ) represents the cosine distance between the cluster center of the i-th cluster and the cluster center of the j-th cluster. The cluster center of the i-th cluster represents the average value of the multiple log texts in the i-th cluster (i.e., the average value of the word vectors of the multiple log texts). Secondly, the maximum cosine distance between each cluster and the cluster center is determined as the maximum intra-cluster distance of the cluster. The average value of the maximum intra-cluster distances of all clusters in the clustering result is calculated to obtain the second distance of the clustering result, and then the second distance corresponding to each clustering result is obtained. Then, the clustering score corresponding to each clustering result can be obtained by dividing the first distance corresponding to each clustering result by the second distance corresponding to each clustering result. Finally, the K value corresponding to the highest clustering score in the N clustering scores corresponding to the N clustering results is determined as the target K value, and the clustering result corresponding to the target K value is determined.

[0063] By calculating the cosine distance between the intra-cluster log texts of the multiple clusters in each clustering result, the first distance corresponding to each clustering result and the second distance corresponding to each clustering result are obtained, which is beneficial to evaluate the clustering quality of the N clustering results according to the first distance and the second distance, and then determine the clustering result with higher clustering quality in the N clustering results, thereby achieving the effect of improving the clustering quality of the clustering result.

[0064] Optionally, in the log text clustering method provided by the first embodiment of the present application, after the target clustering result of the log text set is determined according to the clustering result corresponding to the target K value, the above method further includes: determining the fault information of the target device according to the target clustering result, wherein the target device is the device corresponding to the log text set; and sending the fault information to the target object, wherein the target object optimizes the target device according to the fault information.

[0065] In the first embodiment, in order to optimize the target device, one or more types of log texts in the target clustering result can be determined, and the log texts are processed to obtain the fault information corresponding to the target device. Then, the fault information can be sent to the target object to make the target object adjust or optimize the target device. The target object can be a person who maintains the target device, or a person in charge of a project to which the target device belongs. For example, in the target clustering result, according to the number of log texts contained in each type of log text, it is determined that the type of log text containing the most log texts is the log text related to the database, and the log text with a higher frequency in this type of log text is an input error (i.e., fault information) caused by the input data not meeting the preset field length of the database. The input error is sent to the worker who maintains the database and the relevant person in charge. After the person in charge determines the new field length of the database, the worker adjusts the database accordingly.

[0066] By determining the fault information of the target device through the target clustering result, the target device can be adjusted and optimized according to the fault information, thereby improving the running stability of the target device and further improving the work efficiency of the target device.

[0067] Optionally, in the first embodiment, the process of adjusting the target device according to the target clustering result can be as shown in Figure 2 First, the log texts of the target device can be obtained from the logs of the target device and preprocessed to obtain a set of log texts. The elbow rule is used to calculate part of the log texts in the set of log texts to obtain a first K value, and then a set of candidate K values is determined according to the first K value. Then, the density of each log text in the set of log texts is determined according to the cosine distance between the log texts, the first centroid of each log text is determined according to the density of each log text, and the remaining centroids corresponding to a candidate K value are determined according to the cosine distance between the log texts in the set of log texts. The centroids corresponding to all candidate K values are integrated to obtain the centroid corresponding to each candidate K value. Second, multiple K-Means clustering is performed according to the centroid corresponding to each candidate K value to obtain multiple clustering results. Then, the cosine distance between the cluster centers of each cluster corresponding to each clustering result is calculated to obtain a first distance corresponding to the clustering result. The cosine distance between the cluster center of each cluster corresponding to each clustering result and the log texts in the cluster is calculated to obtain a second distance corresponding to the clustering result. The clustering score corresponding to each clustering result is determined according to the first distance and the second distance, and the clustering result with the largest clustering score is determined as the target clustering result. Finally, the target object adjusts or optimizes the target device according to the target clustering result.

[0068] It is noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] Embodiment Two

[0070] Embodiment Two of the present application also provides a log text clustering apparatus. It is noted that the log text clustering apparatus of Embodiment Two of the present application can be used to execute the log text clustering method provided in Embodiment One of the present application. The log text clustering apparatus provided in Embodiment Two of the present application is introduced as follows.

[0071] Figure 3 is a schematic diagram of the log text clustering apparatus according to Embodiment Two of the present application. As shown in the figure, the apparatus includes an acquisition unit 301, a clustering unit 302, a first determination unit 303, and a second determination unit 304. Figure 3

[0072] Specifically, the acquisition unit 301 is configured to acquire a candidate K value set, wherein the candidate K value set includes N candidate K values, the candidate K value is a preset number of centroids, N is a positive integer, and K is an integer greater than 1.

[0073] The clustering unit 302 is configured to cluster log texts in the log text set according to the N candidate K values to obtain N clustering results.

[0074] The first determination unit 303 is configured to determine a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set.

[0075] The second determination unit 304 is configured to determine a target clustering result of the log text set according to the clustering result corresponding to the target K value.

[0076] ​The clustering device for log texts provided in Embodiment Two of the present application comprises: an acquisition unit 301 configured to acquire a candidate K value set, wherein the candidate K value set comprises N candidate K values, the candidate K value is a preset number of centroids, N is a positive integer, and K is an integer greater than 1; a clustering unit 302 configured to cluster log texts in a log text set according to the N candidate K values to obtain N clustering results; a first determination unit 303 configured to determine a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set; and a second determination unit 304 configured to determine a target clustering result of the log text set according to the clustering result corresponding to the target K value. The clustering device for log texts provided in Embodiment Two of the present application solves the problem that, in the related art, when a preset value determined by human experience is used as the number of centroids for K-Means clustering of log texts, the preset value determined by human experience is subjective and blind, resulting in a relatively low accuracy of the clustering result of the log texts. By clustering each candidate K value in the candidate K value set to obtain multiple clustering results and determining a target clustering result from the multiple clustering results according to the cosine distance between log texts in the log text set, the number of centroids used by the K-Means clustering algorithm can be adaptively adjusted according to the log texts in the log text set, thereby avoiding the problem that the clustering result is inaccurate due to the subjectivity and blindness of human experience when the number of centroids is set, achieving the effect of improving the clustering quality of the clustering result of the K-Means clustering algorithm, and further achieving the effect of improving the clustering quality of the clustering result of the log text set.

[0077] Optionally, in the clustering device for log texts provided in Embodiment Two of the present application, the acquisition unit 301 comprises: a first determination subunit configured to randomly select a preset number of log texts in the log text set to obtain a first data set; a first clustering subunit configured to cluster the log texts in the first data set multiple times to obtain multiple initial K values, and determine a first K value according to the multiple initial K values; a second determination subunit configured to determine a value selection range with the first K value as the center, and determine N candidate K values according to the value selection range; and a generation subunit configured to generate the candidate K value set based on the N candidate K values.

[0078] Optionally, in the clustering device for log texts provided in Embodiment Two of the present application, the clustering unit 302 comprises: a third determination subunit configured to determine, for each candidate K value, K centroids corresponding to the candidate K value in the log text set; a second clustering subunit configured to cluster the log texts in the log text set according to the K centroids to obtain a clustering result corresponding to the candidate K value; and a fourth determination subunit configured to obtain N clustering results by synthesizing the clustering results corresponding to all candidate K values.

[0079] Optionally, in the log text clustering apparatus provided in Embodiment Two of the present application, the third determining subunit comprises: a first calculating module, configured to calculate the cosine distances between the log texts in the log text set to obtain a cosine distance set corresponding to each log text; a second calculating module, configured to calculate the number of cosine distances less than the preset threshold in the cosine distance set corresponding to each log text to obtain the density of each log text; and a determining module, configured to determine the K centroids corresponding to the candidate K value in the log text set according to the density of each log text.

[0080] Optionally, in the log text clustering apparatus provided in Embodiment Two of the present application, the determining module comprises: a sorting sub-module, configured to sort the log texts in the log text set according to the density of each log text, and determine the first log text in the first order as the first centroid; a first determining sub-module, configured to determine the second log text according to the cosine distance between the first log text and the log texts in the log text set except the first log text, and determine the second log text as the second centroid; a second determining sub-module, configured to determine the target cosine distance of each log text according to the cosine distance between the log texts in the log text set and the first centroid and the cosine distance between the log texts in the log text set and the second centroid, and determine the next centroid according to the target cosine distance of each log text; and a processing sub-module, configured to repeatedly perform the steps of determining the target cosine distance of each log text according to the cosine distance between the log texts in the log text set and the determined centroid, and determining the next centroid according to the target cosine distance of each log text, until the Kth centroid is determined.

[0081] Optionally, in the log text clustering apparatus provided in Embodiment Two of the present application, the first determining unit 303 comprises: a fifth determining subunit, configured to determine the K clusters corresponding to each clustering result according to the N clustering results, wherein each cluster comprises a plurality of log texts; a first calculating subunit, configured to calculate the cosine distances between the cluster centers of each cluster corresponding to each clustering result to obtain the first distance corresponding to the clustering result; a second calculating subunit, configured to calculate the cosine distances between the cluster center of each cluster corresponding to each clustering result and the log texts in the cluster to obtain the second distance corresponding to the clustering result; a third calculating subunit, configured to calculate according to the first distance and the second distance to obtain the clustering score corresponding to each clustering result; and a sixth determining subunit, configured to determine the target K value from the candidate K value set and the clustering result corresponding to the target K value from the candidate K value set according to the clustering score corresponding to each clustering result.

[0082] Optionally, in the log text clustering apparatus provided in Embodiment 2 of this application, the apparatus further includes: a third determining unit, configured to determine the fault information of the target device based on the target clustering result of the log text set after determining the target clustering result based on the clustering result corresponding to the target K value, wherein the target device is the device corresponding to the log text set; and a sending unit, configured to send the fault information to the target object, wherein the target object optimizes the target device based on the fault information.

[0083] The log text clustering device includes a processor and a memory. The aforementioned acquisition unit 301, clustering unit 302, first determination unit 303, and second determination unit 304 are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to achieve the corresponding functions.

[0084] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the clustering quality of the results.

[0085] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0086] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a clustering method for log text.

[0087] Embodiment 4 of the present invention provides a processor for running a program, wherein the program executes a clustering method for log text during runtime.

[0088] like Figure 4 As shown, Embodiment 5 of the present invention provides an electronic device. The device includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: obtaining a set of candidate K values, wherein the set of candidate K values ​​includes N candidate K values, the candidate K values ​​are a preset number of centroids, N is a positive integer, and K is an integer greater than 1; clustering the log texts in the log text set according to the N candidate K values ​​to obtain N clustering results; determining the target K value and the clustering result corresponding to the target K value according to the N clustering results and the set of candidate K values; and determining the target clustering result of the log text set according to the clustering result corresponding to the target K value.

[0089] The processor further implements the following steps when executing the program: obtaining the candidate K value set comprises: randomly selecting a preset number of log texts in the log text set to obtain a first data set; clustering the log texts in the first data set multiple times to obtain multiple initial K values, and determining the first K value according to the multiple initial K values; determining a numerical value selection range with the first K value as the center, and determining N candidate K values according to the numerical value selection range; and generating the candidate K value set based on the N candidate K values.

[0090] The processor further implements the following steps when executing the program: clustering the log texts in the log text set according to the N candidate K values to obtain N clustering results comprises: for each candidate K value, determining K centroids corresponding to the candidate K value in the log text set; clustering the log texts in the log text set according to the K centroids to obtain a clustering result corresponding to the candidate K value; and synthesizing the clustering results corresponding to all candidate K values to obtain the N clustering results.

[0091] The processor further implements the following steps when executing the program: for each candidate K value, determining K centroids corresponding to the candidate K value in the log text set comprises: calculating the cosine distance between the log texts in the log text set to obtain a cosine distance set corresponding to each log text; calculating the number of cosine distances less than a preset threshold in the cosine distance set corresponding to each log text to obtain the density of each log text; and determining the K centroids corresponding to the candidate K value in the log text set according to the density of each log text.

[0092] The processor further implements the following steps when executing the program: determining the K centroids corresponding to the candidate K value in the log text set according to the density of each log text comprises: sorting the log texts in the log text set according to the density of each log text, and determining a first log text in the first order as a first centroid; determining a second log text according to the cosine distance between the log texts in the log text set except the first log text and the first log text, and determining the second log text as a second centroid; determining the target cosine distance of each log text according to the cosine distance between the log texts in the log text set and the first centroid and the cosine distance between the log texts in the log text set and the second centroid, and determining a next centroid according to the target cosine distance of each log text; and repeatedly performing the steps of determining the target cosine distance of each log text according to the cosine distance between the log texts in the log text set and the determined centroids, and determining a next centroid according to the target cosine distance of each log text, until the Kth centroid is determined.

[0093] The processor further implements the following steps when executing the program: determining the target K value and the clustering result corresponding to the target K value according to the N clustering results and the candidate K value set, including: determining K clusters corresponding to each clustering result according to the N clustering results, wherein each cluster includes a plurality of log texts; for each clustering result, calculating the cosine distance between the cluster centers of each cluster corresponding to the clustering result to obtain a first distance corresponding to the clustering result; for each clustering result, calculating the cosine distance between the cluster center of each cluster corresponding to the clustering result and the log text in the cluster to obtain a second distance corresponding to the clustering result; calculating according to the first distance and the second distance to obtain a clustering score corresponding to each clustering result; determining the target K value from the candidate K value set and the clustering result corresponding to the target K value from the candidate K value set according to the clustering score corresponding to each clustering result.

[0094] The processor further implements the following steps when executing the program: after determining the target clustering result of the log text set according to the clustering result corresponding to the target K value, the above method further includes: determining the fault information of the target device according to the target clustering result, wherein the target device is a device corresponding to the log text set; and sending the fault information to the target object, wherein the target object optimizes the target device according to the fault information.

[0095] The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0096] The application also provides a computer program product adapted to execute the program initializing the following method steps when executed on a data processing device: obtaining a candidate K value set, wherein the candidate K value set includes N candidate K values, the candidate K value is a preset number of centroids, N is a positive integer, and K is an integer greater than 1; clustering log texts in a log text set according to the N candidate K values to obtain N clustering results; determining a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set; and determining a target clustering result of the log text set according to the clustering result corresponding to the target K value.

[0097] When executed on a data processing device, the program is also adapted to initialize the following method steps: obtaining a candidate K value set includes: randomly selecting a preset number of log texts in a log text set to obtain a first data set; clustering the log texts in the first data set multiple times to obtain multiple initial K values, and determining a first K value according to the multiple initial K values; determining a numerical selection range with the first K value as the center, and determining N candidate K values according to the numerical selection range; and generating a candidate K value set based on the N candidate K values.

[0098] When executed on the data processing device, the program is further adapted to execute the steps of the method by: clustering the log texts in the log text set according to the N candidate K values to obtain N clustering results, comprising: for each candidate K value, determining K centroids corresponding to the candidate K value in the log text set; clustering the log texts in the log text set according to the K centroids to obtain a clustering result corresponding to the candidate K value; and synthesizing the clustering results corresponding to all candidate K values to obtain the N clustering results.

[0099] When executed on the data processing device, the program is further adapted to execute the steps of the method by: for each candidate K value, determining K centroids corresponding to the candidate K value in the log text set comprises: calculating cosine distances between the log texts in the log text set to obtain a cosine distance set corresponding to each log text; calculating the number of cosine distances less than a preset threshold in the cosine distance set corresponding to each log text to obtain a density of each log text; and determining the K centroids corresponding to the candidate K value in the log text set according to the density of each log text.

[0100] When executed on the data processing device, the program is further adapted to execute the steps of the method by: determining the K centroids corresponding to the candidate K value in the log text set according to the density of each log text comprises: sorting the log texts in the log text set according to the density of each log text, and determining a first log text in the first order as a first centroid; determining a second log text according to the cosine distance between the log texts in the log text set except the first log text and the first log text, and determining the second log text as a second centroid; determining a target cosine distance of each log text according to the cosine distance between the log texts in the log text set and the first centroid and the cosine distance between the log texts in the log text set and the second centroid, and determining a next centroid according to the target cosine distance of each log text; and repeating the steps of determining the target cosine distance of each log text according to the cosine distance between the log texts in the log text set and the determined centroids, and determining a next centroid according to the target cosine distance of each log text, until the Kth centroid is determined.

[0101] When executed on the data processing device, the program is further adapted to perform the following method steps: determining the target K value and the clustering result corresponding to the target K value from the N clustering results and the candidate K value set comprises: determining K clusters corresponding to each clustering result according to the N clustering results, wherein each cluster comprises a plurality of log texts; for each clustering result, calculating cosine distances between cluster centers of each cluster corresponding to the clustering result to obtain first distances corresponding to the clustering result; for each clustering result, calculating cosine distances between cluster centers of each cluster corresponding to the clustering result and log texts in the cluster to obtain second distances corresponding to the clustering result; calculating according to the first distances and the second distances to obtain a clustering score corresponding to each clustering result; determining the target K value from the candidate K value set and the clustering result corresponding to the target K value from the candidate K value set according to the clustering score corresponding to each clustering result.

[0102] When executed on the data processing device, the program is further adapted to perform the following method steps: after determining the target clustering result of the log text set according to the clustering result corresponding to the target K value, the method further comprises: determining fault information of the target device according to the target clustering result, wherein the target device is a device corresponding to the log text set; and sending the fault information to the target object, wherein the target object optimizes the target device according to the fault information.

[0103] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0105] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0107] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0108] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or a combination of non-volatile memories in different forms. The memory is an example of computer readable storage media.

[0109] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0110] It is also to be noted that the terms "comprising", "including", and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0111] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0112] The embodiments of the present application are only illustrative and are not intended to limit the present application. Various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method of clustering log texts, characterized by, The method comprises the following steps: obtaining a candidate K value set, wherein the candidate K value set comprises N candidate K values, the candidate K value is a preset number of centroids, N is a positive integer, and K is an integer greater than 1; clustering log texts in a log text set according to the N candidate K values to obtain N clustering results; determining a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set; determining a target clustering result of the log text set according to the clustering result corresponding to the target K value; wherein the cosine distances between the log texts in the log text set are calculated to obtain a cosine distance set corresponding to each log text; the number of cosine distances less than a preset threshold in the cosine distance set corresponding to each log text is calculated to obtain the density of each log text; the log texts in the log text set are sorted according to the density of each log text, and a first log text in the first order is determined as a first centroid; a second log text is determined according to the cosine distance between the first log text and the log texts other than the first log text in the log text set, and the second log text is determined as a second centroid; the target cosine distance of each log text is determined according to the cosine distance between the log texts in the log text set and the first centroid and the cosine distance between the log texts in the log text set and the second centroid, and the next centroid is determined according to the target cosine distance of each log text; the step of determining the target cosine distance of each log text according to the cosine distance between the log texts in the log text set and the determined centroid and determining the next centroid according to the target cosine distance of each log text is repeatedly performed until the Kth centroid is determined, wherein the K centroids are used to obtain the N clustering results.

2. The method of claim 1, wherein, The candidate K value set is obtained by: randomly selecting a preset number of log texts in the log text set to obtain a first data set; clustering the log texts in the first data set multiple times to obtain multiple initial K values, and determining a first K value according to the multiple initial K values; determining a numerical selection range with the first K value as the center, and determining N candidate K values according to the numerical selection range; generating the candidate K value set based on the N candidate K values.

3. The method of claim 1, wherein, The N clustering results are obtained by clustering the log texts in the log text set according to the N candidate K values, which comprises: for each candidate K value, determining K centroids corresponding to the candidate K value in the log text set; clustering the log texts in the log text set according to the K centroids to obtain a clustering result corresponding to the candidate K value; comprehensively obtaining the N clustering results from the clustering results corresponding to all candidate K values.

4. The method of claim 1, wherein, The target K value and the clustering result corresponding to the target K value are determined according to the N clustering results and the candidate K value set, which comprises: determining K clusters corresponding to each clustering result according to the N clustering results, wherein each cluster comprises multiple log texts; For each clustering result, a cosine distance between cluster centers of each cluster corresponding to the clustering result is calculated to obtain a first distance corresponding to the clustering result; For each clustering result, a cosine distance between a cluster center of each cluster corresponding to the clustering result and a log text in the cluster is calculated to obtain a second distance corresponding to the clustering result; The first distance and the second distance are calculated to obtain a clustering score corresponding to each clustering result; According to the clustering score corresponding to each clustering result, a target K value is determined from the candidate K value set, and a clustering result corresponding to the target K value is determined from the candidate K value set.

5. The method of claim 1, wherein, After determining the target clustering result of the log text set according to the clustering result corresponding to the target K value, the method further comprises: According to the target clustering result, the fault information of the target device is determined, wherein the target device is a device corresponding to the log text set; The fault information is sent to a target object, wherein the target object optimizes the target device according to the fault information.

6. An apparatus for clustering log texts, characterized by Comprise: An acquisition unit is configured to acquire a candidate K value set, wherein the candidate K value set includes N candidate K values, the candidate K value is a preset centroid number, N is a positive integer, and K is an integer greater than 1; A clustering unit is configured to cluster log texts in a log text set according to N candidate K values to obtain N clustering results; A first determination unit is configured to determine a target K value and a clustering result corresponding to the target K value according to the N clustering results and the candidate K value set; A second determination unit is configured to determine a target clustering result of the log text set according to the clustering result corresponding to the target K value; The device is further configured to calculate cosine distances between log texts in the log text set to obtain a cosine distance set corresponding to each log text; The number of cosine distances less than a preset threshold in the cosine distance set corresponding to each log text is calculated to obtain a density of each log text; The log texts in the log text set are sorted according to the density of each log text, and a first log text in the first order is determined as a first centroid; A second log text is determined according to cosine distances between the first log text and log texts in the log text set other than the first log text, and the second log text is determined as a second centroid; A target cosine distance of each log text is determined according to cosine distances between the first centroid and log texts in the log text set and cosine distances between the second centroid and log texts in the log text set, and a next centroid is determined according to the target cosine distance of each log text; The steps of determining a target cosine distance of each log text according to cosine distances between log texts in the log text set and determined centroids and determining a next centroid according to the target cosine distance of each log text are repeatedly performed until a Kth centroid is determined, wherein K centroids are used to obtain the N clustering results.

7. A processor, comprising: The processor is configured to run a program, wherein the program, when executed, implements the log text clustering method of any one of claims 1-5.

8. An electronic device, comprising: One or more processors and memory storing one or more programs for execution by the one or more processors, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the log text clustering method of any one of claims 1-5.

Images