Kmeans log classification method based on adaptive radius center selection

Abstract

The application provides a kmeans log classification method based on adaptive radius center selection, and relates to the technical field of big data. The method comprises the following steps: fusing log samples until the number of log samples in a target sample set meets a first preset number condition, and constructing a first distance matrix according to the cosine distance of each log sample; storing a first center point obtained according to the first distance matrix into a center set, if a preset condition is triggered, continuing to fuse and update the target sample set until the number of log samples meets a second preset number condition, constructing a second distance matrix according to the fused and updated target sample set, continuing to obtain a second center point from the fused and updated target sample set according to the second distance matrix, and storing the second center point into the center set until the center point in the center set is K for Kmeans clustering. The scheme improves the quality of initial centroid selection, thereby improving the Kmeans clustering effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a kmeans log classification method based on adaptive radius center selection. BACKGROUND

[0002] Log clustering aims to find similar logs. Application programs usually have error logs during use, which can be divided into different types. By analyzing these different types of error logs, targeted suggestions for solving the errors can be given.

[0003] At present, the existing technology mainly uses the Kmeans clustering algorithm. In the initialization stage, the initial centroid is randomly extracted, and then the distance between each object and the initial centroid is calculated. Each is assigned to the nearest cluster center.

[0004] However, this random selection of initial centroid can lead to poor clustering results. SUMMARY

[0005] The present application provides a kmeans log classification method based on adaptive radius center selection to solve the problem of poor clustering results caused by random selection of initial centroid of Kmeans clustering algorithm.

[0006] In a first aspect, the present application provides a kmeans log classification method based on adaptive radius center selection, comprising:

[0007] Obtain a target sample set, fuse the log samples in the target sample set until the number of log samples in the target sample set meets a first preset number condition, and the target sample set includes at least one log sample;

[0008] Determine the cosine distance between each log sample in the target sample set, and construct a first distance matrix according to the cosine distance;

[0009] According to the first distance matrix, obtain a first center point and store it in a center set, wherein the first center point is the center of two first target log samples in the target sample set that meet a first preset requirement;

[0010] If a preset condition is triggered, the first target log sample is deleted and the target sample set after deleting the first target log sample is fused and updated until the number of log samples meets a second preset number condition;

[0011] According to the cosine distance between each log sample in the fused and updated target sample set, a second distance matrix is constructed;

[0012] According to the second distance matrix, a second center point is continuously obtained from the target sample set after fusion update and stored in the center set until the center points in the center set are K, the second center point is the center of two second target log samples in the target sample set after fusion update and meeting a second preset requirement, and K is a positive integer.

[0013] The center points in the center set are obtained as initial mass points for Kmeans clustering.

[0014] In a second aspect, the application provides a kmeans log classification device based on adaptive radius center selection, comprising:

[0015] An acquisition module is configured to acquire a target sample set, and fuse log samples in the target sample set until the number of log samples in the target sample set meets a first preset number condition, wherein the target sample set comprises at least one log sample.

[0016] A first matrix construction module is configured to determine cosine distances between the log samples in the target sample set, and construct a first distance matrix according to the cosine distances.

[0017] A first center point storage module is configured to acquire a first center point according to the first distance matrix, and store the first center point in a center set, wherein the first center point is the center of two first target log samples in the target sample set and meeting a first preset requirement.

[0018] A sample set update module is configured to delete the first target log sample and fuse and update the target sample set after deletion of the first target log sample if a preset condition is triggered until the number of log samples meets a second preset number condition.

[0019] A second matrix construction module is configured to construct a second distance matrix according to cosine distances between the log samples in the target sample set after fusion update.

[0020] A second center point storage module is configured to continuously acquire a second center point from the target sample set after fusion update according to the second distance matrix, and store the second center point in the center set until the center points in the center set are K, wherein the second center point is the center of two second target log samples in the target sample set after fusion update and meeting a second preset requirement, and K is a positive integer.

[0021] A clustering module is configured to acquire the center points in the center set as initial mass points for Kmeans clustering.

[0022] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory connected with the processor in communication; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method as described above.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method as described above.

[0024] In a fifth aspect, the present application provides a computer program product, wherein the computer program product is executed by a processor to implement the method as described above.

[0025] The kmeans log classification method based on adaptive radius center selection provided by the present application can obtain the cosine distance between each log sample in the target sample set, then construct a distance matrix, and based on the distance matrix, the first center point and the second center point can be stored in the center set as the initial centroid when Kmeans clustering, which can improve the selection quality of the initial centroid, and further improve the Kmeans clustering effect. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate one embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0027] Figure 1 A flowchart of the Kmeans clustering provided by the embodiment of the present application is provided.

[0028] Figure 2 A flowchart of the kmeans log classification method based on adaptive radius center selection provided by the embodiment of the present application is provided.

[0029] Figure 3 A flowchart of the kmeans log classification method based on adaptive radius center selection provided by the embodiment of the present application is provided.

[0030] Figure 4 A structural schematic diagram of the kmeans log classification device based on adaptive radius center selection provided by the embodiment of the present application is provided.

[0031] Figure 5 A structural schematic diagram of the electronic device provided by the embodiment of the present application is provided.

[0032] The specific embodiments of the application have been shown and described in considerable detail in order to explain clearly how to make and use the application. The above examples are illustrative only of the application and are not intended to limit the scope of the application in any way. Rather, reference should be made to the appended claims, which fully indicate the scope of the application, wherein: DETAILED DESCRIPTION

[0033] The exemplary embodiments will be described in relation to the drawings. When the description is made in relation to the drawings, identical numbers on different drawings represent the same or similar elements unless otherwise stated. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the application as detailed in the appended claims.

[0034] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0035] It should be noted that the kmeans log classification method based on adaptive radius center selection provided by the present application can be used in the field of big data technology, and can also be used in any field other than the field of big data technology, and the application field of the kmeans log classification method based on adaptive radius center selection provided by the present application is not limited.

[0036] K-means clustering algorithm: a clustering algorithm that can be used for text clustering, which has a step in the initialization stage that divides the data into K groups, then randomly selects an object in each group as the initial cluster center of the group, then calculates the distance between each object and each seed cluster center, and assigns each object to the cluster center closest to it.

[0037] The Kmeans clustering algorithm has the following disadvantages: (1) the selection of the centroid in the initialization stage is random, and the initial centroid selection is too close or scattered, which is not uniform enough, which will lead to slow convergence speed of the algorithm. (2) The initial centroid selection to the discrete point or noise data is not conducive to the convergence of subsequent clustering, resulting in unsatisfactory clustering effect of the algorithm. (3) The larger the data, the more obvious the defect, and the initialization time is very long.

[0038] To solve the above problems, the application provides a kmeans log classification method based on adaptive radius center selection. The initialization stage of the kmeans algorithm is improved. In the initialization process, the distance matrix of the sample set is obtained, and the initial centroid of the initial stage of the kmeans algorithm is selected based on the distance matrix, so as to accelerate the initialization convergence speed and the quality of the initial centroid selection. The quality of clustering is better, and the log data classification is more accurate.

[0039] The technical solutions of the application and how the technical solutions solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described again in some examples. The embodiments of the application will be described below with reference to the drawings.

[0040] Exemplarily, Figure 1 The flowchart of the Kmeans clustering provided by the embodiments of the application is shown in FIG. 1. As shown in FIG. 1, 20 sample logs B1 to B20 are taken as examples. The 20 sample logs can be divided into two groups. Two initial centroids B1 and B10 are selected, and then other log samples are clustered into two clusters based on the two initial centroids B1 and B10. Figure 1 The log can be a computer term. The application platform will generate logs during operation. Each row of the log records the date, time, user, and action related to the operation.

[0041] Figure 2 The flowchart of the kmeans log classification method based on adaptive radius center selection provided by the embodiments of the application is shown in FIG. 2. The method can be applied to a text clustering scene. The input mass of historical error logs is clustered, and the error logs are divided into multiple groups (i.e., different types of error logs are clustered into different groups). After the initial centroid is obtained by using the algorithm on the input mass of log data, the K-means clustering algorithm is used for clustering, and the K groups of logs with good clustering effect can be quickly obtained. By analyzing and arranging the error logs of each group, the error habits of the user during the use of the application program can be found, so as to give the corresponding solution. Figure 2 As shown in FIG. 2, the method can include the following steps:

[0042] In step S201, a target sample set is obtained. The log samples in the target sample set are fused until the number of log samples in the target sample set meets the first preset number condition. The target sample set includes at least one log sample.

[0043] In the embodiment, the log samples can be error logs generated by the application mentioned above, each error log corresponds to a type, so that the massive error logs need to be grouped by type, and error logs of the same type are classified into the same group. The target sample set can be composed of several error logs filtered from the massive error logs.

[0044] The log samples can be converted into coordinate points in the form of vectorization, for example, the log sample B1 is converted into the coordinate point (Xb1, Yb1). By converting into the coordinate point form, the distance between the log samples can be more intuitively seen, and based on the distance, it can be judged whether the log samples have similarity.

[0045] In the embodiment, two log samples can be merged into one log sample, for example, the center points of the two log samples are selected as a new log sample, so that the number of log samples in the target sample set can be reduced by 1, that is, the number of log samples is reduced by 1 each time the fusion is performed, so that the number of log samples in the target sample set can meet the first preset number condition through multiple fusions. The number of log samples in the target sample set is usually greater than the first preset number (the number can be configured according to the actual situation) when no fusion is performed.

[0046] In step S202, the cosine distances between the log samples in the target sample set are determined, and a first distance matrix is constructed according to the cosine distances.

[0047] In the embodiment, after the log samples are converted into vector form, the cosine distances between the log samples in the log sample set can be calculated, that is, there is a corresponding cosine distance between each two log samples, and then the cosine distances are counted to construct the first distance matrix.

[0048] The cosine distance, also known as cosine similarity, is the cosine value of the angle between two vectors in a vector space as a measure of the difference between the two individuals. It can be understood that there are multiple cosine distances in the first distance matrix, for example, the cosine distance at the first row and the first column of the first distance matrix is the maximum value (i.e. the maximum cosine distance) in the entire first distance matrix. For example, the cosine distance at the last row and the last column of the first distance matrix is the minimum value (i.e. the minimum cosine distance) in the entire first distance matrix.

[0049] In step S203, a first center point is obtained according to the first distance matrix and stored in a center set, and the first center point is the center of two first target log samples in the target sample set that meet the first preset requirement.

[0050] In this embodiment, the log samples satisfying the first preset requirement can be screened out based on the first distance matrix as the first target log samples. For example, the first preset requirement can be that the cosine distance is greater than a first preset threshold (which can be configured according to actual conditions), that is, the cosine distance greater than the first preset threshold is selected from the first distance matrix, and then the log sample corresponding to the cosine distance is found as the first target log sample.

[0051] In step S204, if the preset condition is triggered, the first target log sample is deleted and the target sample set after deleting the first target log sample is fused and updated until the number of log samples satisfies the second preset number condition.

[0052] In this embodiment, the preset condition can be that the first center point cannot be found, that is, two first target log samples satisfying the first preset requirement cannot be found, so that the target sample set can be updated, the first target log sample is deleted, and then the log samples in the target sample set are fused again. The first preset number condition and the second preset number condition are different.

[0053] In this embodiment, when the log samples in the updated target sample set are fused, the two log samples with the closest cosine distance can be selected, and the centers of the two log samples are taken as a new log sample, so that the fusion is realized, and the number of log samples in the updated target sample set is also reduced by 1. After multiple fusions, the number of log samples in the updated target sample set finally satisfies the second preset number condition.

[0054] The second preset number condition can be configured according to actual conditions, and generally will not exceed the number of log samples in the updated target sample set.

[0055] In step S205, a second distance matrix is constructed according to the cosine distances between the log samples in the fused and updated target sample set.

[0056] In this embodiment, after the log samples are converted into vector form, the cosine distances between the log samples in the updated log sample set can be calculated, that is, there is a corresponding cosine distance between each two log samples, and then the cosine distances are counted to construct the second distance matrix.

[0057] In step S206, a second center point is obtained from the fused and updated target sample set according to the second distance matrix and stored in the center set until the center point in the center set is K, the second center point is the center of two second target log samples satisfying the second preset requirement in the fused and updated target sample set, and K is a positive integer.

[0058] In the embodiment, K is the number of initial centroids required when performing Kmeans clustering, which can be numerically adjusted according to the needs of the Kmeans clustering algorithm.

[0059] The second distance matrix can be used to screen log samples that meet the second preset requirement as the second target log sample. For example, the second preset requirement can be that the cosine distance is greater than a preset threshold (which can be configured according to actual conditions), that is, the cosine distance greater than the preset threshold is selected from the second distance matrix, and then the log sample corresponding to the cosine distance is found as the second target log sample.

[0060] In step S207, the center points in the center set are obtained as initial particles to perform Kmeans clustering.

[0061] In the embodiment, the K-means clustering algorithm has been introduced above, which randomly selects K objects as initial cluster centers at the beginning. In the embodiment, the selected cluster centers (i.e., initial centroids) are used to replace the step of randomly selecting K objects in the Kmeans clustering algorithm, which can improve the convergence speed of the algorithm and improve the clustering effect.

[0062] In the embodiment, the cosine distances between the log samples in the target sample set are obtained, and then a distance matrix is constructed. The first center point and the second center point are stored in the center set as initial centroids when performing Kmeans clustering, which can improve the selection quality of the initial centroids and further improve the Kmeans clustering effect.

[0063] In some embodiments, the step of "obtaining a target sample set" in step S201 can be implemented by the following steps: randomly sampling the initial log sample set according to a preset sampling ratio to obtain the sampled log samples; and combining the sampled log samples into a target sample set.

[0064] In the embodiment, the preset sampling ratio can be configured, for example, 15%, and then the massive error logs generated by the application are combined to form an initial log sample set. Then, 15% of the initial log sample set is randomly sampled to form a set dataset as the target sample set.

[0065] For example, in other embodiments, the preset sampling ratio can also be configured according to the number of error logs in the initial log sample set. Generally, the more error logs in the initial log sample set, the smaller the preset sampling ratio configured, and the smaller the error logs in the initial log sample set, the larger the preset sampling ratio configured.

[0066] The embodiment of the application can effectively reduce the order of magnitude of the log samples, reduce the number of subsequent log sample fusion of the target sample set, and thus improve the efficiency of initial centroid selection.

[0067] In some embodiments, the step S201 of "fusing the log samples in the target sample set" can be implemented by the following steps: obtaining the cosine distances of the log samples in the target sample set, deleting the two log samples with the smallest cosine distances; obtaining the center of the deleted two log samples, and restoring the center to the target sample set.

[0068] In the embodiment, the first preset number condition can be configured as 10*K (K is the number of initial centroids required for Kmeans clustering). When fusing the log samples, let dataset represent the target sample set, the log samples in dataset can be calculated two by two, the two log samples with the smallest cosine distances are taken out and deleted, and then the center of the two log samples is calculated and put back into dataset. According to this method, the two log samples with the smallest cosine distances are fused until the number of samples in dataset reaches 10*K.

[0069] The embodiment of the application can reduce the number of log samples in the target sample set by fusing the log samples in the target sample set, facilitate subsequent selection of initial centroids, and at the same time, by deleting the two log samples with the smallest pre-distance and retaining the center of the two log samples as a new fused log sample, the quality of subsequent selection of initial centroids can be ensured.

[0070] In some embodiments, the step S203 of "obtaining the first center point according to the first distance matrix" can be implemented by the following steps: obtaining the two first target log samples corresponding to the smallest cosine distance from the first distance matrix; if the smallest cosine distance is smaller than the first preset threshold, the center of the two first target log samples is taken as the first center point.

[0071] In the embodiment, the cosine distances between the log samples in dataset are calculated two by two to obtain the first distance matrix D, and then the two log samples with the smallest cosine distance are found from the first distance matrix D. If the smallest cosine distance is smaller than the first preset threshold T1, the value at the position in the first distance matrix D is set to infinity, the center of the two log samples is recorded as the first center point C1, and the first center point C1 is added to the center set Cens.

[0072] Setting the value at that position in the first distance matrix D to infinity is equivalent to updating the minimum cosine distance. That is, the original minimum cosine distance becomes infinity, and the original second smallest cosine distance becomes the minimum cosine distance in the updated first distance matrix D.

[0073] In this embodiment, two log samples with the minimum cosine distance are selected as the first target log samples. When the minimum cosine distance is less than a first preset threshold, the center of these two first target log samples can be selected as an initial centroid, thereby improving the quality of the initial centroid selection in the initialization stage of the Kmeans clustering algorithm and thus improving the clustering effect of the Kmeans clustering algorithm.

[0074] Furthermore, based on the above embodiments, in some other embodiments, the first center point can also be obtained through the following steps: obtaining the maximum cosine distance in the first distance matrix, and calculating the first adaptive radius based on the maximum cosine distance; replacing the minimum cosine distance in the first distance matrix with infinity to obtain an updated first distance matrix; continuing to obtain the updated minimum cosine distance from the updated first distance matrix; if the updated minimum cosine distance is less than a first preset threshold, then continuing to obtain the two first target log samples corresponding to the updated minimum cosine distance; if the distance between the center of the two first target log samples corresponding to the updated minimum cosine distance and the center point in the center set is less than the first adaptive radius, then the center is taken as the first center point.

[0075] In this embodiment, for example, the maximum cosine distance can be found from the first distance matrix D, denoted as distmax, and the first adaptive radius R1 can be calculated, R1 = distmax / k, where k is the number of initial centroids required by the Kmeans clustering algorithm.

[0076] When the minimum cosine distance in the first distance matrix is ​​replaced with infinity, the second smallest cosine distance in the updated first distance matrix becomes the minimum cosine distance in the updated first distance matrix. At this time, the log samples corresponding to the minimum cosine distance in the updated first distance matrix are the two new first target log samples. If the distance between the center of these two new first target log samples and the center point in the center set (e.g., the first center point added to the center set in the above embodiment) is less than the first adaptive radius R1, then the center of these two new first target log samples can also be used as the first center point.

[0077] For example, the minimum cosine distance of the first distance matrix D is Z1, the second smallest cosine distance is Z2, the two first target log samples corresponding to the minimum cosine distance Z1 are Z11 and Z12, and the two first target log samples corresponding to the second smallest cosine distance Z2 are Z21 and Z22. Then the center of Z11 and Z12 can be used as the first center point. At the same time, the minimum cosine distance Z1 is set to infinity, and the minimum cosine distance in the updated first distance matrix becomes Z2. At this time, if the distance between the center of Z21 and Z22 and the first center point (i.e. the center of Z11 and Z12) is less than the first adaptive radius R1, then the center of Z21 and Z22 can also be used as the first center point and stored in the center set (the center set can include multiple first center points).

[0078] This application embodiment improves the quality of initial centroid selection by incorporating the adaptive radius into a first preset requirement as a criterion for selecting the first centroid, accelerates the initial convergence speed of the Kmeans clustering algorithm, further improves the clustering effect of the Kmeans clustering algorithm, and makes the error log classification more accurate.

[0079] In some embodiments, the preset condition may be that the minimum cosine distance in the first distance matrix is ​​greater than or equal to a preset first threshold.

[0080] In this embodiment, the first distance matrix includes several cosine distances, each of which is the cosine distance between two log samples. When the minimum cosine distance in the first distance matrix is ​​greater than or equal to a preset first threshold, it indicates that the log samples are too scattered and are not suitable for selection and further centering as the initial centroid. At this time, it is necessary to trigger an update of the target sample set.

[0081] The number of first target log samples can be multiple. In this case, the target sample set will be updated (i.e., the first target log sample will be deleted) to obtain the updated target sample set.

[0082] Furthermore, in other embodiments, when fusing log samples in the updated target sample set, the log samples in the updated target sample set can be calculated pairwise, the two log samples with the smallest cosine distance can be removed and deleted, the center of these two log samples can be calculated, and the calculated center can be put back into the updated target sample set. This method is repeated until the two log samples with the smallest cosine distance are fused, until the dataset has reached a second preset number of samples.

[0083] The second preset quantity condition can be 10 * the remaining required centroids, where the remaining required centroids are the initial centroids K required by the K-means clustering algorithm minus the total number of centroids in the center set.

[0084] This application embodiment sets preset conditions. When the preset conditions are triggered, the target log sample can be updated in a timely manner. This avoids the log samples in the target log sample being too scattered, which would lead to a decrease in the quality of the selected centroids and reduce the quality of the initial centroids. This further improves the effect of the Kmeans clustering algorithm.

[0085] In some embodiments, the step S206 above, "to continue to obtain the second center point from the fused and updated target sample set according to the second distance matrix," can be implemented through the following steps: obtaining the maximum cosine distance from the second distance matrix, and calculating the second adaptive radius based on the maximum cosine distance; obtaining the two second target log samples corresponding to the minimum cosine distance from the second distance matrix; obtaining the centers of the two second target log samples, and determining whether the distance between the centers of the two second target log samples and the center point in the center set is greater than or equal to the second adaptive radius; if the distance between the centers of the two second target log samples and the center point in the center set is greater than or equal to the second adaptive radius, then the centers of the two second target log samples are taken as the second center point.

[0086] In this embodiment, after updating the target sample set, if there are not enough centroids in the centroid set (for example, the K-means clustering algorithm requires K initial centroids, but the centroid set only has K-1 centroids), a second centroid can be added to the centroid set until the centroid set has K centroids.

[0087] In this process, the cosine distance between log samples in the updated target sample set can be calculated pairwise to obtain the second distance matrix D1. The maximum cosine distance is found in the second distance matrix D1 and denoted as distmax1. The second adaptive radius R2 is calculated, where R2 = distmax1 / the number of remaining centroids required. The number of remaining centroids required is the number of log samples in the K-center set Cens, and K is the number of initial centroids required by the K-means clustering algorithm.

[0088] In this embodiment, the two log samples corresponding to the minimum cosine distance are found from the second distance matrix D1. The value at that position in the second distance matrix D1 is set to infinity (i.e., the second distance matrix D1 is updated). Simultaneously, the center of these two log samples is calculated. If the distance between this center and all log samples in the center set Cens is greater than or equal to the second adaptive radius R2, then this center is designated as the second center C and added to the center set Cens. If the center does not satisfy the condition that the distance between it and all log samples in the set Cens is greater than or equal to the second adaptive radius R2, then this center is ignored, and the search continues from the second distance matrix D1.

[0089] This application embodiment calculates a second adaptive radius and adds it to a second preset requirement as a criterion, which can select a center with better quality as the second centroid (i.e., the initial centroid), thereby improving the selection quality of the initial centroid and further improving the clustering effect of the Kmeans clustering algorithm. This can improve the accuracy of error log clustering and ensure that error logs of the same type can be accurately divided into the same group.

[0090] Furthermore, based on the above embodiments, in some other embodiments, the second center point can also be obtained by the following method: replacing the minimum cosine distance in the second distance matrix with infinity to obtain an updated second distance matrix; continuing to obtain the updated minimum cosine distance from the updated second distance matrix; if the updated minimum cosine distance is less than a first preset threshold, then obtaining the two updated second target log samples corresponding to the minimum cosine distance; if the distance between the center of the two updated second target log samples and the center point in the center set is greater than or equal to the second adaptive radius, then taking the center of the two updated second target log samples as the second center point.

[0091] In this embodiment, after the second center point is selected and added to the center set in the above embodiment, if the number of center points in the center set is still less than K, it is necessary to continue selecting a second center point.

[0092] For example, taking the minimum cosine distance in the second distance matrix as Y1, the second smallest cosine distance as Y2, the two second target log samples corresponding to the minimum cosine distance Y1 as Y11 and Y12, and the two second target log samples corresponding to the second smallest cosine distance Y2 as Y21 and Y22, after updating the second distance matrix, the minimum cosine distance Y1 is replaced with infinity. At this time, the minimum cosine distance of the updated second distance matrix is ​​Y2. Then, the center of Y21 and Y22 can be selected, and the distance between the center and the center point in the center set can be calculated to see if it is greater than or equal to the second adaptive radius R2. If so, the center of Y21 and Y22 is taken as the second center point and added to the center set.

[0093] This embodiment of the application can continuously filter out new minimum cosine distances by updating the second distance matrix, and use this to find the second centroid. This can improve the quality of the initial centroid and also ensure that the number of centroids in the centroid set can reach K as much as possible, thereby improving the initial centroid filtering efficiency of the K-means clustering algorithm.

[0094] For example, Table 1 below shows the code logic of the k-means log classification method based on adaptive radius center selection provided in the embodiments of this application:

[0095]

[0096]

[0097] Table 1

[0098] For example, Figure 3 The overall flowchart of the k-means log classification method based on adaptive radius center selection provided in the embodiments of this application is as follows: Figure 3 As shown, it includes the following steps: Step S301, calculate the cosine distance between log samples in datase pairwise to obtain the distance matrix D. Step S302, find the largest cosine distance in the distance matrix D, denoted as distmax, and calculate the adaptive radius R1, R1 = distmax / K. Step S303, find the two log samples with the smallest cosine distance in the distance matrix D. If the cosine distance is less than the threshold T1, set the value at that position in the distance matrix D to infinity, and denote the center of these two log samples as C1, and add C1 to the center set Cens. Step S304, determine whether the number of log samples in the center set Cens is less than K. Step S305, determine whether the smallest cosine distance is greater than or equal to the threshold T1. Step S3061, find the two log samples with the smallest cosine distance in the distance matrix D1. If the cosine distance is less than the threshold T1, set the value at that position in the distance matrix D to infinity. Simultaneously calculate the centers of the two log samples. If the distance between this center and the log samples in Cens is greater than or equal to R1, then denote this center as C1 and add it to the set Cens. Simultaneously, add both log samples to the set dataC. If the distance between this center and C1 is less than R1, then ignore this center and continue searching in the distance matrix until a center that meets the conditions is found and added to Cens. Step S3062: Calculate the cosine distance between each pair of log samples in daataset to obtain the distance matrix D1. Step S307: Find the largest cosine distance in the distance matrix D1, denote this distance as distmax1, and calculate the adaptive radius R2, where R2 = distmax1 / the remaining required initial centroids. Step S308: Check if the number of log samples in Cens is less than K. Step S309: Find the two log samples with the smallest cosine distance from the distance matrix D1, set the value of that position in the distance matrix D to infinity, and calculate the center of these two samples. If the distance between the center and the samples in the set Cens is greater than or equal to R2, then denote the center as C and add it to the set Cens. If the center does not satisfy the condition that the distance between it and the samples in the set Cens is greater than or equal to R2, then ignore this center and continue searching from the distance matrix D1.

[0099] This application improves the initialization phase of the kmenes algorithm by proposing an adaptive radius center selection method to disperse the centroid selection. This accelerates the initialization convergence speed and improves the quality of initial centroid selection. The resulting clustering quality is better, leading to more accurate classification of log data.

[0100] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0101] Figure 4 A schematic diagram of the structure of the k-means log classification device based on adaptive radius center selection provided in the embodiments of this application is shown below. Figure 4 As shown, the log classification device 400 includes an acquisition module 410, a first matrix construction module 420, a first centroid storage module 430, a sample set update module 440, a second matrix construction module 450, a second centroid storage module 460, and a clustering module 470.

[0102] The acquisition module 410 is used to acquire a target sample set and fuse log samples in the target sample set until the number of log samples in the target sample set meets a first preset quantity condition, wherein the target sample set includes at least one log sample. The first matrix construction module 420 is used to determine the cosine distance between each log sample in the target sample set and construct a first distance matrix based on the cosine distance. The first center point storage module 430 is used to acquire a first center point based on the first distance matrix and store it in a center set, wherein the first center point is the center of two first target log samples in the target sample set that meet the first preset requirement. The sample set update module 440 is used to delete the first target log samples and fuse and update the target sample set after deleting the first target log samples if the preset condition is triggered, until the number of log samples meets a second preset quantity condition. The second matrix construction module 450 is used to construct a second distance matrix based on the cosine distance between each log sample in the fused and updated target sample set. The second centroid storage module 460 is used to continue obtaining second centroids from the fused and updated target sample set according to the second distance matrix, and storing them in the centroid set until the centroids in the centroid set are K. The second centroids are the centers of two second target log samples in the fused and updated target sample set that meet the second preset requirements, where K is a positive integer. The clustering module 470 is used to obtain the centroids in the centroid set as initial points and perform K-means clustering.

[0103] Optionally, the acquisition module can be used to: randomly sample the initial log sample set according to a preset sampling ratio to obtain the sampled log samples; and combine the sampled log samples into a target sample set.

[0104] Optionally, the acquisition module can be used to: acquire the cosine distance of each log sample in the target sample set, delete the two log samples with the smallest cosine distance; acquire the center of the two deleted log samples, and restore the center to the target sample set.

[0105] Optionally, the first center point storage module can be used to: obtain the two first target log samples corresponding to the minimum cosine distance from the first distance matrix; if the minimum cosine distance is less than the first preset threshold, then take the center of the two first target log samples as the first center point.

[0106] Optionally, the first center point storage module can be used to: obtain the maximum cosine distance in the first distance matrix, and calculate the first adaptive radius based on the maximum cosine distance; replace the minimum cosine distance in the first distance matrix with infinity to obtain the updated first distance matrix; continue to obtain the updated minimum cosine distance from the updated first distance matrix; if the updated minimum cosine distance is less than a first preset threshold, then continue to obtain the two first target log samples corresponding to the updated minimum cosine distance; if the distance between the center of the two first target log samples corresponding to the updated minimum cosine distance and the center point in the center set is less than the first adaptive radius, then take that center as the first center point.

[0107] Optionally, the sample set update module can be used to trigger a preset condition when the minimum cosine distance in the first distance matrix is ​​greater than or equal to a preset first threshold.

[0108] Optionally, the second center point storage module can be used to: obtain the maximum cosine distance from the second distance matrix, and calculate the second adaptive radius based on the maximum cosine distance; obtain the two second target log samples corresponding to the minimum cosine distance from the second distance matrix; obtain the centers of the two second target log samples, and determine whether the distance between the centers of the two second target log samples and the center point in the center set is greater than or equal to the second adaptive radius; if the distance between the centers of the two second target log samples and the center point in the center set is greater than or equal to the second adaptive radius, then the centers of the two second target log samples are taken as the second center point.

[0109] Optionally, the second center point storage module can be used to: replace the minimum cosine distance in the second distance matrix with infinity to obtain an updated second distance matrix; continue to obtain the updated minimum cosine distance from the updated second distance matrix; if the updated minimum cosine distance is less than a first preset threshold, then obtain the two updated second target log samples corresponding to the minimum cosine distance; if the distance between the center of the two updated second target log samples and the center point in the center set is greater than or equal to the second adaptive radius, then take the center of the two updated second target log samples as the second center point.

[0110] The apparatus provided in this application embodiment can be used to execute the methods in the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0111] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the acquisition module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0112] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes at least one processor 501, a memory 502, a bus 503, and a communication interface 504. The processor, communication interface, and memory communicate with each other via the bus. The communication interface is used to communicate with other devices. This communication interface includes a communication interface for data transmission and a display interface or operation interface for human-computer interaction. The processor executes computer instructions stored in the memory, specifically performing the relevant steps in the methods described in the above embodiments.

[0113] The processor may be a central processing unit, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0114] Memory is used to store instructions executed by a computer. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk drive.

[0115] This embodiment also provides a computer-readable storage medium storing computer instructions, which, when executed by at least one processor of an electronic device, enable the electronic device to perform the methods provided in the various embodiments described above.

[0116] This embodiment also provides a computer program product including computer instructions stored in a readable storage medium. At least one processor of an electronic device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to cause the electronic device to perform the methods provided in the various embodiments described above.

[0117] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0118] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0120] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A k-means log classification method based on adaptive radius center selection, characterized in that, include: Obtain a target sample set, and merge the log samples in the target sample set until the number of log samples in the target sample set meets a first preset quantity condition, wherein the target sample set includes at least one log sample; Determine the cosine distance between each log sample in the target sample set, and construct a first distance matrix based on the cosine distance; Based on the first distance matrix, the first center point is obtained and stored in the center set. The first center point is the center of two first target log samples in the target sample set that meet the first preset requirements. If the preset condition is triggered, the first target log sample will be deleted and the target sample set after deleting the first target log sample will be merged and updated until the number of log samples meets the second preset number condition. Construct a second distance matrix based on the cosine distance between each log sample in the merged and updated target sample set; According to the second distance matrix, the second center point is obtained from the fused and updated target sample set and stored in the center set until the center point in the center set is K. The second center point is the center of the two second target log samples in the fused and updated target sample set that meet the second preset requirements, and K is a positive integer. The center points in the center set are used as initial mass points, and K-means clustering is performed. The step of obtaining the first center point based on the first distance matrix includes: Obtain the maximum cosine distance in the first distance matrix, and calculate the first adaptive radius based on the maximum cosine distance; Replace the minimum cosine distance in the first distance matrix with infinity to obtain the updated first distance matrix; Continue to obtain the updated minimum cosine distance from the updated first distance matrix; If the updated minimum cosine distance is less than the first preset threshold, then continue to obtain the two first target log samples corresponding to the updated minimum cosine distance; If the distance between the center of the two first target log samples corresponding to the updated minimum cosine distance and the center point in the center set is less than the first adaptive radius, then the center is taken as the first center point; The step of obtaining the second center point from the fused and updated target sample set based on the second distance matrix includes: Replace the minimum cosine distance in the second distance matrix with infinity to obtain the updated second distance matrix; Continue to obtain the updated minimum cosine distance from the updated second distance matrix; If the updated minimum cosine distance is less than the first preset threshold, then obtain the two updated second target log samples corresponding to the minimum cosine distance. If the distance between the center of the two updated second target log samples and the center point in the center set is greater than or equal to the second adaptive radius, then the center of the two updated second target log samples is taken as the second center point.

2. The method according to claim 1, characterized in that, The acquisition of the target sample set includes: According to the preset sampling ratio, the initial log sample set is randomly sampled to obtain the sampled log samples; The sampled log samples are combined into the target sample set.

3. The method according to claim 1, characterized in that, The fusion of log samples in the target sample set includes: Obtain the cosine distance between each log sample in the target sample set, and delete the two log samples with the smallest cosine distance; Obtain the center of the two deleted log samples and restore the center to the target sample set.

4. The method according to claim 1, characterized in that, The preset conditions are triggered, including: The preset condition is triggered when the minimum cosine distance in the first distance matrix is ​​greater than or equal to a preset first threshold.

5. A k-means log classification device based on adaptive radius center selection, wherein the k-means log classification device based on adaptive radius center selection is used to implement the k-means log classification method based on adaptive radius center selection as described in any one of claims 1-4, characterized in that, include: The acquisition module is used to acquire a target sample set and merge the log samples in the target sample set until the number of log samples in the target sample set meets a first preset quantity condition, wherein the target sample set includes at least one log sample. The first matrix construction module is used to determine the cosine distance between each log sample in the target sample set, and construct a first distance matrix based on the cosine distance. The first center point storage module is used to obtain the first center point according to the first distance matrix and store it in the center set. The first center point is the center of two first target log samples in the target sample set that meet the first preset requirements. The sample set update module is used to delete the first target log sample and merge and update the target sample set after deleting the first target log sample if a preset condition is triggered, until the number of log samples meets the second preset quantity condition. The second matrix construction module is used to construct a second distance matrix based on the cosine distance between each log sample in the fused and updated target sample set. The second center point storage module is used to continue to obtain second center points from the fused and updated target sample set according to the second distance matrix and store them into the center set until the center point in the center set is K. The second center point is the center of two second target log samples in the fused and updated target sample set that meet the second preset requirement, and K is a positive integer. The clustering module is used to obtain the center points in the center set as initial mass points and perform K-means clustering.

6. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, When executed by a processor, the computer program product is used to implement the method as described in any one of claims 1 to 4.

Images