Method for monitoring kmeans anomaly log retrieval based on semi-supervised density gradient

Abstract

The application provides a semi-supervised density gradient monitoring Kmeans anomaly log retrieval method and device, and relates to the field of big data.The method comprises the following steps: dividing log samples in a log sample set into K clusters, and each cluster comprises at least one log sample as a centroid;obtaining edge log samples in the cluster, and determining the cosine angle between the centroid of the cluster and the edge log samples;constructing N sample sets according to the cosine angle between the centroid of the cluster and the edge log samples;according to the cosine angle between each log sample in the cluster and the centroid of the cluster, determining the sample set corresponding to the log sample, and dividing the log sample into the sample set;according to the number of log samples in each sample set, determining non-stable sample sets in all sample sets and combining to form a target sample set, and determining abnormal log samples in the target sample set.The scheme can improve the retrieval accuracy of abnormal samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, in particular to a semi-supervised density gradient monitoring Kmeans abnormal log retrieval method and device. BACKGROUND

[0002] Log is a computer term, and application platforms will generate logs when they are running. Each line of log records the date, time, user, and description of related operations. When the application platform is running normally, it generates normal logs. When the application platform has a problem, it is easy to generate abnormal logs. These abnormal logs play an important role in preventing and analyzing potential problems of the application platform, facilitating the discovery of new problem categories, and being conducive to the formation of measures to avoid such problems and the maintenance and management of the application platform.

[0003] In the prior art, the retrieval of abnormal logs mainly relies on clustering algorithms, which use text clustering to cluster logs and find out log samples that do not belong to any cluster as abnormal logs.

[0004] However, the existing clustering method cannot finely select samples that do not belong to any cluster, and it is also difficult to accurately find abnormal points, resulting in poor clustering effect and low accuracy of abnormal log retrieval. SUMMARY

[0005] The present application provides a semi-supervised density gradient monitoring Kmeans abnormal log retrieval method and device to solve the problem of low accuracy of abnormal log retrieval.

[0006] In a first aspect, the present application provides a semi-supervised density gradient monitoring Kmeans abnormal log retrieval method, comprising:

[0007] Obtaining a log sample set, dividing the log samples in the log sample set into K clusters, the log sample set including at least K log samples, each cluster including at least one log sample as a centroid, and K being a positive integer;

[0008] Obtaining an edge log sample in the cluster, determining the cosine angle between the centroid of the cluster and the edge log sample;

[0009] According to the cosine angle between the centroid of the cluster and the edge log sample, constructing N sample sets, N being a positive integer;

[0010] Obtaining the cosine angle between each log sample in the cluster and the centroid of the cluster, determining the sample set corresponding to the log sample according to the cosine angle between each log sample in the cluster and the centroid of the cluster, and dividing the log sample into the sample set;

[0011] determine, from all the sample sets, a non-stable sample set according to the number of log samples in each sample set;

[0012] combine the log samples in all the non-stable sample sets to form a target sample set, and determine abnormal log samples in the target sample set.

[0013] In a second aspect, the present application provides a semi-supervised density gradient monitoring Kmeans abnormal log retrieval device, comprising:

[0014] a sample set obtaining module, configured to obtain a log sample set, divide log samples in the log sample set into K clusters, the log sample set comprising at least K log samples, and each cluster comprising at least one log sample as a centroid, K being a positive integer;

[0015] a cosine angle determining module, configured to obtain an edge log sample in the cluster, and determine a cosine angle between the centroid of the cluster and the edge log sample;

[0016] a sample set constructing module, configured to construct N sample sets according to the cosine angle between the centroid of the cluster and the edge log sample, N being a positive integer;

[0017] a sample dividing module, configured to obtain a cosine angle between each log sample in the cluster and the centroid of the cluster, determine a sample set corresponding to the log sample according to the cosine angle between each log sample in the cluster and the centroid of the cluster, and divide the log sample into the sample set;

[0018] a non-stable sample set determining module, configured to determine, from all the sample sets, a non-stable sample set according to the number of log samples in each sample set;

[0019] an abnormal sample determining module, configured to combine the log samples in all the non-stable sample sets to form a target sample set, and determine abnormal log samples in the target sample set.

[0020] In a third aspect, the present application provides an electronic device, comprising a processor and a memory connected with the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method as described above.

[0021] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium storing computer execution instructions, the computer execution instructions being executed by a processor to implement the method as described above.

[0022] In a fifth aspect, the present application provides a computer program product, comprising a computer program, the computer program being executed by a processor to implement the method as described above.

[0023] The semi-supervised density gradient monitoring Kmeans anomaly log retrieval method and device provided in the application can improve the retrieval accuracy of abnormal samples by using the cosine angle as a measure to distinguish the proximity of samples and cluster centers, setting a cosine angle gradient detection to determine N sample intervals, and finding sample points that are both outliers and sparse as abnormal points based on log samples in the N sample intervals. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0025] Figure 1 A schematic diagram of the clustering algorithm provided for the embodiments of the application;

[0026] Figure 2 A flowchart of the semi-supervised density gradient monitoring Kmeans anomaly log retrieval method provided for the embodiments of the application;

[0027] Figure 3 A log sample schematic diagram provided for the embodiments of the application;

[0028] Figure 4 A flowchart of the semi-supervised density gradient monitoring Kmeans anomaly log retrieval method provided for the embodiments of the application;

[0029] Figure 5 A structural schematic diagram of the semi-supervised density gradient monitoring Kmeans anomaly log retrieval device provided for the embodiments of the application;

[0030] Figure 6 A structural schematic diagram of the electronic device provided for the embodiments of the application.

[0031] The above-described drawings have shown specific embodiments of the application, and the following will have a more detailed description. These drawings and written descriptions are not intended to limit the scope of the concept of the application in any way, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0032] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to designate the same elements, unless otherwise indicated, in different drawings. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the application. Rather, they are merely examples of devices and methods consistent with some aspects of the application, as detailed in the appended claims.

[0033] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0034] It should be noted that the semi-supervised density gradient monitoring K-means anomaly log retrieval method and apparatus provided in this application can be used in the field of big data, or in any field other than big data. The application field of the semi-supervised density gradient monitoring K-means anomaly log retrieval method and apparatus provided in this application is not limited.

[0035] The k-means clustering algorithm is an iterative clustering analysis algorithm. Its steps are as follows: First, the data is pre-divided into K groups. Then, K objects are randomly selected as initial cluster centers. Next, the distance between each object and each seed cluster center is calculated, and each object is assigned to the nearest cluster center. The cluster centers and the objects assigned to them represent a cluster. Each time a sample is assigned, the cluster centers are recalculated based on the existing objects in the cluster. This process is repeated until a termination condition is met. The termination condition could be that no (or a minimum number) objects have been reassigned to different clusters.

[0036] Abnormal log detection aims to find logs that do not belong to the regular classification, and such logs are generally a minority compared to the overall amount of logs. Because of the minority of special and alternative, it plays an important role in preventing potential problems of the analysis application platform, facilitates the discovery of new problem categories, is conducive to the formation of measures to avoid such problems, and is conducive to the maintenance and management of the application platform. In the initialization stage, the traditional kmeans clustering algorithm randomly extracts the initial centroid, which will cause the extracted centroid to be too scattered or concentrated, not particularly uniform, so as to slow down the clustering convergence speed and make the clustering effect bad. Especially under the condition of massive text data, such defects will be highlighted. After re-clustering, whether the sample log is an abnormal sample point is judged by calculating the distance between the sample log and the cluster center. This kind of judgment method is not good at selecting the sample that does not belong to any cluster, and it is difficult to accurately find the abnormal point only by the distance index. In summary, the traditional kmeans clustering algorithm has the following problems: ①In the initialization stage, the centroid selection has randomness, the initial centroid is too close or scattered, not uniform enough, and cannot balance the selection of the centroid of all categories. ②Only by the distance between the sample point and the cluster center to judge whether the sample point is an abnormal point (and how to define the distance is not easy) is not accurate enough, and if the log sample is relatively dense as a whole, it is difficult to define the log sample far from the center as an abnormal point.

[0037] In view of the above problems, the embodiment of the application provides a Kmeans abnormal log retrieval method and device based on semi-supervised density gradient monitoring, which uses the cosine angle as a measure to distinguish the distance between the sample and the cluster center, and sets the cosine angle gradient detection to determine N sample intervals, and considers the sample point density of each sample interval, so as to find the sample point that is both outlier and sparse as abnormal point, which can improve the retrieval accuracy of abnormal samples.

[0038] The technical solutions of the application and how the technical solutions of the application solve the above technical problems will be described in detail in the specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the application will be described below with reference to the drawings.

[0039] Figure 1 The schematic diagram of the clustering algorithm provided by the embodiment of the application is as follows: Figure 1As shown, Y1 to Y20 are taken as an example, Y1 to Y8 are clustered into one cluster, and Y9 to Y20 are clustered into another cluster. If there is an abnormal outlier that does not belong to the two clusters, the sample log represented by the abnormal outlier can be determined as an abnormal log. The log can be a computer term, and the application platform generates logs during operation; each line of log records the date, time, user, and action related to the description of the operation.

[0040] Figure 2 The flowchart of the semi-supervised density gradient monitoring Kmeans abnormal log retrieval method provided in the embodiment of the application can be applied to the retrieval of logs generated by the application platform to find abnormal logs existing therein. As an example, the execution subject of the method is an electronic device, as shown in Figure 2 The method can include the following steps: in step S201, a log sample set is obtained, and the log samples in the log sample set are divided into K clusters. The log sample set includes at least K log samples, and each cluster includes at least one log sample as a centroid. K is a positive integer.

[0041] In the embodiment, the log sample set can be a collection of logs generated by the application platform during operation, for example, all logs generated by the application platform within a certain time period are collected to form the log sample set. It can be understood that the number of logs in the log sample set is large, and the longer the time period, the more logs are collected from the application platform, and the larger the log capacity of the log sample set. Generally, the log sample set includes a large number of regular logs and a small number of irregular logs (i.e., abnormal logs), and these abnormal logs usually account for a small proportion. Therefore, it is difficult to accurately locate the abnormal logs from the log sample set, and inaccurate positioning may occur. It should be noted that the abnormal logs often represent potential problems of the application platform, and by analyzing the abnormal logs, potential problems of the application platform can be found in time, so that measures can be taken in advance.

[0042] In the embodiment, K log samples of different categories can be selected from the log sample set by human beings, as centroids. Then, based on the K centroids, the log samples in the log sample set are clustered, for example, the distance between each log sample and each centroid is calculated. If the log sample is closest to a centroid, the log sample and the centroid are clustered into the same cluster. The log sample set can include log samples of various categories, such as user logs, program logs, and operating system logs.

[0043] Figure 3 The log sample diagram provided in the embodiment of the application is shown in Figure 3As shown, by digitizing the log samples, each log sample is replaced by a coordinate point, and L1, L2 and L3 can be selected as the K initial specified centroids. In other embodiments, the K different categories of log samples can be specified in an artificial manner. Specifically, a target instruction can be obtained, the target instruction at least including indication information of selecting K different categories of log samples; and according to the target instruction, K different categories of log samples are selected from the log sample set.

[0044] The target instruction is inputted by a human being, and after the electronic device receives the target instruction, the K different categories of log samples are determined from the log sample set as the K centroids based on the target instruction. It should be noted that the current kmeans clustering algorithm randomly selects K objects as the centroids, which is not representative. By artificially specifying the K initial centroids, the problem that the initial centroids cannot represent all log categories can be avoided, thereby further improving the accuracy of subsequent abnormal log retrieval.

[0045] In this embodiment, each centroid is taken as the center of each category cluster, and then other log samples in the log sample set are clustered to form K category clusters. Specifically, if a log sample is close to one of the centroids, the log sample will be divided into a category cluster with the centroid as the center.

[0046] For example, continuing to refer to Figure 3 Taking L1, L2 and L3 as the three initial specified centroids as an example, log samples L4, L5 and L6 are close to the centroid L1, and at this time, L1, L4, L5 and L6 are divided into the same category cluster, and the center of the category cluster is the centroid L1. Log samples L7 and L8 are close to the centroid L2, and at this time, L2, L7 and L8 are divided into another category cluster, and the center of the category cluster is the centroid L2.

[0047] In step S202, the edge log sample in the cluster is obtained, and the cosine angle between the centroid of the cluster and the edge log sample is determined.

[0048] In this embodiment, there will be an edge log sample at the edge of each cluster, which can continue to refer to Figure 3 In the first cluster (including log samples L1, L4, L5 and L6), the log sample L4 is at the edge of the cluster, and the log sample L4 can be taken as the edge log sample.

[0049] After the edge log sample is determined, the cosine distance between the log sample L4 and the centroid L1 of the cluster can be calculated, and then the cosine angle between the log sample L4 and the centroid L1 of the cluster is further inferred based on the cosine distance.

[0050] Step S203, constructing N sample sets according to the cosine angle between the centroid of the cluster and the edge log sample, N being a positive integer.

[0051] In this embodiment, the cosine angle can be divided into angle intervals, for example, the cosine angle is A, which can be divided into multiple angle intervals, for example, angle intervals [0, A1], (A1, A2], (A2, A]. At the same time, each angle interval corresponds to a sample set constructed, that is, 3 sample sets are obtained (that is, the value of N is 3 at this time).

[0052] Step S204, obtaining the cosine angle between each log sample in the cluster and the centroid of the cluster, determining the sample set corresponding to the log sample according to the cosine angle between each log sample in the cluster and the centroid of the cluster, and dividing the log sample into the sample set.

[0053] In this embodiment, the cosine distance between each log sample in the cluster and the centroid of the cluster can still be calculated in the manner mentioned in the above steps, and then the cosine angle between the log sample and the centroid of the cluster is further inferred based on the cosine distance. For example, the cosine angle between the log sample and the centroid of the cluster is A3, if A3 is greater than A2 and less than A, the log sample will be divided into the sample set corresponding to the angle interval (A2, A].

[0054] Step S205, determining a non-stable sample set in all sample sets according to the number of log samples in each sample set.

[0055] In this embodiment, a threshold value can be set, if the number of log samples in a sample set is lower than the threshold value, the sample set can be determined as a non-stable sample set. In other embodiments, the number of log samples in the current sample set can also be calculated based on the number of log samples in the previous sample set, the number of log samples in the current sample set, and the cosine angle A, and the number of log samples in the current sample set is determined by the size of the number change rate to determine whether the sample set is a non-stable sample set.

[0056] Step S206, combining the log samples in all non-stable sample sets to form a target sample set, and determining the abnormal log samples in the target sample set.

[0057] In this embodiment, the density of each sample can be calculated (that is, the density of each log sample is determined based on the number of log samples existing around the log sample), and then based on the density of each log sample, it is determined which log samples are abnormal log samples.

[0058] The embodiment of the application distinguishes the distance between the sample and the cluster center by using the cosine angle as a measure, sets a cosine angle gradient detection to determine N sample intervals, and considers the sample point density of each sample interval to find sample points that are both outliers and sparse as abnormal points, thereby improving the retrieval accuracy of abnormal samples.

[0059] In some embodiments, the above-mentioned dividing the log samples in the log sample set into K clusters can be implemented by the following steps: selecting K log samples of different categories from the log sample set as the initial specified K centroids; calculating the cosine distance between each log sample in the log sample set and the initial specified K centroids; and dividing each log sample in the log sample set into different clusters according to the cosine distance between each log sample and each centroid, to obtain K clusters.

[0060] In the embodiment, the log sample set can include a plurality of log samples, and the cosine distance between each log sample and each centroid can be calculated. For example, the above-mentioned Figure 3 Taking the centroids L1, L2 and L3 as an example, the cosine distance between the log sample and L1, L2 and L3 can be calculated to obtain the corresponding cosine distances L11, L12 and L13. When L11 is less than L12 which is less than L13, the log sample is divided into the same cluster with the centroid L1. It should be noted that each centroid has a corresponding cluster, that is, the K centroids have K clusters.

[0061] In the embodiment, each initial specified centroid can create a corresponding empty cluster, and then the log sample is divided into the cluster, thereby obtaining K clusters with log samples. Specifically, based on the nearest distance principle, if the cosine distance between the log sample and the centroid L1 of the K centroids is the smallest, the log sample is divided into the cluster corresponding to the centroid L1.

[0062] The embodiment of the application improves the original kmeans clustering algorithm, specifies K log samples of different categories as K centroids, and re-clusters the log samples in the log sample set based on the K centroids, thereby avoiding the problem that the randomly selected centroid cannot represent all log categories, and improving the clustering effect.

[0063] In other embodiments, the above-mentioned obtaining the edge log sample in the cluster can be implemented by the following steps: calculating the cosine distance between each log sample in the cluster and the centroid of the cluster; and obtaining the log sample with the largest cosine distance in the cluster as the edge log sample.

[0064] In the embodiment, a plurality of log samples can be included in each cluster, the cosine distance between each log sample and the centroid of the cluster can be calculated, and then the cosine distances are compared in size, and a log sample with the largest cosine distance is selected as the edge log sample. Each log sample can be converted into a vector form, and the centroid of the cluster can also be converted into a vector form. The cosine distance can be calculated by using the vector form of the log sample and the centroid. Figure 3 Taking the revised centroid L1 as an example, the cosine distance between the log sample L4 and the revised centroid L1 is usually smaller than the cosine distance between the log sample L5 and the revised centroid L1, that is, the cosine distance can represent the proximity between the log samples.

[0065] By finding the edge log sample in each cluster, the cosine angle gradient can be set based on the cosine angle between the edge log sample and the centroid of the cluster, so as to further divide other log samples in the cluster into different sample sets, and the accuracy of the log anomaly search is further improved.

[0066] In some embodiments, the step of "determining the cosine angle between the centroid of the cluster and the edge log sample" can be implemented by the following steps: obtaining the modulus of the centroid of the cluster and the modulus of the edge log sample; calculating the cosine distance between the centroid of the cluster and the edge log sample; and calculating the cosine angle between the centroid of the cluster and the edge log sample according to the modulus of the centroid of the cluster, the modulus of the edge log sample, the cosine distance between the centroid of the cluster and the edge log sample, and a pre-designed calculation formula.

[0067] In the embodiment, each log sample can be converted into a vector form, and the centroid of the cluster can also be converted into a vector form. The cosine distance can be calculated by using the vector form of the log sample and the centroid.

[0068] Cosine distance = modulus of cluster centroid * modulus of edge point * cos A

[0069] In the above formula, the cluster centroid refers to the vector form of the centroid of the cluster, and the edge point refers to the vector form of the edge log sample. If the cluster centroid is (X, Y), the modulus of the cluster centroid is the square root of the sum of the square of X and the square of Y.

[0070] By calculating the cosine angle between the centroid of the cluster and the edge log sample, the cosine angle gradient can be set based on the cosine angle A, so as to divide other log samples in the cluster to obtain N sample sets, and the accuracy of the log anomaly search is further improved.

[0071] In some embodiments, when the N sample sets are constructed, the following steps can be specifically implemented: dividing the cosine angle of the cluster center and the edge log sample into N equal parts to obtain N angle intervals; and constructing a corresponding sample set for each angle interval to obtain N sample sets. Correspondingly, the sample set corresponding to the log sample in the cluster can be determined by the following steps: determining the angle interval in which the cosine angle of each log sample in the cluster and the cluster center is located; and determining the sample set corresponding to the log sample according to the angle interval in which the cosine angle of each log sample in the cluster and the cluster center is located.

[0072] In this embodiment, the cosine angle A / N can be obtained to obtain an interval angle Aa. For example, N can be 10, and Aa=A / 10. The log sample in the cluster can be placed in the sample set Dn (n is 1-10). Specifically, the cosine angle of the log sample in the cluster and the cluster center is calculated, and then the log sample is divided based on the following criteria:

[0073] That is, the sample with a cosine angle of 0<Aa is placed in the D1 sample set;

[0074] The sample with a cosine angle of Aa<2Aa is placed in the D2 sample set;

[0075] The sample with a cosine angle of 2Aa<3Aa is placed in the D3 sample set; ...

[0077] The sample with a cosine angle of 9Aa<10Aa is placed in the D10 sample set.

[0078] The embodiments of the present application divide the cosine angle A into N equal parts, and configure a corresponding sample set for the divided cosine angle, so that the gradient detection is used to observe and find the area far from the cluster center, which can facilitate the subsequent finding of the non-stable sample set with abnormal logs and improve the retrieval accuracy of abnormal logs.

[0079] In some embodiments, when the non-stable sample set is determined, the following steps can be specifically implemented: calculating the log sample quantity change rate of each sample set according to the number of log samples in each sample set; comparing the log sample quantity change rate of each sample set with a preset change rate threshold; and if the log sample quantity change rate of the sample set is greater than the preset change rate threshold, determining that the sample set is a non-stable sample set.

[0080] In this embodiment, a preset change rate threshold Th can be set, and then the change rate of each sample set (for example, the sample sets D1 to D10 described above) is calculated. For example, the change rate of each sample set is calculated according to the following formula:

[0081] The change rate t1 of the number of log samples in D1 to D2, i.e., t1=(the number of log samples in D2-the number of log samples in D1) / Aa, is calculated as the change rate of the sample set D2. The change rate of D1 can be the number of log samples in D1 / Aa. Aa is the angle interval mentioned above.

[0082] The change rate t2 of the number of log samples in D2 to D3, i.e., t2=(the number of log samples in D3-the number of log samples in D2) / Aa, is calculated as the change rate of the sample set D3. ......

[0084] The change rate t9 of the number of log samples in D9 to D10, i.e., t9=(the number of log samples in D10-the number of log samples in D9) / Aa, is calculated as the change rate of the sample set D10.

[0085] In this embodiment, the sizes of t1, t2,..., t9 and the threshold value Th can be compared one by one until the change rate tx greater than the preset change rate threshold value Th is found. Then D1, D2,..., Dx are marked as stable regions, and Dx+1,..., D10 are marked as non-stable regions, i.e., non-stable sample sets.

[0086] The embodiments of the present application can represent which sample sets have a gradually decreasing number of log samples by calculating the change rate of the number of log samples of the sample sets, so that the non-stable sample sets can be found, the search range of abnormal log samples is reduced, and the search efficiency of abnormal log samples is improved.

[0087] In some embodiments, the abnormal log sample determined in the target log sample set can be realized by the following steps: calculating the density of each log sample in the target sample set; sorting all log samples in the target sample set according to the density of each log sample to obtain a log sample sequence; and determining the abnormal log sample in the log sample sequence.

[0088] In this embodiment, the density can represent the number of other log samples around the log sample. Generally, the larger the number of other log samples, the smaller the density of the log sample, and the smaller the number of other log samples, the larger the density of the log sample.

[0089] In this embodiment, all log samples in the target sample set can be sorted in descending order to obtain a log sample sequence, and then the sample at the end of the log sample sequence List is set as the abnormal log sample.

[0090] Further, in other embodiments, the density of the log sample can be determined by the following steps: obtaining the cosine distance between each log sample; obtaining Kn neighbor log samples of the target log sample according to the cosine distance between each log sample, to form a first neighbor set, the neighbor log sample being a log sample whose cosine distance with the target log sample meets a preset distance condition, and Kn being a positive integer; obtaining Kn neighbor log samples of each log sample in the first neighbor set to form a second neighbor set; obtaining the union set of the first neighbor set and the second neighbor set, and determining the total number of log samples contained in the union set; and determining the density of the target log sample according to the total number of log samples contained in the union set.

[0091] In the present embodiment, the density calculation formula of the log sample is as follows:

[0092] Density of log sample X = 1 / (second-order Kn neighbor sample number of X)

[0093] Wherein, the second-order Kn neighbor sample number of X refers to: the set of Kn neighbor samples closest to the log sample point X (cosine distance) is neighbourl, the set of Kn neighbor samples closest to each neighbor sample point in neighbourl is neighbour2, the union set of neighbourl and neighbour2 is neighbour_total, and the number of log sample points in neighbour_total is the second-order Kn neighbor sample number of X.

[0094] The present embodiment finds abnormal samples that are both outliers and sparse through sample point density (the density of the sample point is represented by the number of times the sample point is cited by other sample points), which can improve the search accuracy of abnormal samples and avoid the problem that only considering the cosine distance in a single dimension cannot better identify abnormal outliers.

[0095] Exemplarily, Figure 4 The flowchart of the Kmeans abnormal log search method based on semi-supervised density gradient monitoring provided by the present embodiment is as follows: Figure 4As shown, it includes the following steps: S401, obtaining log data. Step S402, manually specifying K centroids, calculating the cosine distance between each log sample in the log data and the K centroids, dividing the samples to the nearest centroid, and finally obtaining K clusters. Step S403, for each cluster, find the cosine distance between the cluster centroid and the log samples in the cluster, find the sample point farthest from the centroid as the edge log sample, and calculate the cosine distance A between the cluster center and the edge log sample. Step S404, divide the cosine distance A into 10 equal parts, i.e. obtain the interval distance Aa=A / 10, calculate the cosine angle between all log samples in the cluster and the cluster centroid, and put each log sample into the corresponding sample set. Step S405, calculate the change rate tn between the sets Dn, find n whose change rate tn is greater than the threshold value Th, and take Dn+1, Dn+2,..., D10 as the unstable region. Step S406, form the target sample set dataF from the unstable region, then calculate the density of each log sample in dataF, sort the log sample sequence List in descending order of density, and take the sample at the end of the log sample sequence List as the abnormal log sample according to the set threshold value.

[0096] In this embodiment, computer execution instructions can be constructed to implement the scheme through the computer execution instructions. For example, the computer instructions can be as shown in Table 1:

[0097]

[0098] Table 1

[0099] The following is an embodiment of the device of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0100] Figure 5 The structure diagram of the device for retrieving abnormal logs based on semi-supervised density gradient monitoring Kmeans provided by the embodiment of the present application is as shown in Figure 5 The device for retrieving abnormal logs 500 can specifically include a sample set acquisition module 510, a cosine angle determination module 520, a sample set construction module 530, a sample division module 540, a non-stable sample set determination module 550, and an abnormal sample determination module 560.

[0101] The sample set obtaining module 510 is configured to obtain a log sample set, divide log samples in the log sample set into K clusters, the log sample set comprising at least K log samples, and each cluster comprising at least one log sample as a centroid, K being a positive integer. The cosine angle determining module 520 is configured to obtain an edge log sample in the cluster, and determine a cosine angle between the centroid of the cluster and the edge log sample. The sample set constructing module 530 is configured to construct N sample sets according to the cosine angle between the centroid of the cluster and the edge log sample, N being a positive integer. The sample dividing module 540 is configured to obtain a cosine angle between each log sample in the cluster and the centroid of the cluster, determine a sample set corresponding to the log sample according to the cosine angle between each log sample in the cluster and the centroid of the cluster, and divide the log sample into the sample set. The unstable sample set determining module 550 is configured to determine, from all sample sets, an unstable sample set according to a number of log samples in each sample set. The abnormal sample determining module 560 is configured to combine log samples in all unstable sample sets to form a target sample set, and determine an abnormal log sample in the target sample set.

[0102] Optionally, the sample set obtaining module can be specifically configured to: select K log samples of different categories from the log sample set as the K specified centroids; calculate a cosine distance between each log sample in the log sample set and the K specified centroids; and divide each log sample in the log sample set into different clusters according to the cosine distance between each log sample and each centroid, to obtain the K clusters.

[0103] Optionally, the cosine angle determining module can be specifically configured to: calculate a cosine distance between each log sample in the cluster and the centroid of the cluster; and obtain a log sample with the largest cosine distance in the cluster as the edge log sample.

[0104] Optionally, the cosine angle determining module can be specifically configured to: obtain a modulus of the centroid of the cluster and a modulus of the edge log sample; calculate a cosine distance between the centroid of the cluster and the edge log sample; and calculate the cosine angle between the centroid of the cluster and the edge log sample according to the modulus of the centroid of the cluster, the modulus of the edge log sample, the cosine distance between the centroid of the cluster and the edge log sample, and a pre-designed calculation formula.

[0105] Optionally, the sample set constructing module can be specifically configured to: divide the cosine angle between the centroid of the cluster and the edge log sample into N equal angle intervals, to obtain N angle intervals; and construct a corresponding sample set for each angle interval, to obtain the N sample sets. Correspondingly, the sample dividing module can be specifically configured to: determine an angle interval in which a cosine angle between each log sample in the cluster and the centroid of the cluster is located; and determine a sample set corresponding to the log sample according to the angle interval in which the cosine angle between each log sample in the cluster and the centroid of the cluster is located.

[0106] Optionally, the non-stable sample set determining module can be specifically configured to: calculate a log sample quantity change rate of each sample set according to the number of log samples in each sample set; compare the log sample quantity change rate of each sample set with a preset change rate threshold; and determine that the sample set is a non-stable sample set if the log sample quantity change rate of the sample set is greater than the preset change rate threshold.

[0107] Optionally, the abnormal sample determining module can be specifically configured to: calculate the density of each log sample in the target sample set; sort all log samples in the target sample set according to the density of each log sample to obtain a log sample sequence; and determine the abnormal log sample in the log sample sequence.

[0108] Optionally, the abnormal sample determining module can be specifically configured to: obtain the cosine distance between each log sample; obtain Kn neighbor log samples of a target log sample to form a first neighbor set according to the cosine distance between each log sample, wherein the neighbor log sample is a log sample whose cosine distance with the target log sample meets a preset distance condition, and Kn is a positive integer; obtain Kn neighbor log samples of each log sample in the first neighbor set to form a second neighbor set; obtain the union set of the first neighbor set and the second neighbor set, and determine the total number of log samples contained in the union set; and determine the density of the target log sample according to the total number of log samples contained in the union set.

[0109] The apparatus provided by the embodiments of the present application can be used to execute the method in the above embodiments, and has similar implementation principles and technical effects, which will not be described here in detail.

[0110] It should be noted that the division of each module of the above apparatus is only a logical functional division, and all or part of the modules can be integrated into one physical entity, or can be physically separated. Moreover, all of the modules can be implemented in the form of software invoked by a processing element; all of the modules can be implemented in the form of hardware; or some modules are implemented in the form of software invoked by a processing element, and some modules are implemented in the form of hardware. For example, the cluster division module can be a separately established processing element, or can be integrated into a chip of the above apparatus, and in addition, the cluster division module can be stored in the form of program code in the memory of the above apparatus, and the functions of the cluster division module can be invoked and executed by a processing element of the above apparatus. The implementation of other modules is similar. Moreover, all or part of the modules can be integrated together, or can be independently implemented. The processing element herein can be an integrated circuit having a signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0111] Figure 6 A structural schematic diagram of an electronic device is provided in the embodiments of the present application. As shown in the figure, the electronic device 600 includes at least one processor 601, a memory 602, a bus 603 and a communication interface 604. The processor, the communication interface and the memory communicate with each other through the bus. The communication interface is used for communication with other devices. The communication interface includes a communication interface for data transmission and a display interface or an operation interface for human-computer interaction. The processor is used for executing computer-executed instructions stored in the memory, and can execute the related steps in the method described in the above embodiments. Figure 6

[0112] The processor can be a central processing unit, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the electronic device can be the same type of processors, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0113] The memory is used for storing computer-executed instructions. The memory can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0114] The embodiments also provide a computer-readable storage medium, and the computer-readable storage medium stores computer instructions. When at least one processor of the electronic device executes the computer instructions, the electronic device executes the semi-supervised density gradient monitoring Kmeans anomaly log retrieval method provided in the various embodiments described above.

[0115] The embodiments also provide a computer program product, and the program product includes computer instructions stored in a readable storage medium. At least one processor of the electronic device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to make the electronic device implement the semi-supervised density gradient monitoring Kmeans anomaly log retrieval method provided in the various embodiments described above.

[0116] ​In this application, "at least one" means one or more, "multiple" means two or more. "And / or" describes the association relationship of associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents the relationship between the front and rear associated objects as "or"; in the formula, the character " / ", represents the relationship between the front and rear associated objects as "division". "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c, can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0117] It can be understood that various numerical numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the serial number of the above processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0118] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are considered exemplary only, and the true scope and spirit of the application is indicated by the following claims.

[0119] It should be understood that the present application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the application is limited only by the claims that follow.

Claims

1. A semi-supervised density gradient-based monitoring Kmeans anomaly log retrieval method, characterized in that, The method comprises the following steps: obtaining a log sample set, and dividing log samples in the log sample set into K clusters, wherein the log sample set comprises at least K log samples, each cluster comprises at least one log sample as a centroid, and K is a positive integer; obtaining an edge log sample in the cluster, and determining a cosine angle between the centroid of the cluster and the edge log sample; constructing N sample sets according to the cosine angle between the centroid of the cluster and the edge log sample, wherein N is a positive integer; obtaining a cosine angle between each log sample in the cluster and the centroid of the cluster, determining a sample set corresponding to the log sample according to the cosine angle between each log sample in the cluster and the centroid of the cluster, and dividing the log sample into the sample set; determining a non-stable sample set from all sample sets according to the number of log samples in each sample set; combining log samples in all non-stable sample sets to form a target sample set, and determining an abnormal log sample in the target sample set; the step of determining the abnormal log sample in the target sample set comprises the following steps: calculating a density of each log sample in the target sample set; sorting all log samples in the target sample set according to the density of each log sample to obtain a log sample sequence; determining the abnormal log sample in the log sample sequence.

2. The method of claim 1, wherein, the step of dividing the log samples in the log sample set into K clusters comprises the following steps: selecting K log samples of different categories from the log sample set as K initial specified centroids; calculating a cosine distance between each log sample in the log sample set and the K initial specified centroids; dividing each log sample in the log sample set into different clusters according to the cosine distance between each log sample and each centroid to obtain K clusters.

3. The method of claim 1, wherein, the step of obtaining the edge log sample in the cluster comprises the following steps: calculating a cosine distance between each log sample in the cluster and the centroid of the cluster; obtaining a log sample with the largest cosine distance in the cluster as the edge log sample.

4. The method of claim 1, wherein, the step of determining the cosine angle between the centroid of the cluster and the edge log sample comprises the following steps: obtaining a module of the centroid of the cluster and a module of the edge log sample; calculating a cosine distance between the centroid of the cluster and the edge log sample; calculating the cosine angle between the centroid of the cluster and the edge log sample according to the module of the centroid of the cluster, the module of the edge log sample, the cosine distance between the centroid of the cluster and the edge log sample, and a pre-designed calculation formula.

5. The method of claim 1, wherein, the step of constructing N sample sets according to the cosine angle between the centroid of the cluster and the edge log sample comprises the following steps: dividing the cosine angle between the centroid of the cluster and the edge log sample into N equal parts to obtain N angle intervals; constructing a corresponding sample set for each angle interval to obtain N sample sets; correspondingly, the step of determining the sample set corresponding to the log sample according to the cosine angle between each log sample in the cluster and the centroid of the cluster comprises the following steps: determining an angle interval in which the cosine angle between each log sample in the cluster and the centroid of the cluster is located; determining the sample set corresponding to the log sample according to the angle interval in which the cosine angle between each log sample in the cluster and the centroid of the cluster is located.

6. The method of claim 1, wherein, The non-stable sample set is determined from all sample sets according to the number of log samples in each sample set, including: The log sample number change rate of each sample set is calculated according to the number of log samples in each sample set; The log sample number change rate of each sample set is compared with a preset change rate threshold; If the log sample number change rate of the sample set is greater than the preset change rate threshold, the sample set is determined as a non-stable sample set.

7. The method of claim 1, wherein, The density of each log sample in the target sample set is calculated, including: The cosine distance between each log sample is obtained; According to the cosine distance between each log sample, Kn neighbor log samples of the target log sample are obtained to form a first neighbor set, the neighbor log sample is a log sample whose cosine distance with the target log sample meets a preset distance condition, and Kn is a positive integer; Kn neighbor log samples of each log sample in the first neighbor set are obtained to form a second neighbor set; The union set of the first neighbor set and the second neighbor set is obtained, and the total number of log samples contained in the union set is determined; The density of the target log sample is determined according to the total number of log samples contained in the union set.

8. A semi-supervised based density gradient monitoring Kmeans anomaly log retrieval device, characterized by, It includes: A sample set acquisition module is configured to acquire a log sample set, divide log samples in the log sample set into K clusters, the log sample set includes at least K log samples, each cluster includes at least one log sample as a centroid, and K is a positive integer; A cosine angle determination module is configured to acquire an edge log sample in the cluster, and determine a cosine angle between the centroid of the cluster and the edge log sample; A sample set construction module is configured to construct N sample sets according to the cosine angle between the centroid of the cluster and the edge log sample, and N is a positive integer; A sample division module is configured to acquire a cosine angle between each log sample in the cluster and the centroid of the cluster, determine a sample set corresponding to the log sample according to the cosine angle between each log sample in the cluster and the centroid of the cluster, and divide the log sample into the sample set; A non-stable sample set determination module is configured to determine a non-stable sample set from all sample sets according to the number of log samples in each sample set; An abnormal sample determination module is configured to combine log samples in all non-stable sample sets to form a target sample set, and determine an abnormal log sample in the target sample set. The abnormal sample determination module is specifically configured to: Calculate the density of each log sample in the target sample set; Sort all log samples in the target sample set according to the density of each log sample to obtain a log sample sequence; Determine an abnormal log sample in the log sample sequence.

9. An electronic device, comprising: It includes: A processor and a memory connected with the processor in communication; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer-executable instructions, which, when executed by the processor, implement the method of any one of claims 1 to 7.

11. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images