Self-correcting center kmeans anomaly log retrieval method based on semi-supervised

Abstract

The application provides a semi-supervised self-correcting center Kmeans abnormal log retrieval method and device, and relates to the field of big data. The method comprises the following steps: selecting K log samples of different categories from a log sample set as K initial specified centroids, wherein K is a positive integer; dividing the log samples in the log sample set into K clusters according to the K initial specified centroids; correcting the centroid of each cluster to obtain a corrected centroid of each cluster; obtaining the cosine distance between each log sample in each cluster and the corrected centroid of the cluster; and determining the abnormal log sample in each cluster according to the cosine distance. The method of the application selects K log samples of different categories as the K initial specified centroids in advance, corrects the K initial specified centroids, and then measures the distance between the log samples and the corrected centroids according to the cosine distance, so as to find the log samples far away from the centroids and improve the accuracy of abnormal log retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a self-correcting center Kmeans abnormal log retrieval method based on semi-supervision. BACKGROUND

[0002] Log is a term in the computer field, and an application platform generates logs when operating; each line of log records the date, time, user, and description of related operations. The application platform generates normal logs when operating normally, and abnormal logs when problems occur. These abnormal logs play an important role in preventing and analyzing potential problems of the application platform, facilitating the discovery of new problem categories, being conducive to the formation of measures to avoid such problems, and being conducive to the maintenance and management of the application platform.

[0003] In the prior art, abnormal log retrieval mainly relies on a clustering algorithm, and text clustering is used to cluster logs to find out log samples not belonging to any cluster as abnormal logs.

[0004] However, the existing clustering method cannot finely select samples not belonging to any cluster, and it is also difficult to accurately find abnormal points, resulting in poor clustering effect and low accuracy of abnormal log retrieval. SUMMARY

[0005] The present application provides a self-correcting center Kmeans abnormal log retrieval method and device based on semi-supervision, to solve the problem of low accuracy of abnormal log retrieval by the existing clustering algorithm.

[0006] In a first aspect, the present application provides a self-correcting center Kmeans abnormal log retrieval method based on semi-supervision, comprising:

[0007] Selecting K log samples of different categories from a log sample set as K initial specified centroids, K being a positive integer;

[0008] Dividing log samples in the log sample set into K clusters according to the K initial specified centroids;

[0009] Correcting the centroid of each cluster to obtain a corrected centroid of each cluster;

[0010] Obtaining the cosine distance between each log sample in each cluster and the corrected centroid of the cluster;

[0011] According to the cosine distance, determining the log sample existing in each cluster.

[0012] In a second aspect, the present application provides a self-correcting center Kmeans abnormal log retrieval device based on semi-supervision, comprising:

[0013] a centroid designation module, configured to select K log samples of different categories from the log sample set as the K initial designated centroids;

[0014] a cluster division module, configured to divide the log samples in the log sample set into K clusters according to the K initial designated centroids;

[0015] a centroid correction module, configured to correct the centroid of each cluster to obtain a corrected centroid of each cluster;

[0016] a distance acquisition module, configured to acquire a cosine distance between each log sample in each cluster and the corrected centroid of the cluster;

[0017] an abnormal sample determination module, configured to determine the log sample that is abnormal in each cluster according to the cosine distance.

[0018] In a third aspect, the present application provides an electronic device, comprising a processor and a memory connected with the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method as described above.

[0019] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions; the computer execution instructions are executed by a processor to implement the method as described above.

[0020] In a fifth aspect, the present application provides a computer program product, comprising a computer program; the computer program is executed by a processor to implement the method as described above.

[0021] The self-correcting center Kmeans abnormal log retrieval method and device based on semi-supervision provided by the present application can improve the retrieval accuracy of abnormal logs by pre-selecting K log samples of different categories as the K initial designated centroids, correcting the K initial designated centroids, measuring the distance between the log samples and the corrected centroids by cosine distance, and finding the log samples far away from the centroids as the abnormal log samples. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0023] Figure 1 a schematic diagram of the clustering algorithm provided by the embodiments of the present application;

[0024] Figure 2 a flowchart of the self-correcting center Kmeans abnormal log retrieval method based on semi-supervision provided by the embodiments of the present application;

[0025] Figure 3 A log sample schematic diagram provided for the embodiment of the application;

[0026] Figure 4 A flowchart of the semi-supervised self-correcting center Kmeans abnormal log retrieval method provided for the embodiment of the application;

[0027] Figure 5 A structure schematic diagram of the semi-supervised self-correcting center Kmeans abnormal log retrieval device provided for the embodiment of the application;

[0028] Figure 6 A structure schematic diagram of the electronic device provided for the embodiment of the application.

[0029] The specific embodiments of the application have been shown through the above-described drawings, and will be described in more detail hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the application in any way, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0030] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is only one of the specific embodiments consistent with the present application. Rather, it is merely an example of apparatus and methods in accordance with some aspects of the present application, as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0032] It should be noted that the semi-supervised self-correcting center Kmeans abnormal log retrieval method and device provided by the present application can be used in the field of big data, and can also be used in any field other than big data. The application field of the semi-supervised self-correcting center Kmeans abnormal log retrieval method and device provided by the present application is not limited.

[0033] K-means clustering algorithm: is a kind of iterative solution clustering analysis algorithm, the steps are, pre-data is divided into K groups, then randomly select K objects as initial cluster centers, then calculate the distance between each object and each seed cluster center, assign each object to the nearest cluster center. The cluster center and the object assigned to it represent a cluster. Each sample is assigned, and the cluster center of the cluster is recalculated according to the existing objects in the cluster. This process will be repeated until a certain termination condition is met. The termination condition can be that no (or a minimum number of) objects are reassigned to different clusters.

[0034] Abnormal log detection aims to find logs that do not belong to the regular category, which is generally a minority compared to the overall amount of logs. Because of the minority of special and alternative, it plays an important role in preventing potential problems of the analysis application platform, facilitating the discovery of new problem categories, and helping to form evasive measures for such problems, which is conducive to the maintenance and governance of the application platform. The traditional kmeans clustering algorithm randomly extracts initial centroids in the initialization stage, which can lead to the extraction of too dispersed or concentrated centroids, which are not particularly uniform, thereby slowing down the convergence speed of clustering and resulting in poor clustering effect. Especially under the condition of massive text data, this defect will be more prominent. After clustering, whether the sample log is an abnormal sample point is determined by calculating the distance between the sample log and the cluster center. This judgment method is not good at selecting sample points that do not belong to any cluster, and it is difficult to accurately find abnormal points by only using the distance index. In summary, the traditional kmeans clustering algorithm has the following problems: ①The selection of the centroid in the initialization stage has randomness, and the initial centroid is too close or dispersed, which is not uniform enough, and cannot balance the selection of the centroid of all categories. ②It is not accurate to determine whether a sample point is an abnormal point by only using the distance between the sample point and the cluster center (and it is not easy to define the distance), and it is difficult to determine the abnormal point if the overall log sample is relatively dense and the log sample is far from the center.

[0035] To solve the above problems, the embodiment of the present application provides a self-correcting center Kmeans abnormal log retrieval method and device based on semi-supervised, by pre-selecting K log samples of different categories as K initial specified centroids, by correcting the K initial specified centroids, and by measuring the distance between the log sample and the corrected centroid by cosine distance, the log sample far from the centroid is found as the abnormal log sample, which can improve the retrieval accuracy of abnormal logs.

[0036] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic diagram of the clustering algorithm provided in the embodiments of this application, as shown below. Figure 1 As shown, taking 20 sample logs (X1 to X20) as an example, X1 to X8 are clustered into one cluster, and X9 to X20 are clustered into another. If there is an outlier that does not belong to either of the above two clusters, then the sample log represented by that outlier can be identified as an anomalous log. Here, "log" can be a term from the computer science field; application platforms generate logs during operation; each log line records a description of the date, time, user, and action.

[0038] Figure 2 This is a flowchart illustrating a semi-supervised, self-correcting center K-means anomaly log retrieval method provided in this application embodiment. This method can be applied to retrieving logs generated by an application platform to discover anomaly logs. For example, taking an electronic device as the executing entity of this method... Figure 2 As shown, the method may specifically include the following steps: Step S201, select K log samples of different categories from the log sample set as the K centroids to be initialized.

[0039] In this embodiment, the log sample set can be a collection of logs generated by the application platform during runtime. For example, it can be a collection of all logs generated by the application platform within a certain time period. It is understood that the log sample set contains a large number of logs, and the longer the time period, the more logs are collected from the application platform, resulting in a larger log capacity for the log sample set. The log sample set typically includes a large number of regular logs and a small number of irregular logs (i.e., abnormal logs). These abnormal logs usually account for a small proportion, making it difficult to accurately locate abnormal logs from the log sample set, which can easily lead to inaccurate location. It should be noted that abnormal logs often represent potential problems in the application platform. By analyzing abnormal logs, potential problems in the application platform can be discovered in a timely manner, allowing for proactive avoidance measures.

[0040] In this embodiment, K is a positive integer, and K different categories of log samples can be manually selected from the log sample set. The log sample set can include various categories of log samples, such as user logs, program logs, operating system logs, etc.

[0041] Figure 3 A log sample diagram provided by the embodiment of the present application is shown in Figure 3 As shown, by digitizing the log samples, each log sample is replaced by a coordinate point, and K1, K2 and K3 can be selected as the initialized specified K centroids.

[0042] In other embodiments, the K log samples of different categories can be specified by manual means. Specifically, a target instruction is obtained, the target instruction at least including the indication information of selecting the K log samples of different categories; and the K log samples of different categories are selected from the log sample set according to the target instruction.

[0043] The target instruction is manually input, and the electronic device determines the K log samples of different categories from the log sample set as the K centroids based on the target instruction after receiving the target instruction. It should be noted that the kmeans clustering algorithm randomly selects K objects as the centroids, which is not representative. The embodiment can avoid the problem that the initialized centroid cannot represent all log categories by manually specifying the K initialized centroids, thereby further improving the accuracy of subsequent abnormal log retrieval.

[0044] In step S202, the log samples in the log sample set are divided into K clusters according to the K initialized specified centroids.

[0045] In the embodiment, each centroid is taken as the center of each category cluster, and then the other log samples in the log sample set are clustered to form K category clusters. Specifically, if a log sample is close to one of the centroids, the log sample will be divided into the category cluster with the centroid as the center.

[0046] For example, continuing to refer to Figure 3 Taking K1, K2 and K3 as the three initialized specified centroids as an example, the log samples K4, K5 and K6 are close to the centroid K1, and K1, K4, K5 and K6 are divided into the same category cluster, and the center of the category cluster is the centroid K1. The log samples K7 and K8 are close to the centroid K2, and K2, K7 and K8 are divided into another category cluster, and the center of the category cluster is the centroid K2.

[0047] In step S203, the centroid of each cluster is corrected to obtain the corrected centroid of each cluster.

[0048] In the embodiment, the initialization of the specified centroid can be inaccurate, and the centroid can be corrected to be more accurate. Specifically, after the K clusters are divided, the centroid of each cluster can be determined based on the log samples in the cluster (including the centroid of the cluster and the log samples divided into the cluster), i.e., the centroid of each cluster is corrected to select a new centroid. It can be understood that when the centroid of each cluster is corrected, the corrected centroid of the cluster can be the original specified centroid, or the corrected centroid of the cluster can no longer be the original specified centroid, but can be another log sample in the cluster.

[0049] In step S204, the cosine distance between each log sample in each cluster and the corrected centroid of the cluster is obtained.

[0050] In the embodiment, each log sample can be converted into a vector form, and the corrected centroid of the cluster can also be converted into a vector form. Then, the cosine distance is calculated based on the vector form of the log sample and the corrected centroid. For example, continuing to refer to the above example, the log sample K4 and the corrected centroid K1 are converted into vectors, and the cosine distance between the log sample K4 and the corrected centroid K1 is calculated. Figure 3 For example, the cosine distance between the log sample K4 and the corrected centroid K1 is smaller than the cosine distance between the log sample K5 and the corrected centroid K1, i.e., the cosine distance can represent the distance between the log samples.

[0051] In step S205, the log sample with an anomaly in each cluster is determined based on the cosine distance.

[0052] In the embodiment, each log sample can be sorted based on the cosine distance between the log sample and the corrected centroid. For example, if the cosine distance between the log sample A and the corrected centroid is D1, the cosine distance between the log sample B and the corrected centroid is D2, and the cosine distance between the log sample C and the corrected centroid is D3, and D1 is greater than D2 which is greater than D3, the log sample A is arranged at the front, the log sample B is arranged in the middle, and the log sample C is arranged at the end. At this time, the log sample C arranged at the end can be selected as the log sample with an anomaly. Further, in other embodiments, a threshold value can be set, for example, the threshold value D4. When D1 is less than the threshold value D4, and D2 and D3 are greater than the threshold value D4, the log sample B and the log sample C are both selected as the log sample with an anomaly.

[0053] In the embodiment, K log samples of different categories are selected as K centroids, the logs are clustered, the K centroids are corrected, and finally the cosine distance between the corrected centroid and each log sample can be used to more accurately determine the log sample with an anomaly, so that the abnormal log can be found in the massive logs.

[0054] In some embodiments, the step S202 can be implemented by the following steps: calculating the cosine distance between each log sample in the log sample set and the K initial specified centroids; and dividing each log sample in the log sample set into different clusters according to the cosine distance between each log sample and each centroid, to obtain K clusters.

[0055] In the embodiment, the log sample set can include a plurality of log samples, and the cosine distance between each log sample and each centroid can be calculated. For example, the cosine distances between a log sample A and K1, K2 and K3 are calculated, to obtain corresponding cosine distances K11, K12 and K13. When K11 is less than K12 which is less than K13, the log sample A is divided into the same cluster with the centroid K1. It should be noted that each centroid has a corresponding cluster, i.e., K centroids have K clusters.

[0056] Further, a corresponding cluster can be created for each centroid, and whether the log sample satisfies the preset nearest distance principle is determined according to the size of the cosine distance between each log sample and each centroid. If the log sample satisfies the nearest distance principle, the log sample is divided into the cluster corresponding to the centroid, to obtain K clusters.

[0057] In the embodiment, each initial specified centroid can create a corresponding empty cluster, and then the log sample is divided into the cluster, to obtain K clusters with log samples. Specifically, based on the nearest distance principle, if the cosine distance between the log sample and a centroid K1 in the K centroids is the smallest, the log sample is divided into the cluster corresponding to the centroid K1.

[0058] The embodiments of the present application improve the original kmeans clustering algorithm, specify K log samples of different categories as K centroids, and re-cluster the log samples in the log sample set based on the K centroids, which avoids the problem that the randomly selected centroid cannot represent all log categories, and improves the clustering effect.

[0059] In some embodiments, the step S203 can be implemented by the following steps: determining, for each cluster, a near neighbor sample set of the centroid of the cluster, the near neighbor sample set including at least one log sample; and correcting the centroid of the cluster according to the near neighbor sample set of the centroid of the cluster, to obtain a corrected centroid of the cluster.

[0060] In the embodiment, when the specified centroid of each cluster is modified, the log samples adjacent to the centroid can be found first to form a near neighbor sample set (the near neighbor sample set includes the centroid and the log samples adjacent to the centroid), and then a new centroid is selected from the near neighbor sample set as a modified centroid to replace the originally specified centroid, so as to achieve the purpose of modifying the centroid. The log sample in the center of the near neighbor sample set can be found as the modified centroid, or a suitable target log sample can be selected by manual operation as the modified centroid.

[0061] Further, when the near neighbor sample set is determined, the following steps can be used: for each cluster, the cosine distance between each log sample in the cluster and the centroid of the cluster is obtained; each log sample in the cluster is sorted according to the size of the cosine distance between the log sample and the centroid of the cluster, to obtain a first sorting result; according to the first sorting result, Kn target log samples are selected from the log samples in the cluster to form a target sample set, the cosine distance between the target log sample and the centroid of the cluster is less than a target distance value, and Kn is a positive integer; the neighbor sample set of each target log sample in the cluster is determined, and at least one log sample exists in the neighbor sample set; the intersection of the target sample set and the neighbor sample set is obtained as the near neighbor sample set.

[0062] In the embodiment, in each cluster, each log sample is first sorted according to the cosine distance with the initially specified centroid to obtain a first sorting result (the first sorting result can be obtained by sorting according to the cosine distance from near to far). For example, in a cluster, log samples K4, K5 and K6 are included, the initially specified centroid of the cluster is K1, the cosine distance between the log sample K4 and K1 is K44, the cosine distance between the log sample K5 and K1 is K55, and the cosine distance between the log sample K6 and K1 is K66. If K44 is less than K55 and K66, the first sorting result is {K4, K5, K6}.

[0063] The target distance value can be set, and if K44 and K55 are less than the target distance value, K4 and K5 can be target log samples and form a target sample set. Each target log sample can have a corresponding neighbor sample set, and the intersection of the target sample set and the neighbor sample set corresponding to each log sample is taken as the near neighbor sample set.

[0064] For example, the neighbor sample set corresponding to the log sample K4 can be K4A, the neighbor sample set corresponding to the log sample K5 can be K5A, and the log samples in K4A, K5A and the target sample set are combined to obtain the near neighbor sample set.

[0065] Further, when determining the neighbor sample set of each target log sample in the cluster, the following steps can be specifically implemented: for each target log sample in the cluster, the cosine distance between each log sample in the cluster and the target log sample is obtained; each log sample in the cluster is sorted according to the size of the cosine distance between the log sample and the target log sample, to obtain a second sorting result; and Kn log samples are selected from the cluster according to the second sorting result to form the neighbor sample set of the target log sample.

[0066] In this embodiment, taking target log samples K4 and K5 as examples, the cluster can further include log samples K6, K8 and K9, the cosine distance K64 between log sample K6 and K4 can be calculated, the cosine distance K65 between log sample K6 and K5 can be calculated, the cosine distance K84 between log sample K8 and K4 can be calculated, the cosine distance K85 between log sample K8 and K5 can be calculated, the cosine distance K94 between log sample K9 and K4 can be calculated, the cosine distance K95 between log sample K9 and K5 can be calculated, and then sorting is performed to obtain a second sorting result.

[0067] Each target log sample can have a corresponding second sorting result (cosine distance from near to far), for example, the second sorting result corresponding to target log sample K4 is {K6, K8, K9}, and the second sorting result corresponding to target log sample K5 is {K8, K9, K6}. At this time, based on the second sorting result, Kn can take a value of 1 or 2, that is, 1 or 2 log samples with a high ranking in the second sorting result are selected to form the neighbor sample set of the target log sample.

[0068] The embodiments of the present application can sort log samples by using cosine distance, finally construct a neighbor sample set, and correct the original initialization specified centroid based on the neighbor sample set, so that the abnormal outlier log can be more accurately found, and the accuracy of abnormal log detection can be improved.

[0069] In some embodiments, when obtaining the corrected centroid of the cluster, the following steps can be specifically implemented: the center of all log samples in the neighbor sample set is calculated as a first corrected centroid; for each cluster, the neighbor sample set of the first corrected centroid of the cluster is determined; the center of all log samples in the neighbor sample set of the first corrected centroid of the cluster is calculated as a second corrected centroid; for each cluster, the neighbor sample set of the second corrected centroid of the cluster is determined; the center of all log samples in the neighbor sample set of the second corrected centroid of the cluster is calculated to obtain a third corrected centroid as the corrected centroid of the cluster.

[0070] In the embodiment, for each cluster, a set dataN1 consisting of the second order Kn nearest neighbors of the original initialization designated centroid of the cluster is found, and then the center of all samples in dataN1 is calculated to obtain a first modified centroid C1. Again, a set dataN2 consisting of the second order Kn nearest neighbors of the first modified centroid C1 in the cluster is found, and then the center of all samples in dataN2 is calculated to obtain a second modified centroid C2. Finally, a set dataN3 consisting of the second order Kn nearest neighbors of the second modified centroid C2 in the cluster is found, and then the center of all samples in dataN3 is calculated to obtain a third modified centroid C3 as the final modified centroid of the cluster.

[0071] wherein the number of the second order Kn nearest neighbors of X is: a set neighbour1 consisting of Kn nearest neighbor samples of the sample point X (cosine distance), a set neighbour2 consisting of Kn nearest neighbor samples of each sample point in neighbour1, and a set neighbour_total consisting of the union of neighbour1 and neighbour2, and the number of sample points in neighbour_total is the number of the second order Kn nearest neighbors of X.

[0072] The embodiment of the application can ensure that the modified centroid represents all log types and avoid the problem that kmeans initialization randomly designates a plurality of different log categories as the centroid, which cannot accurately represent all log categories. Meanwhile, the cosine distance is used to measure the distance between the log sample and the cluster center, which can more accurately find abnormal log samples.

[0073] On the basis of the above embodiment, in some embodiments, the step S205 can be implemented by the following steps: according to the cosine distance, sorting each log sample in each cluster to obtain a sample sequence, and the cosine distance of the log sample located at the front end of the sample sequence is smaller than the cosine distance of the log sample located at the rear end of the sample sequence; obtaining the log sample located at the rear end of the sample sequence; and determining the abnormal log sample from the log sample located at the rear end of the sample sequence.

[0074] In the embodiment, the cosine distance of each log sample in each cluster to the third modified centroid C3 (i.e. the final modified centroid) of the cluster is calculated, the cosine distance of the log sample to the modified centroid C3 of the cluster is taken as the scoring standard, the sample sequence List of the cluster is sorted according to the cosine distance from near to far, and the sample at the end of the sample sequence List is taken as the abnormal log sample according to the set threshold.

[0075] The log sample and the cosine distance of the corrected centroid C3 are used to screen out the log sample far away from the corrected centroid C3 as an abnormal log sample, so that the accuracy of abnormal log sample searching can be improved.

[0076] Exemplarily, Figure 4 A flowchart of a semi-supervised self-corrected centroid Kmeans abnormal log searching method and device provided by the embodiment of the present application is shown in FIG. 4, which includes the following steps: Figure 4 S401, log data is acquired. In step S402, K centroids are manually specified, the cosine distance of each log sample in the log data and the K centroids is calculated, the sample is divided into the nearest centroid, and finally K clusters are obtained. In step S403, for each cluster, a set dataN1 composed of the second-order Kn neighbors of the cluster centroid is found, and then the center of all log samples in dataN1 is calculated to obtain a corrected centroid C1. In step S404, for each cluster, a set dataN2 composed of the second-order Kn neighbors of the first corrected centroid C1 in the cluster is found, and then the center of all log samples in dataN2 is calculated to obtain a second corrected centroid C2. In step S405, for each cluster, a set dataN3 composed of the second-order Kn neighbors of the second corrected centroid C2 in the cluster is found, and then the center of all samples in dataN3 is calculated to obtain a third corrected centroid C3. In step S406, the cosine distance of each log sample in the cluster to the corrected centroid C3 of the cluster is calculated, the cosine distance of the log sample to the corrected centroid C3 of the cluster is taken as a scoring standard, the sample sequence List of the cluster is obtained by sorting the cosine distance from near to far, and the log sample at the end of the sample sequence List is taken as an abnormal log sample according to a set threshold.

[0077] In the embodiment, computer execution instructions can be constructed to implement the scheme. Exemplarily, the computer instructions can be as shown in Table 1.

[0078]

[0079] Table 1

[0080] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0081] Figure 5 A structure diagram of a semi-supervised self-corrected centroid Kmeans abnormal log searching device provided by the embodiment of the present application is shown in FIG. 4, which includes the following steps: Figure 5As shown, the abnormal log retrieval device 500 can specifically include a centroid specifying module 510, a cluster division module 520, a centroid correction module 530, a distance acquisition module 540, and an abnormal sample determination module 550. The centroid specifying module 510 is configured to select K log samples of different categories from the log sample set as the K initial specified centroids. The cluster division module 520 is configured to divide the log samples in the log sample set into K clusters according to the K initial specified centroids. The centroid correction module 530 is configured to correct the centroid of each cluster to obtain a corrected centroid of each cluster. The distance acquisition module 540 is configured to acquire the cosine distance between each log sample in each cluster and the corrected centroid of the cluster. The abnormal sample determination module 550 is configured to determine the log sample that is abnormal in each cluster according to the cosine distance.

[0082] Optionally, the cluster division module can be specifically configured to: calculate the cosine distance between each log sample in the log sample set and the K initial specified centroids; and divide each log sample in the log sample set into different clusters according to the cosine distance between each log sample and each centroid to obtain the K clusters.

[0083] Optionally, the cluster division module can be specifically configured to: create a corresponding cluster for each centroid; determine whether each log sample satisfies a preset nearest distance principle according to the size of the cosine distance between the log sample and each centroid; and divide the log sample into the cluster corresponding to the centroid if the log sample satisfies the nearest distance principle to obtain the K clusters.

[0084] Optionally, the centroid correction module can be specifically configured to: for each cluster, determine a near neighbor sample set of the centroid of the cluster, the near neighbor sample set including at least one log sample; and correct the centroid of the cluster according to the near neighbor sample set of the centroid of the cluster to obtain a corrected centroid of the cluster.

[0085] Optionally, the centroid correction module can be specifically configured to: for each cluster, acquire the cosine distance between each log sample in the cluster and the centroid of the cluster; sort each log sample in the cluster according to the size of the cosine distance between the log sample and the centroid of the cluster to obtain a first sorting result; select K target log samples from the log samples in the cluster according to the first sorting result to form a target sample set, the target log samples having a cosine distance smaller than a target distance value with the centroid of the cluster, and K being a positive integer; determine a neighbor sample set of each target log sample in the cluster, the neighbor sample set including at least one log sample; and acquire the intersection of the target sample set and the neighbor sample set as the near neighbor sample set.

[0086] Optionally, the centroid correction module can be specifically configured to: for each target log sample in the cluster, obtain the cosine distance between each log sample in the cluster and the target log sample; sort each log sample in the cluster according to the size of the cosine distance between the log sample and the target log sample, to obtain a second sorting result; and select Kn log samples from the cluster according to the second sorting result to form the neighbor sample set of the target log sample.

[0087] Optionally, the centroid correction module can be specifically configured to: calculate the center of all log samples in the neighbor sample set as a first modified centroid; determine, for each cluster, the neighbor sample set of the first modified centroid of the cluster; calculate the center of all log samples in the neighbor sample set of the first modified centroid of the cluster as a second modified centroid; determine, for each cluster, the neighbor sample set of the second modified centroid of the cluster; and calculate the center of all log samples in the neighbor sample set of the second modified centroid of the cluster to obtain a third modified centroid as the modified centroid of the cluster.

[0088] Optionally, the abnormal sample determination module can be specifically configured to: sort each log sample in each cluster according to the cosine distance to obtain a sample sequence, wherein the cosine distance of a log sample located at the front end of the sample sequence is smaller than the cosine distance of a log sample located at the rear end of the sample sequence; obtain the log sample located at the rear end of the sample sequence; and determine the abnormal log sample from the log sample located at the rear end of the sample sequence.

[0089] Optionally, the centroid designation module can be specifically configured to: obtain a target instruction, the target instruction including at least indication information of selecting K log samples of different categories; and select K log samples of different categories from the log sample set according to the target instruction.

[0090] The apparatus provided by the embodiments of the present application can be used to execute the method in the above embodiments, and has similar implementation principles and technical effects, which will not be described here in detail.

[0091] It should be noted that the division of the various modules of the above apparatus is only a logical functional division, and all or part of them can be integrated into a physical entity, or physically separated. These modules can all be implemented in the form of software invoked by a processing element; all in the form of hardware; or some modules are implemented in the form of software invoked by a processing element, and some modules are implemented in the form of hardware. For example, the cluster division module can be a separately established processing element, or it can be integrated into a chip of the above apparatus, in addition, it can also be stored in the form of program code in the memory of the above apparatus, and the function of the above cluster division module is invoked and executed by a processing element of the above apparatus. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or can be independently implemented. The processing element here can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0092] Figure 6 The structure schematic diagram of the electronic device provided by the embodiment of the present application is shown in the figure. Figure 6 As shown in the figure, the electronic device 600 includes at least one processor 601, a memory 602, a bus 603 and a communication interface 604. Among them: the processor, the communication interface and the memory complete the communication among each other through the bus. The communication interface is used for communication with other devices. The communication interface includes a communication interface for data transmission and a display interface or an operation interface for human-computer interaction. The processor is used for executing the computer execution instructions stored in the memory, and can execute the related steps in the method described in the above embodiment.

[0093] Among them, the processor can be a central processing unit, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the electronic device can be the same type of processor, such as one or more CPUs; or different types of processors, such as one or more CPUs and one or more ASICs.

[0094] The memory is used to store computer execution instructions. The memory can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0095] The embodiment also provides a computer readable storage medium, and the computer readable storage medium stores computer instructions. When at least one processor of the electronic device executes the computer instructions, the electronic device executes the method provided by the various embodiments described above.

[0096] The embodiment also provides a computer program product, which includes computer instructions stored in a readable storage medium. At least one processor of an electronic device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to enable the electronic device to implement the method provided by the various embodiments described above.

[0097] In the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it; in the formula, the character " / ", represents an "division" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c, can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0098] It can be understood that the various numbers involved in the embodiments of the present application are only distinguished for the convenience of description, and are not used to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the serial number of the above processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0099] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application be limited only by the scope of the claims, which will encompass any variations that are within the spirit and scope of the present application. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0100] It should be understood that the present application is not limited to the precise construction that has been described and shown in the accompanying drawings, and that various modifications and changes can be effected therein by those skilled in the art without departing from the scope of the application. The scope of the application should be limited only by the appended claims.

Claims

1. A semi-supervised based self-correcting center Kmeans anomaly log retrieval method, characterized in that, The method comprises the following steps: selecting K log samples of different categories from the log sample set as K initial specified centroids, K being a positive integer; dividing the log samples in the log sample set into K clusters according to the K initial specified centroids; correcting the centroid of each cluster to obtain a corrected centroid of each cluster; obtaining the cosine distance between each log sample in each cluster and the corrected centroid of the cluster; determining the log sample that exists abnormally in each cluster according to the cosine distance; the correction of the centroid of each cluster to obtain the corrected centroid of each cluster comprises: for each cluster, determining a near neighbor sample set of the centroid of the cluster, the near neighbor sample set comprising at least one log sample; calculating the center of all log samples in the near neighbor sample set as a first corrected centroid; for each cluster, determining a near neighbor sample set of the first corrected centroid of the cluster; calculating the center of all log samples in the near neighbor sample set of the first corrected centroid as a second corrected centroid; for each cluster, determining a near neighbor sample set of the second corrected centroid of the cluster; calculating the center of all log samples in the near neighbor sample set of the second corrected centroid to obtain a third corrected centroid as the corrected centroid of the cluster.

2. The method of claim 1, wherein, the division of the log samples in the log sample set into K clusters according to the K initial specified centroids comprises: calculating the cosine distance between each log sample in the log sample set and the K initial specified centroids; dividing each log sample in the log sample set into different clusters according to the cosine distance between each log sample and each centroid to obtain K clusters.

3. The method of claim 2, wherein, the division of each log sample in the log sample set into different clusters according to the cosine distance between each log sample and each centroid to obtain K clusters comprises: creating a corresponding cluster for each centroid; determining whether the log sample satisfies a preset nearest distance principle according to the size of the cosine distance between each log sample and each centroid; if the log sample satisfies the nearest distance principle, dividing the log sample into the cluster corresponding to the centroid to obtain K clusters.

4. The method of claim 1, wherein, the determination of the near neighbor sample set of the centroid of each cluster comprises: for each cluster, obtaining the cosine distance between each log sample in the cluster and the centroid of the cluster; sorting each log sample in the cluster according to the size of the cosine distance between the log sample and the centroid of the cluster to obtain a first sorting result; selecting K target log samples from the log samples in the cluster to form a target sample set according to the first sorting result, the cosine distance between the target log sample and the centroid of the cluster being less than a target distance value, K being a positive integer; determining a neighbor sample set of each target log sample in the cluster, the neighbor sample set comprising at least one log sample; obtaining the intersection of the target sample set and the neighbor sample set as the near neighbor sample set.

5. The method of claim 4, wherein, the determination of the neighbor sample set of each target log sample in the cluster comprises: for each target log sample in the cluster, obtaining the cosine distance between each log sample in the cluster and the target log sample. According to the size of the cosine distance between each log sample in the cluster and the target log sample, each log sample in the cluster is sorted to obtain a second sorting result; According to the second sorting result, Kn log samples are selected from the cluster to form a neighbor sample set of the target log sample.

6. The method of claim 1, wherein, The determining of the abnormal log sample in each cluster according to the cosine distance comprises: According to the cosine distance, each log sample in each cluster is sorted to obtain a sample sequence, and the cosine distance of the log sample at the front end of the sample sequence is smaller than the cosine distance of the log sample at the rear end of the sample sequence; The log sample at the rear end of the sample sequence is obtained; The abnormal log sample is determined in the log sample at the rear end of the sample sequence.

7. The method of claim 1, wherein, The selecting of the K log samples of different categories from the log sample set comprises: Obtaining a target instruction, the target instruction at least including indication information of selecting K log samples of different categories; According to the target instruction, K log samples of different categories are selected from the log sample set.

8. A semi-supervised based self-correcting center Kmeans anomaly log retrieval device, characterized by, Comprise: The centroid specifying module is used for selecting K log samples of different categories from the log sample set as the K initial specified centroids; The cluster division module is used for dividing the log samples in the log sample set into K clusters according to the K initial specified centroids; The centroid correction module is used for correcting the centroid of each cluster to obtain the corrected centroid of each cluster; The distance acquisition module is used for acquiring the cosine distance between each log sample in each cluster and the corrected centroid of the cluster; The abnormal sample determination module is used for determining the abnormal log sample in each cluster according to the cosine distance; The centroid correction module is specifically used for: For each cluster, the neighbor sample set of the centroid of the cluster is determined, and the neighbor sample set includes at least one log sample; The center of all log samples in the neighbor sample set is calculated as a first corrected centroid; For each cluster, the neighbor sample set of the first corrected centroid of the cluster is determined; The center of all log samples in the neighbor sample set of the first corrected centroid of the cluster is calculated as a second corrected centroid; For each cluster, the neighbor sample set of the second corrected centroid of the cluster is determined; The center of all log samples in the neighbor sample set of the second corrected centroid of the cluster is calculated to obtain a third corrected centroid as the corrected centroid of the cluster.

9. An electronic device, comprising: Comprise: A processor and a memory connected with the processor in communication; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to realize the method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to realize the method in any one of claims 1 to 7.

11. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

Images