System log classification method based on LSTM and Kmeans

Abstract

The invention relates to the technical field of data processing, in particular to a system log classification method based on LSTM and Kmeans. According to the method, a Kmeans clustering algorithm is mainly used, the clustering algorithm randomly extracts an initial centroid in an initialization stage, then the distance between each object and the initial centroid is calculated, and each object is distributed to a clustering center closest to the object. By applying the algorithm, the cost of manual classification can be reduced, and the efficiency can be remarkably improved under the condition of large data volume.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a system log classification method based on LSTM and Kmeans. Background Art

[0002] Clustering algorithms are designed to find similar data. Applications usually generate error logs during use. These error logs can be divided into different types. By analyzing these different types of error logs, targeted suggestions for solving the error can be given. Currently, manual error log classification is inefficient, and if the amount of data increases, the workload on personnel will increase further.

[0003] On the other hand, manual labeling of features of a large number of logs is extremely time-consuming and prone to errors. The bidirectional LSTM long short-term memory network can vectorize text data and label each text feature, saving the time of manual feature labeling. Summary of the invention

[0004] To address the above issues, this paper mainly uses the Kmeans clustering algorithm, which randomly extracts the initial centroid in the initialization phase, then calculates the distance between each object and the initial centroid, and assigns each object to the cluster center closest to it. The application of this algorithm can not only reduce the cost of manual classification, but also significantly improve efficiency in the case of large amounts of data.

[0005] The system log classification method based on LSTM and Kmeans includes the following steps: The feature acquisition module is composed of a bidirectional LSTM network, which captures the local and position invariant features in the text and obtains the features of contextual semantics. Kmeans log classification method is divided into: S1. Use the elbow rule to find appropriate K data points as the initial cluster centers in the target sample feature vector set, and randomly select K sample points as the initial centers; S2. Assign data points: For each data point, calculate its distance to the K cluster centers, which is the cosine distance between each feature vector extracted by the bidirectional LSTM and the cluster center vector. After the calculation is completed, assign it to the cluster center closest to it; S3. Update cluster center: For each cluster, calculate the average vector according to the vector cluster, and use the average vector as the new cluster center; S4. Repeat S2 and S3 until the stop condition is reached.

[0006] Kmeans log classification is completed by a distance matrix construction module, which is used to determine the cosine distance between each log feature vector in the target sample set, and construct a distance matrix based on the cosine distance.

[0007] The present invention provides the beneficial effects of the present invention: (1) LSTM is used to vectorize logs and extract text features, which improves the system's information organization capabilities and does not rely on manual information extraction.

[0008] (2) Using Kmeans to directly perform unsupervised learning not only saves labor costs, but also improves classification efficiency when the amount of data is large. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a schematic diagram of the LSTM structure; Figure 2 A schematic diagram of Kmeans clustering provided in an embodiment of the present application; Figure 3 A flow chart of the LSTM and Kmeans log classification method provided in the embodiment of the present application; Figure 4 Schematic diagram of finding K value for the elbow rule; Figure 5 This is a schematic diagram of the convergence effect of this application. DETAILED DESCRIPTION

[0010] System log classification method based on LSTM and Kmeans: First, the feature acquisition module is composed of a bidirectional LSTM network. The bidirectional LSTM network is a special recurrent neural network that can process the input sequence from front to back and from back to front at the same time. This network can more effectively capture the long-term dependencies in the sequence data and perform better for some tasks that require understanding contextual information. The design concept of the bidirectional LSTM is to make the feature data obtained at time t have information between the past and the future. Experiments have shown that the efficiency and performance of the bidirectional LSTM model for text feature extraction are better than those of a single LSTM structure model. The long short-term memory network is also widely used in text processing tasks. The network can effectively utilize the historical information of the sequence data through its internal hidden state, and the bidirectional LSTM network can capture the local and positional invariant features in the text from both the front and back directions. Therefore, the bidirectional LSTM network is selected to obtain the features of contextual semantics.

[0011] In a second aspect, the present application provides a Kmeans log classification method with random center selection, comprising: 1. Use the elbow rule to find appropriate K data points as the initial cluster centers in the target sample feature vector set, and randomly select K sample points as the initial centers. The elbow rule is a method for determining the optimal parameters. The optimal parameters are determined by drawing an image and observing the "elbow" position in the image. Specifically, when the "elbow" position in the image appears, it means that the improvement of system performance has leveled off. At this time, the parameters corresponding to this position can be selected as the optimal parameters.

[0012] 2. Assign data points: For each data point, calculate its distance to the K cluster centers, which is the cosine distance between each feature vector extracted by the bidirectional LSTM and the cluster center vector. After the calculation, assign it to the cluster center closest to it.

[0013] 3. Update cluster center: For each cluster, calculate the average vector based on the vector cluster and use the average vector as the new cluster center.

[0014] 4. Repeat 2 and 3 until the stop condition is reached.

[0015] A distance matrix construction module, used to determine the cosine distance between each log feature vector in the target sample set, and construct a distance matrix according to the cosine distance; The Kmeans log classification method provided in the present application obtains the feature vectors of each log in the target sample set, then constructs a vector matrix, randomly selects the center point as the initial centroid for Kmeans clustering according to the required category, and then uses Kmeans clustering.

[0016] The application of text vectorization and clustering analysis can improve the efficiency of information organization and retrieval. For scenarios that require manual text classification, the method in this article can save more costs, and the probability of error can be effectively controlled when the amount of data increases.

[0017] The technical solution of the present invention is further described in detail below in conjunction with specific implementation methods.

[0018] The dataset in this paper uses 3,000 log data from the GAIA intelligent operation and maintenance field for training. The data mainly comes from the MicroSS microservice simulation system developed by Cloud Wisdom and the authorized and strictly desensitized Cloud Wisdom user data.

[0019] LSTM structure reference Figure 1 The present invention provides a method for extracting features based on bidirectional LSTM and then using Kmeans to cluster log text. The specific operation steps are as follows: S1: The text data obtained is Y. When the length of the text data is L, the input is ; S2: Measure the data elements and obtain the measurement characteristics: Define the data type, record a log text data element, and establish a jxj matrix and the weight matrix .

[0020] The central elements are obtained from equation (1), and the diagonal elements are obtained from equation (2).

[0021] Where p(x,y) is the joint probability distribution of data elements and text data, and p(x) and p(y) are marginal probability distributions.

[0022] The central elements are obtained by equation (3), and the diagonal elements are obtained by equation (4).

[0023] Construct the metric feature H: in, The elements are obtained by equation (5), and the diagonal elements are obtained by equation (6).

[0024] S4: Text vectorization: The text data to be obtained , using a bidirectional LSTM network to extract features. Two LSTM layers in opposite directions are used to extract text features, defined as: In the formula, is the weight parameter, and is the state of the hidden layer in the forward and backward directions. Finally, each moment is combined with the output of the corresponding moment in the opposite direction as the final output: S5: Figure 2 A schematic diagram of Kmeans clustering provided in the embodiment of the present application is shown in FIG. Figure 2As shown in the figure, taking 20 sample logs from B1 to B20 as an example, the 20 log vectors can be divided into two groups. At this time, two initial centroids B1 and B10 are selected, and then the other log samples are clustered into two clusters based on these two initial centroids B1 and B10. Among them, log can be a term in the computer field. The application platform will generate logs when it is running; each line of log records the description of related operations such as date, time, user and action.

[0025] In this embodiment, the log sample may refer to the error log generated by the application mentioned above. Each error log corresponds to a type, so it is necessary to group the massive error logs by type, and classify the error logs of the same type into the same group. Among them, the target sample set may be composed of several error logs selected from the massive error logs.

[0026] Among them, the log samples can be converted into coordinate points in the form of vectorization, for example, log sample B1 is converted into coordinate point (Xb1, Yb1). By converting into coordinate point form, the distance between each log vector can be more intuitively seen, and based on the distance, it can be judged whether each log sample has similarity.

[0027] In this embodiment, after the log samples are converted into vector form, the cosine distance between the two log samples in the log sample set can be calculated, that is, there is a corresponding cosine distance between each two log vectors, and then these cosine distances are counted to construct a first distance matrix. Among them, the cosine distance is also called cosine similarity, which uses the cosine value of the angle between two vectors in the vector space as a measure of the size of the difference between two individuals.

[0028] S6: Figure 3 The flowchart of the LSTM and Kmeans log classification method provided in the embodiment of the present application is as follows: (1) First, randomly select 30% of the log data to form the dataset.

[0029] (2) Use bidirectional LSTM to extract feature vectors from the extracted data.

[0030] (3) Use the elbow rule to find a suitable K value, that is, find the position of the "elbow" in the K value and the sum of squared errors (SSE) graph. In this experiment, the K value found by the elbow rule is 3, as shown in Figure 4 shown.

[0031] (4) Randomly select K feature vectors as cluster centers according to the K value, calculate the cosine distance between each vector and the K centers, and assign each vector to the center cluster with the closest cosine distance.

[0032] (5) For each vector cluster obtained, calculate the average value of all data points and use the average value as the new cluster center. The final convergence effect is shown in the figure below: Figure 5 shown.

[0033] (6) Repeat steps 3 and 4 until convergence. The test results are shown in Table 1.

[0034] Table 1 Kmeans clustering and running schedule Glossary LSTM (Long-Short Term Memory): It is a time recurrent neural network specially designed to solve the long-term dependency problem of general recurrent neural networks.

[0035] Kmeans (k-means clustering algorithm): is an iterative clustering analysis algorithm. Its steps are: pre-divide the data into K groups, randomly select K objects as the initial cluster centers, and then calculate the distance between each object and each seed cluster center, and assign each object to the cluster center closest to it. The cluster centers and the objects assigned to them represent a cluster. Each time a sample is assigned, the cluster center of the cluster is recalculated based on the existing objects in the cluster. This process will be repeated until a certain termination condition is met. The termination condition can be that no (or the minimum number) of objects are reassigned to different clusters, no (or the minimum number) of cluster centers change again, and the sum of squared errors is locally minimized.

Claims

1. System log classification method based on LSTM and Kmeans, characterized by The steps include: The feature acquisition module is composed of a bidirectional LSTM network, which captures local and position invariant features in the text and obtains the features of contextual semantics. The Kmeans log classification is completed by a distance matrix construction module, which is used to determine the cosine distance between each log feature vector in the target sample set, and construct a distance matrix based on the cosine distance. Kmeans log classification method is divided into: S1. Use the elbow rule to find appropriate K data points as the initial cluster centers in the target sample feature vector set, and randomly select K sample points as the initial centers; S2. Assign data points: For each data point, calculate its distance to the K cluster centers, which is the cosine distance between each feature vector extracted by the bidirectional LSTM and the cluster center vector. After the calculation is completed, assign it to the cluster center closest to it; S3. Update cluster center: For each cluster, calculate the average vector according to the vector cluster, and use the average vector as the new cluster center; S4. Repeat S2 and S3 until the stop condition is reached.

Images