Abnormal log detection model construction method and abnormal log detection method and device

Abstract

The invention relates to the technical field of log analysis under a big data background, in particular to an abnormal log detection model construction method and an abnormal log detection method and device. The method comprises the steps of obtaining log data; determining a clustering center point based on the distance between event vectors in the log data, and clustering the log data to obtain normal logs and abnormal logs; and mining the normal logs by adopting an Apriori algorithm and HashMap to obtain a normal event rule base, and performing abnormal log detection by taking the normal event rule base as an abnormal log detection model. According to the method and the device, the log data are clustered, so that the log data can be quickly divided into the normal logs and the abnormal logs; and then, an Apriori algorithm is adopted to carry out association rule mining on the normal logs, so that a normal event rule base is constructed, and the abnormal logs can be rapidly and accurately detected through the normal event rule base.

Description

Technical Field

[0001] The present invention relates to the technical field of log analysis under the background of big data, and in particular to a method for constructing an abnormal log detection model and a method and device for abnormal log detection. Background Art

[0002] The system's operation log contains a large amount of information. By analyzing the log, we can prevent network attacks and locate the problem when the system fails. Nowadays, in the context of big data, the number of logs generated every day reaches PB or even EB. If the logs are analyzed only through manual analysis based on scripts or feature files or by keyword matching, the efficiency will be very low, and there will be a large number of meaningless suspected abnormal logs. At the same time, the analysis effect of unknown abnormalities will also be very unsatisfactory. Therefore, it is necessary to build an abnormal log detection system to analyze the logs, so as to effectively improve the efficiency while ensuring the accuracy. Summary of the invention

[0003] In view of this, the present invention provides an abnormal log detection model construction method and an abnormal log detection method and device to solve the problem of low efficiency of existing abnormal log detection methods.

[0004] In a first aspect, the present invention provides a method for constructing an abnormal log detection model, the method comprising: obtaining log data; determining cluster centers based on the distance between event vectors in the log data, clustering the log data to obtain normal logs and abnormal logs; using an Apriori algorithm and a HashMap to mine the normal logs to obtain a normal event rule base, and using the normal event rule base as an abnormal log detection model to perform abnormal log detection.

[0005] In the present invention, by clustering log data, log data can be quickly divided into normal logs and abnormal logs; then the Apriori algorithm is used to mine association rules for normal logs, thereby building a normal event rule base, thereby realizing rapid and accurate detection of abnormal logs through the normal event rule base. At the same time, when clustering, the cluster center point is determined based on the distance between event vectors in the log data, so that the initial cluster center point used in clustering is more reasonable, avoiding poor clustering results caused by random selection; and when the Apriori algorithm is used to determine the normal event rule base, the HashMap technology is used to effectively reduce the number of scans of the database, thereby improving the efficiency of obtaining frequent sets. In addition, both the clustering algorithm and the Apriori algorithm can be run in parallel on a big data platform. Thus, efficient implementation when building an abnormal log detection model can be guaranteed.

[0006] In an optional embodiment, after obtaining the log data, the method further includes: grouping the log data using window technology to construct a log sequence; extracting events in the log sequence; counting the number of occurrences of the event vector in each log sequence to obtain an event number vector; calculating the weight of the event using inverse document frequency; and calculating the event weight vector of the log sequence based on the number of occurrences and the weight.

[0007] In the present invention, since abnormal logs only account for a very small proportion of the total logs, in order to achieve a better division effect between normal logs and abnormal logs during clustering, the statistical method IDF is used to vectorize the log data during vectorization processing, so that abnormal events can obtain higher weights.

[0008] In an optional embodiment, cluster centers are determined based on the distances between event vectors in the log data, and the log data is clustered to obtain normal logs and abnormal logs, including: determining a first cluster center in the cluster center set based on the average value of the event vector; determining a second cluster center in the cluster center set based on the maximum value of the Euclidean distance between the event vector and the first cluster center; determining a third cluster center in the cluster center set based on the maximum value of the Euclidean distance between the event vector and the cluster center set, the Euclidean distance between the event vector and the cluster center set being determined by the minimum value of the Euclidean distance between the event vector and each cluster center; repeatedly calculating the Euclidean distance between the event vector and the cluster center set until a preset number of cluster centers are obtained; clustering the log data using a preset number of cluster centers and a k-means clustering algorithm to obtain normal logs and abnormal logs.

[0009] In the present invention, the first cluster center point is obtained by first calculating the average value of the event vector, and then the second cluster center point is determined by the Euclidean distance between each event vector and the first cluster center point. The distance calculation process is repeated to obtain multiple cluster center points as initial cluster center points, and clustering is performed based on the initial cluster center points. This can improve the quality of clustering, accelerate the convergence speed, enhance the robustness, and improve the interpretability of the clustering results.

[0010] In an optional implementation, the Apriori algorithm and HashMap are used to mine the normal log to obtain a normal event rule base, and the normal event rule base is used as an abnormal log detection model to perform abnormal log detection, including: using HashMap to construct an event set of normal logs; taking the event set as a first candidate item set, counting the support of item sets in the first candidate item set, and determining a first frequent item set; when the first frequent item set is not empty, based on the first frequent item set and HashMap, a second candidate item set is determined through a pruning operation; counting the support of item sets in the second candidate item set, and determining a second frequent item set; repeating the above steps of determining candidate item sets and frequent item sets until the determined frequent item set is empty, and constructing a normal event rule base of normal logs based on all determined frequent item sets.

[0011] In the present invention, by using HashMap when constructing a normal event rule base using the Apriori algorithm, the support calculation of multiple candidate sets can be processed in batches by scanning the database once, thereby reducing the overhead of multiple scans. By using pruning operations, unnecessary candidate set generation and database scanning are reduced. Therefore, the method effectively reduces the number of database scans and improves the efficiency of obtaining frequent sets.

[0012] In a second aspect, the present invention provides an abnormal log detection method based on the first aspect and the abnormal log detection model construction method described in any one of the first aspects, the method comprising: obtaining log data to be detected; using a normal event rule library to match event vectors in the log data to be detected, and determining whether the log to be detected is abnormal.

[0013] In an optional implementation, a normal event rule base is used to match event vectors in the log data to be detected to determine whether the log to be detected is abnormal, including: extracting events in the log data to be detected; calculating event weight vectors of the events as event vectors of the log data to be detected; matching based on the normal event rule base and the event vectors of the log data to be detected, and determining whether the log to be detected is abnormal based on the matching results.

[0014] In the present invention, the normal event rule base determined by the above-mentioned abnormal log detection model construction method is used as the abnormal log detection model for abnormal log detection. Compared with the method of analyzing the log based on scripts or feature files or by keyword matching, it can effectively improve the efficiency while ensuring the accuracy.

[0015] In a third aspect, the present invention provides a device for building an abnormal log detection model, the device comprising: a data acquisition module for acquiring log data; a clustering module for determining cluster centers based on the distance between event vectors in the log data, clustering the log data to obtain normal logs and abnormal logs; a model construction module for mining the normal logs using an Apriori algorithm and a HashMap to obtain a normal event rule base, and the normal event rule base is used as an abnormal log detection model to perform abnormal log detection.

[0016] In a fourth aspect, the present invention provides an abnormal log detection device based on the first aspect and the abnormal log detection model construction method described in any one of the first aspects, the device comprising: a data acquisition module to be detected, used to obtain the log data to be detected; a detection module, used to use a normal event rule library to match the event vector in the log data to be detected, and determine whether the log to be detected is abnormal.

[0017] In a fifth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the abnormal log detection model construction method of the above-mentioned first aspect or any corresponding embodiment thereof, or executes the abnormal log detection model construction method of the above-mentioned second aspect or any corresponding embodiment thereof by executing the computer instructions.

[0018] In a sixth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the abnormal log detection model construction method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the abnormal log detection model construction method of the above-mentioned second aspect or any corresponding embodiment thereof.

[0019] In the seventh aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the abnormal log detection model construction method of the above-mentioned first aspect or any corresponding embodiment, or execute the abnormal log detection model method of the above-mentioned second aspect or any corresponding embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1is a flow chart of a method for constructing an abnormal log detection model according to an embodiment of the present invention;

[0022] Figure 2 is a flow chart of another abnormal log detection method according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of an optimized k-means algorithm flow according to an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of an optimized Apriori algorithm flow according to an embodiment of the present invention;

[0025] Figure 5 is a structural block diagram of an abnormal log detection model building device according to an embodiment of the present invention;

[0026] Figure 6 is a structural block diagram of an abnormal log detection device according to an embodiment of the present invention;

[0027] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] According to an embodiment of the present invention, an embodiment of a method for building an abnormal log detection model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] In this embodiment, a method for constructing an abnormal log detection model is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 is a flowchart of a method for building an abnormal log detection model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0031] Step S101, obtaining log data. Specifically, the method for obtaining log data can be determined according to specific scenarios and requirements. For example, for small systems or applications, log data can be directly read from log files. For large distributed systems, log aggregation tools such as Fluentd or log management platforms such as Elasticsearch can be used to collect log data. In other implementations, log data can also be collected through a database or cloud platform that stores log data.

[0032] In addition, after obtaining the log data, the log data can also be preprocessed, such as cleaning, denoising, and removing isolated points. Among them, data cleaning can remove invalid, erroneous or incomplete log entries. Specifically, format verification, such as regular expression verification of log entry format, can be used to exclude entries that do not conform to the format; at the same time, the same log entries can be deleted to avoid repeated analysis; in addition, the log entries containing missing values ​​can be filled or deleted to ensure data integrity. Data denoising can remove noise in the log, such as irrelevant information or redundant information. Denoising can be achieved by keyword filtering or length restriction. Isolated points may be caused by system failures, temporary problems or abnormal situations, and are not representative. Therefore, isolated points in log data can be identified and excluded by frequency filtering and clustering algorithms. In addition, log data that is not related to anomaly detection can be removed during preprocessing, such as daily operation logs, debugging logs, information logs or duplicate logs, etc., and can be removed by setting filtering conditions. Therefore, through preprocessing, the influence of useless information on subsequent processing is avoided, and the efficiency of subsequent processing is improved.

[0033] Step S102: determining a cluster center point based on the distance between event vectors in the log data, clustering the log data to obtain normal logs and abnormal logs.

[0034] Among them, when clustering, clustering algorithms in related technologies can be used, such as the Kmeans algorithm, which is mainly used for unsupervised learning. Compared with hierarchical clustering, it can quickly process large amounts of data. The main idea is: given the K value and K initial cluster center points, each point (i.e., data record) is assigned to the cluster represented by the cluster center point closest to it. After all points are assigned, the center point of a cluster is recalculated based on the Euclidean distance of all points in a cluster (taking the average), and then the steps of assigning points and updating cluster center points are iterated until the change in the cluster center point is small, or the specified number of iterations is reached. In the Kmeans algorithm, the selection of K value and initial center point is very important.

[0035] However, the running result of the Kmeans clustering algorithm converges to a local optimum. In the related art, the Kmeans algorithm has a great deal of randomness in the selection of the initial cluster center point. If the selection is inappropriate, the result will be an undesirable local optimal solution. In view of this, the present embodiment determines the cluster center point by calculating the distance. When the cluster center point is determined by calculating the distance, the distance between the determined cluster center points can be made farther, thereby effectively improving the clustering effect. At the same time, before the distance calculation, the events in the log data can be vectorized for easy calculation.

[0036] In addition, it should be noted that the cluster center point determined here is used as the initial cluster center point during clustering. The log data can then be clustered based on the initial cluster center point and the Kmeans algorithm, and then classified based on the multiple clusters obtained after clustering, that is, the log data can be classified into normal logs and abnormal logs.

[0037] Step S103, the Apriori algorithm and HashMap are used to mine the normal log to obtain a normal event rule base, and the normal event rule base is used as an abnormal log detection model to perform abnormal log detection. Specifically, HashMap is a data structure, which belongs to the implementation class of the Map interface, is used to store key-value pairs, and can quickly find corresponding values ​​through keys. In this embodiment, in the key-value pairs stored in HashMap, the key is the event and the value is the number of occurrences of the event. In addition, HashMap has the advantage of fast search, that is, after using HashMap to store event sets, when performing data search, it has a time complexity of O(1) on average, so no matter how large the data set is, the search operation is very efficient. At the same time, HashMap stores data in the form of key-value pairs, and each key is unique, which makes it possible to quickly access the corresponding value through the key. In addition, HashMap can also be dynamically expanded. When the internal array capacity is insufficient, it automatically expands, usually twice the original capacity, which enables it to handle growing data sets.

[0038] Specifically, the Apriori algorithm is an association rule mining algorithm. It uses an iterative method of layer-by-layer search to find the relationship between item sets in the database to form rules. The process consists of connection (matrix-like operation) and pruning (removing unnecessary intermediate results). The concept of item set in this algorithm is a set of items. A set containing K items is a k-item set. The frequency of an item set is the number of transactions containing the item set, which is called the frequency of the item set. If an item set meets the minimum support, it is called a frequent item set.

[0039] However, when obtaining frequent sets, the traditional Apriori algorithm needs to scan the entire database in each iteration to calculate the support of each candidate set. As the number of candidate sets increases, the number of scans will also increase, resulting in low efficiency. In the context of big data, this means an excessive time burden. In view of this, in the process of executing the Apriori algorithm, this embodiment can use HashMap to store candidate sets, and then scan the database once to calculate the support of all items in the candidate set. Since HashMap provides the ability to quickly search and update, the support calculation of multiple candidate sets can be completed in one scan. This effectively reduces the number of scans of the database and improves the efficiency of obtaining frequent sets.

[0040] The abnormal log detection model construction method provided by the embodiment of the present invention can quickly divide the log data into normal logs and abnormal logs by clustering the log data; then the Apriori algorithm is used to mine the association rules of the normal logs, so as to build a normal event rule base, thereby realizing the rapid and accurate detection of abnormal logs through the normal event rule base. At the same time, when clustering, the cluster center point is determined based on the distance between the event vectors in the log data, so that the initial cluster center point used in clustering is more reasonable, avoiding the poor clustering results caused by random selection; and when the Apriori algorithm is used to determine the normal event rule base, the HashMap technology is used to effectively reduce the number of scans of the database, and improve the efficiency of obtaining frequent sets. In addition, the clustering algorithm and the Apriori algorithm can be run in parallel on the big data platform. Thereby, the efficient implementation of the abnormal log detection model construction can be guaranteed.

[0041] In this embodiment, a method for building an abnormal log detection model is provided, and the process includes the following steps:

[0042] Step S201, obtaining log data.

[0043] Step S202, using window technology to group the log data and construct a log sequence; extracting events in the log sequence; counting the number of occurrences of events in each log sequence to obtain an event number vector; using inverse document frequency to calculate the weight of the event; calculating the event weight vector of the log sequence based on the number of occurrences and the weight, the event number vector and the event weight vector constitute the event vector of the log sequence.

[0044] Specifically, when acquiring log data for analysis and constructing an abnormal log detection model, it is usually necessary to acquire a large amount of log data. In order to facilitate the analysis and processing of the log data, the acquired log data can be grouped first. For example, window technologies such as sliding windows or session windows can be used to group the log data. In this embodiment, the session window technology is used. In the session window technology, each session window has a unique identifier, and the identifier is used to mark different execution paths in some log data, so that the acquired log data can be grouped according to the identifier to construct a log sequence.

[0045] After constructing the log sequence, the events in each log sequence can be extracted and vectorized. When extracting events, each log sequence can be parsed, for example, the log sequence can be parsed into an event template with specific parameters, thereby realizing the extraction of events. In addition, since abnormal logs only account for a very small proportion of the total log volume, in order to achieve a better division effect between normal logs and abnormal logs during clustering, the statistical method IDF (inverse document frequency) can be used to vectorize the log data during vectorization.

[0046] When processing log vectorization, we first count the number of occurrences of each event in the log sequence to form an event count vector for each log sequence. In order to make the weight of abnormal events greater, we use the statistical method IDF (inverse document frequency) to calculate the weights of different events. The calculation formula is:

[0047] IDF(t)=log(N / (1+n t ))

[0048] Where N is the total number of log sequences, n t is the number of sequences in which event t occurs.

[0049] After the weight of the event is calculated using the above formula, the number of occurrences of the above statistical event and the corresponding event weight can be further processed by weighted sum to obtain the event weight vector of each log sequence.

[0050] Step S203: determining a cluster center point based on the distance between event vectors in the log data, clustering the log data to obtain normal logs and abnormal logs.

[0051] Specifically, the above step S203 includes:

[0052] Step S2031, determining the first cluster center point in the cluster center set based on the average value of the event vector; specifically, when constructing the log sequence X i(i=1,2,3,…,n), the log data can be expressed as S={X1,X2,…,X n}, for each log sequence, it can be represented by the obtained vector. For example, the event weight vector calculated above can be used to represent the log sequence X i ={x1,x2,…,x p} can be expressed as a p-dimensional vector. Then the average value of the vector is calculated to obtain the first cluster center point O1, and put it into the cluster center set O = {O1}.

[0053] Step S2032, based on the maximum value of the Euclidean distance between the event vector and the first cluster center point, determine the second cluster center point in the cluster center set; specifically, after calculating the first cluster center point, further calculate the Euclidean distance between the vector of each log sequence and the first cluster center point, and then use the event vector corresponding to the maximum Euclidean distance (i.e., the point farthest from the first cluster center point) as the second cluster center point.

[0054] Step S2033, based on the maximum value of the Euclidean distance between the event vector and the cluster center set, determine the third cluster center point in the cluster center set, the Euclidean distance between the event vector and the cluster center set is determined by the minimum value of the Euclidean distance between the event vector and each cluster center point. Specifically, when the cluster center set contains two cluster center points, the Euclidean distance between each remaining event vector and the two cluster center points can be calculated respectively, and the minimum value of the Euclidean distance is used as the distance between each event vector and the cluster center set. Afterwards, the event vector corresponding to the maximum value of the distance is selected as the third cluster center point according to the distance.

[0055] Step S2034, repeat the step of calculating the Euclidean distance between the event vector and the cluster center set until a preset number of cluster center points are obtained; specifically, when the cluster center set includes three cluster center points, the fourth cluster center point can be further determined by the method of the above step S2033, and then continue to repeat the above process until a preset number, such as k cluster center points, is determined.

[0056] Step S2035, clustering the log data using a preset number of cluster centers and the k-means clustering algorithm to obtain normal logs and abnormal logs. Specifically, after determining the preset number of cluster centers, use them as initial cluster centers, calculate the distance between the vectors in the log sequence and each initial cluster center, and then classify them and the nearest initial cluster center into one category. After completing the calculation of all log sequences to obtain a preset number of clusters, recalculate the mean of each cluster to obtain a new preset number of cluster centers. Then, based on the new cluster center, clustering is performed again by calculating the distance, and a new cluster is obtained, and the cluster center of the new cluster is calculated. After that, the distance between the new cluster center and the previously calculated cluster center is calculated. If the distance meets the convergence condition, the new cluster is output. If the distance does not meet the convergence condition, clustering continues until the distance between the new and old cluster centers meets the convergence condition. After the final preset number of clusters is obtained through clustering, each cluster is determined to be a normal log or an abnormal log, thereby classifying the multiple clusters into normal logs or abnormal logs.

[0057] Step S204, using Apriori algorithm and HashMap to mine the normal log to obtain a normal event rule base, and using the normal event rule base as an abnormal log detection model to perform abnormal log detection;

[0058] Specifically, the above step S204 includes:

[0059] Step S2041, using HashMap to construct an event set of a normal log; specifically, by scanning the normal log, using the events in the normal log as keys and the number of occurrences of the events as values, the event set is constructed.

[0060] Step S2042, taking the event set as the first candidate item set, counting the support of the item sets in the first candidate item set, and determining the first frequent item set; specifically, the event set constructed by HashMap can be directly taken as the first candidate item set, and then by scanning the first candidate item set, the support of each item set is calculated, and the item sets with support greater than the minimum support are added to the first frequent item set. If the obtained first frequent item set is empty, stop.

[0061] Step S2043, when the first frequent itemset is not empty, based on the first frequent itemset and HashMap, a second candidate itemset is determined through a pruning operation; specifically, when the first frequent itemset is not empty, the first frequent itemset is stored in a HashMap data structure, and then a pruning operation is performed to determine whether all its subsets are frequent, and if not, the corresponding subsets are deleted to obtain a second candidate itemset.

[0062] Step S2044: Count the support of the itemsets in the second candidate itemset to determine the second frequent itemset; specifically, by scanning the second candidate itemset, calculate the support of each itemset, and add the itemsets with support greater than the minimum support to the second frequent itemset. If the obtained second frequent itemset is empty, stop.

[0063] Step S2045, repeat the above steps of determining candidate item sets and frequent item sets until the determined frequent item sets are empty. Specifically, when the determined second frequent item sets are not empty, continue to use the above method to determine new candidate item sets and new frequent item sets until the determined frequent item sets are empty.

[0064] Step S2046, construct a normal event rule base of a normal log based on all determined frequent item sets. Specifically, after the generation of frequent item sets is stopped through the above steps, association rules are generated based on all generated frequent item sets to obtain a normal event rule base. When the association rules are generated, they can be determined by calculating the confidence level, which can be specifically implemented with reference to related technologies and will not be described in detail here.

[0065] In this embodiment, a method for detecting abnormal logs based on the method for building an abnormal log detection model in the above embodiment is also provided. The method comprises the following steps:

[0066] Step S301, obtaining log data to be detected. Specifically, the log data to be detected can be log data collected in real time in the system, or can be log data that needs to be judged whether it is abnormal.

[0067] Step S302, using a normal event rule base to match the event vector in the log data to be detected, to determine whether the log to be detected is abnormal. The event vector in the log data to be detected can be calculated using the event vector calculation method in the above embodiment.

[0068] Specifically, the above step S302 includes:

[0069] Step S3021, extracting events from the log data to be detected; specifically, if the amount of the log data to be detected is not large, the log sequence can be divided without performing the division, and the events from the log data to be detected can be directly extracted. When extracting, the log data to be detected can be parsed, for example, the log data to be detected can be parsed into an event template with specific parameters, thereby realizing the extraction of events.

[0070] Step S3022, calculate the event weight vector of the event as the event vector of the log data to be detected; specifically, when calculating the event vector, first count the number of occurrences of the extracted event, then use the statistical method IDF (inverse document frequency) to calculate the weights of different events, and then perform weighted sum and other processing based on the number of occurrences of the event and the corresponding event weight to obtain the event weight vector of the log data to be detected.

[0071] Step S3023, match the event vector of the log data to be detected based on the normal event rule base, and determine whether the log to be detected is abnormal based on the matching result. Specifically, during the matching, the event vector of the log data to be detected is matched with the association rule in the normal event rule base, and it is checked whether the log to be detected conforms to the normal event rule. If it conforms, the log to be detected is determined to be normal. If it does not conform, it can be directly judged as an abnormal log, or an expert knowledge base or manual judgment can be involved to improve accuracy and reduce false alarm rate. For the log data to be detected that is judged to be normal, it can be used to update the normal event rule base.

[0072] As a specific application example of the embodiment of the present invention, Figure 2 , Figure 3 as well as Figure 4 As shown, the abnormal log detection method can be implemented using the following process:

[0073] First, obtain a normal event rule base constructed using the abnormal log detection model construction method. The normal event rule base can be determined using the following process:

[0074] Step 1: Obtain log data and preprocess it to remove noise, isolated points, and log data irrelevant to the study.

[0075] Step 2: Build a log sequence and vectorize the data. Since each session window has a unique identifier, the logs can be grouped according to the identifier and built into a log sequence. Next, use the statistical method IDF to vectorize the log data as follows:

[0076] Step a1, generating an event count vector: extracting characteristic events from the log, counting the number of occurrences of each characteristic event in each log sequence, and forming an event count vector for each log sequence.

[0077] Step a1, generate event weight vector: In order to make the weight of abnormal events larger, the statistical method IDF (inverse document frequency) is used to calculate the weights of different events. The calculation formula is:

[0078] IDF(t)=log(N / (1+n t) )

[0079] Where N is the total number of log sequences, n t is the number of sequences in which event t occurs.

[0080] According to the weight calculated by the formula and the number of occurrences of the event, the event weight vector of each log sequence is generated.

[0081] Step 3: Classify the log data by k-means clustering: Use the optimized k-means algorithm to cluster the logs, as follows:

[0082] Step b1: Input log data set S = {X1, X2, …, X n}, where the log sequence X i ={x1,x2,…,x p} is a p-dimensional vector, and at the same time, input the k value.

[0083] Step b2, calculate the average value of the vectors in S in each dimension, so as to obtain the first cluster center point O1, and put it into the cluster center set O = {O1}.

[0084] Step b3: Select the point with the farthest Euclidean distance from O1 from the remaining points as O2, and add it to the cluster center set O = {O1, O2}.

[0085] Step b4, calculate the minimum value of each point and each point in O (ie O1, O2), which is used as the distance of each point to the cluster center set O, and then select the point with the farthest distance to O as O3. At this time, O = {O1, O2, O3}.

[0086] Step b5, repeat step b4, until k points are selected, which will become the initial cluster center points.

[0087] Step b6, calculate the distance between the log data in S and each center point, and classify them according to the minimum distance.

[0088] Step b7, recalculate the mean of each cluster to obtain a new cluster center point. If the distance between the new and old center points meets the convergence condition, the process ends; otherwise, repeat steps b6 and b7 based on the new cluster center.

[0089] Step b8: output k clusters and classify the logs into normal logs and abnormal logs.

[0090] Step 4: Use the association rule algorithm to mine normal logs and build a normal event rule base: Use the optimized Apriori algorithm to mine, as follows:

[0091] Step c1, scan the event database of normal logs, build a HashMap of events, denoted as M, with the key being the event and the value being the number of occurrences of the event.

[0092] Step c2: According to M, count the support of item set C1, add item sets with support greater than the minimum support to the frequent set, and obtain the frequent set L1.

[0093] Step c3: if the frequent set L1 is an empty set, the algorithm ends; otherwise, the frequent set L = L1.

[0094] Step c4, construct L i (i>=2) Frequent set: First, according to L i-1 The candidate item set (candidate frequent set) and the corresponding event set are obtained by adding M. Then, the support of each item in the candidate item set is counted one by one, and the item set with support greater than the minimum support is added to L. i .

[0095] Step c5, if L i If it is an empty set, then the frequent set L is output, the algorithm ends, and the normal event rule base is constructed; otherwise, L=L∪L i , repeat steps c4 to c5.

[0096] Specifically, after determining the normal event rule base, the following steps are used to perform anomaly detection:

[0097] Step 5, when a new log is collected, it is compared with the normal event rule base through correlation data mining and pattern matching to determine whether the log is abnormal. Once abnormal behavior is detected, an alarm is issued in time. Among them, those that can be successfully matched within a certain error range are judged as normal behavior, otherwise they are unknown patterns. For unknown patterns, they can be directly judged as abnormal; expert knowledge bases or manual judgments can also be involved to improve accuracy and reduce false alarm rates. For log data judged as normal patterns, it is necessary to update the normal event rule base.

[0098] In the present invention, the abnormal log detection method can run in parallel on a big data platform, so it can quickly build an abnormal log detection system for a large amount of log data. In the current environment where the amount of data is increasing, its accuracy and efficiency advantages will be more obvious, and the analysis of abnormal logs is of great significance for locating system operation problems, protecting information security and other fields.

[0099] In this embodiment, a device for building an abnormal log detection model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0100] This embodiment provides a device for constructing an abnormal log detection model. Figure 5 As shown, including:

[0101] A data acquisition module 51 is used to acquire log data;

[0102] A clustering module 52, configured to determine a cluster center point based on the distance between event vectors in the log data, cluster the log data, and obtain normal logs and abnormal logs;

[0103] The model building module 53 is used to mine the normal logs using the Apriori algorithm and HashMap to obtain a normal event rule base, and the normal event rule base is used as an abnormal log detection model to perform abnormal log detection.

[0104] In this embodiment, an abnormal log detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0105] This embodiment provides an abnormal log detection model construction method based on the above embodiment and an abnormal log detection device, such as Figure 6 As shown, including:

[0106] The to-be-detected data acquisition module 61 is used to acquire the to-be-detected log data;

[0107] The detection module 62 is used to match the event vector in the log data to be detected using the normal event rule base to determine whether the log to be detected is abnormal.

[0108] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0109] The embodiment of the present invention also provides a computer device having the above Figure 5 The abnormal log detection model building device shown and Figure 6 The abnormal log detection device shown.

[0110] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0111] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0112] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0113] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0115] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0116] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0117] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0118] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing an abnormal log detection model, characterized in that: The method comprises: Get log data; Determine a cluster center point based on the distance between event vectors in the log data, cluster the log data, and obtain normal logs and abnormal logs; The Apriori algorithm and HashMap are used to mine the normal logs to obtain a normal event rule base, and the normal event rule base is used as an abnormal log detection model to perform abnormal log detection.

2. The method according to claim 1, characterized in that: After obtaining the log data, the method further includes: The log data is grouped using window technology to construct a log sequence; Extract events from log sequences; Count the number of occurrences of the event vector in each log sequence to obtain the event count vector; The inverse document frequency is used to calculate the weight of events; An event weight vector of the log sequence is calculated according to the number of occurrences and the weight.

3. The method according to claim 1, characterized in that Determining a cluster center point based on the distance between event vectors in the log data, clustering the log data, and obtaining normal logs and abnormal logs, including: Determine a first cluster center point in the cluster center set based on the average value of the event vector; Determine a second cluster center point in the cluster center set based on the maximum value of the Euclidean distance between the event vector and the first cluster center point; Determining a third cluster center point in the cluster center set based on the maximum value of the Euclidean distance between the event vector and the cluster center set, wherein the Euclidean distance between the event vector and the cluster center set is determined by the minimum value of the Euclidean distance between the event vector and each cluster center point; Repeat the step of calculating the Euclidean distance between the event vector and the cluster center set until a preset number of cluster center points are obtained; The log data is clustered using a preset number of cluster centers and a k-means clustering algorithm to obtain normal logs and abnormal logs.

4. The method according to claim 1, characterized in that: The Apriori algorithm and HashMap are used to mine the normal logs to obtain a normal event rule base, including: Use HashMap to construct the event collection of normal logs; Taking the event set as the first candidate item set, counting the support of the item set in the first candidate item set, and determining the first frequent item set; When the first frequent itemset is not empty, the second candidate itemset is determined through pruning operation based on the first frequent itemset and HashMap; Count the support of the itemsets in the second candidate itemset to determine the second frequent itemset; Repeat the above steps of determining candidate item sets and frequent item sets until the determined frequent item sets are empty. A normal event rule base of normal logs is constructed based on all determined frequent item sets.

5. An abnormal log detection method based on the abnormal log detection model construction method according to any one of claims 1 to 4, characterized in that: The method comprises: Get the log data to be detected; The normal event rule base is used to match the event vectors in the log data to be tested to determine whether the log to be tested is abnormal.

6. The method according to claim 5, characterized in that The normal event rule base is used to match the event vectors in the log data to be tested to determine whether the log to be tested is abnormal, including: Extracting events from log data to be detected; Calculate the event weight vector of the event as the event vector of the log data to be detected; Matching is performed based on the normal event rule base and the event vector of the log data to be detected, and whether the log to be detected is abnormal is determined based on the matching result.

7. A device for constructing an abnormal log detection model, characterized in that: The device comprises: A data acquisition module is used to acquire log data; A clustering module, used to determine a cluster center point based on the distance between event vectors in the log data, cluster the log data, and obtain normal logs and abnormal logs; The model building module is used to mine the normal logs using the Apriori algorithm and HashMap to obtain a normal event rule base, and the normal event rule base is used as an abnormal log detection model to perform abnormal log detection.

8. An abnormal log detection device based on the abnormal log detection model construction method according to any one of claims 1 to 4, characterized in that: The device comprises: The module for acquiring data to be detected is used to acquire log data to be detected; The detection module is used to match the event vector in the log data to be detected using the normal event rule base to determine whether the log to be detected is abnormal.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the abnormal log detection model construction method described in any one of claims 1 to 4 or the abnormal log detection method described in claim 5 or 6 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the abnormal log detection model construction method described in any one of claims 1 to 4 or the abnormal log detection method described in claim 5 or 6.

Images