Log anomaly detection method, system, medium and device based on machine learning

Abstract

The invention discloses a log anomaly detection method, system, medium and device based on machine learning, which belong to the technical field of log anomaly detection. The method comprises: obtaining multiple system logs; determining and marking the attack category of each system log; dividing all system logs according to the attack category to obtain multiple types of system logs; clustering each type of system log respectively to obtain multiple cluster centers of each type of system log as representative logs of each type of system log; extracting the access path feature value vector of each representative log; training a constructed log anomaly detection model by using the access path feature value vectors of all representative logs, and obtaining a trained log anomaly detection model after the training is completed; performing anomaly detection on the system log by using the trained log anomaly detection model, thereby improving the accuracy of system log detection and solving the problem of low accuracy of current system log detection.

Description

Technical Field

[0001] The present invention relates to the technical field of log anomaly detection, and in particular to a log anomaly detection method, system, medium and device based on machine learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] System anomaly detection is an essential and important task for detecting system failures, debugging machine performance, and maintaining system security. System log files can be extracted from all computer systems, which record the operating status of the system and the events that occurred at different times. They are important data for fault analysis, performance testing, and system anomaly detection.

[0004] In the related art, a large amount of system log data of known attack types is used to train a fault diagnosis model. Afterwards, the trained fault diagnosis model is used to perform anomaly detection on the collected system logs to determine whether the system is under attack.

[0005] However, not all system log data of known attack types are typical. When the fault diagnosis model is trained using system log data of all known attack types, the training accuracy and training efficiency of the fault diagnosis model are low. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a log anomaly detection method, system, medium and device based on machine learning, which improves the training precision and training efficiency of the log anomaly detection model, thereby improving the accuracy of log anomaly detection.

[0007] To achieve the above object, the present invention adopts the following technical solution:

[0008] First, a log anomaly detection method based on machine learning is proposed, including:

[0009] Get multiple system logs;

[0010] Identify and label the attack category of each system log;

[0011] Divide all system logs according to attack categories to obtain multiple types of system logs;

[0012] Cluster each type of system log respectively, and obtain multiple cluster centers of each type of system log as representative logs of each type of system log;

[0013] Extract the access path feature value vector of each representative log;

[0014] The constructed log anomaly detection model is trained using all access path feature value vectors representing the logs. After the training is completed, a trained log anomaly detection model is obtained.

[0015] Use the trained log anomaly detection model to perform anomaly detection on system logs.

[0016] Further, the request information of each system log is extracted;

[0017] Match the extracted request information with the regular expression of each attack category;

[0018] An attack category corresponding to a regular expression matching the request information is selected as the attack category of the log to which the request information belongs.

[0019] Furthermore, multiple clustering centers of a certain type of log are determined by performing multiple rounds of clustering on each type of system log; for each type of system log, the shortest distance between each system log sample in that type of system log and the existing clustering center of that type of system log is calculated during each round of clustering; the fitness of each system log is calculated based on the shortest distance between each system log and the existing clustering center; the proportion of the fitness of each system log to the total fitness of that type of system log is calculated as the individual probability of each system log; based on the individual probability of each system log, the cumulative probability of each system log is calculated and determined; based on the cumulative probability of each system log, the clustering center of this round of clustering is determined.

[0020] Furthermore, the total fitness of a certain type of system log is the sum of the fitness of all system logs in the type of system log.

[0021] Furthermore, the process of extracting the access path feature value vector representing the log includes:

[0022] Determine the access path of the representative log;

[0023] Divide the access path representing the log into words to obtain multiple word groups;

[0024] Calculate the importance of each phrase;

[0025] A phrase whose importance is greater than a set value is selected as the access path feature value vector of the representative log.

[0026] Furthermore, the term frequency and inverse document frequency of each phrase are calculated;

[0027] Multiply the term frequency and inverse document frequency of each phrase to obtain the importance of each phrase.

[0028] Furthermore, the log anomaly detection model is constructed using a single-classification support vector machine.

[0029] Secondly, a log anomaly detection system based on machine learning is proposed, including:

[0030] Log acquisition module, used to obtain multiple system logs;

[0031] An attack category determination module is used to determine and mark the attack category of each system log;

[0032] The log classification module is used to classify all system logs according to attack categories to obtain multiple types of system logs;

[0033] A representative log extraction module is used to cluster each type of system log respectively, and obtain multiple cluster centers of each type of system log as representative logs of each type of system log;

[0034] A feature extraction module is used to extract a feature value vector of an access path representing each log;

[0035] The model training module is used to train the constructed log anomaly detection model by using the access path feature value vectors representing all logs, and after the training is completed, a trained log anomaly detection model is obtained;

[0036] The anomaly detection module is used to perform anomaly detection on system logs using the trained log anomaly detection model.

[0037] In a third aspect, a computer device is provided, the device comprising:

[0038] a processor adapted to execute a computer program;

[0039] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the log anomaly detection method based on machine learning proposed in the first aspect is implemented.

[0040] In a fourth aspect, a computer-readable storage medium is proposed, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the machine learning-based log anomaly detection method proposed in the first aspect.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention proposes a log anomaly detection method, system, medium and device based on machine learning. After obtaining multiple system logs, the method determines and marks the attack category of each system log, and divides all system logs based on the determined attack category; for each type of divided system log, a representative log is determined by a clustering method; then, an access path feature value vector of each representative log is extracted; the access path feature value vectors of all representative logs are used to train a constructed log anomaly detection model. Since the selected representative log is the system log that best represents a type of attack category, when the access path feature value vectors of the representative log are used to train the constructed log anomaly detection model, the training accuracy and training efficiency of the log anomaly detection model are improved; when the trained log anomaly detection model is used to perform anomaly detection on the system log, the accuracy of log anomaly detection is improved.

[0043] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0045] Figure 1 A flow chart of a log anomaly detection method based on machine learning disclosed in an embodiment;

[0046] Figure 2 A flowchart of obtaining a trained log anomaly detection model disclosed in the embodiment;

[0047] Figure 3 A flowchart of actively detecting a system disclosed in an embodiment;

[0048] Figure 4 The processing flow for each type of system log disclosed in the embodiment;

[0049] Figure 5 A flowchart of the log anomaly detection model training disclosed in the embodiment;

[0050] Figure 6 The structure diagram of the application system of the log anomaly detection method based on machine learning disclosed in the embodiment is as follows. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0054] Example 1

[0055] Terminology explanation:

[0056] 1. One-Class SVM (OCSVM):

[0057] Single-class support vector machine is an unsupervised learning algorithm mainly used for anomaly detection or outlier identification. It is not used for traditional binary classification tasks, but only considers one type of data (positive samples), and the goal is to find a hyperplane that maximizes the interval between this class and all other potential input points (called "support vector"). When new data points fall inside this hyperplane or close to the edge, they are considered to belong to the known category, while data points far away from this area are considered abnormal. The core idea of ​​​​OCSVM is to construct a decision boundary so that normal data points are distributed outside the boundary as much as possible, while maximizing the "degrees of freedom" within the boundary. It can effectively identify unknown abnormal situations.

[0058] 2. TF-IDF algorithm (term frequency-inverse document frequency):

[0059] The TF-IDF algorithm is a method for measuring the importance of words in a text, especially for information retrieval and text mining tasks. The calculation process of TF-IDF can be divided into two main parts: term frequency (TF) and inverse document frequency (IDF).

[0060] 3. Bagging algorithm:

[0061] The Bagging algorithm is an algorithm based on ensemble learning and is a widely used machine learning technique. The full name of Bagging is Bootstrap Aggregating, and its idea is to improve the generalization ability of the overall learner by combining the results of many independent learners.

[0062] 4. N-gram algorithm:

[0063] The N-gram algorithm is a phrase algorithm that can split a text into several consecutive sequences of n words and count the frequency of these sequences in the text. Here n is a positive integer representing the number of words.

[0064] The log anomaly detection method based on machine learning disclosed in this embodiment is applied to Figure 6 In the system shown, the system includes a data display layer, a business logic layer, a basic service layer, a data storage layer and a data collection layer.

[0065] The data display layer includes detection logs and diagnostic tasks. The detection logs are used to actively send data collection or passively receive server data collection through timing or periodicity; the diagnostic tasks are used to process and store abnormal events and related data (freeze frames and extended data) and provide fault information.

[0066] The business logic layer includes data collection, alarm services and analysis services. Data collection is used to collect various indicators, events and logs from containers and systems running on the server, and collect them into indicator information for data storage; the alarm service is used to issue warnings and notifications based on the diagnosis results, and notify operation and maintenance through pop-up reminders and information push notifications. The analysis service is used to train a log anomaly detection model through the logs when the system is attacked.

[0067] The basic service layer includes system management, log management and other services. System management includes user management, permission management, menu management and module management. Log management is used to ensure that there are enough logs for troubleshooting and security analysis, while avoiding the negative impact of excessive expansion of log files on the system.

[0068] The data storage layer includes a database (MySQL) that is used to store and collect unprocessed logs, processed logs, and anomaly detection data.

[0069] The data collection layer includes I / O and network communication, and is used to collect multi-dimensional indicator data in a variety of ways, including: actively sending data collection or passively receiving server data collection at a scheduled or periodic interval; extracting monitoring indicators through third-party APIs; monitoring indicators through StatsD and Kafka consumer services; and sending indicators to various data storage, services, and message queues.

[0070] Next, the log anomaly detection method based on machine learning disclosed in this embodiment is described in detail.

[0071] The log anomaly detection method based on machine learning disclosed in this embodiment is as follows: Figure 1-Figure 5 As shown, including:

[0072] S1: Get multiple system logs;

[0073] This embodiment actively detects the system; obtains the system log when the system actively detects.

[0074] like Figure 3 As shown in the figure, the process of actively detecting the system includes:

[0075] S11: Automatically detect the system network, detect the approximate location of the network fault, and give corresponding prompt information. The process of automatic detection by the system includes: firstly, perform workstation network detection to determine the connectivity of the website URL or IP address; if it can be connected, prompt the user of the possible reasons for not being able to log in to the website; if it cannot be connected, then detect the workstation to determine the location of the network fault and give corresponding fault location prompt.

[0076] S12: Detect the system workstation, including network interface detection, IP address detection, network card address detection and network configuration detection, check the workstation's own information, and eliminate the workstation's own faults.

[0077] This embodiment uses log collection tools to obtain logs when the system is attacked, where the log collection tools include Logstash, Flume, Fluentd, and Kafka. Logstash is an open source data collection engine with real-time pipeline capabilities. It can dynamically unify data from different data sources and standardize the data to the destination of your choice. Flume is a highly available, highly reliable, distributed system for massive log collection, aggregation, and transmission. Fluentd is an open source data collector for unifying and simplifying the collection, processing, and output of log data. Kafka is a distributed stream processing platform that can process real-time data pipelines and streaming applications.

[0078] S2: Determine and label the attack category of each system log.

[0079] The process of determining the attack category of each system log in this embodiment includes:

[0080] Extract request information from each system log;

[0081] Match the extracted request information with the regular expression of each attack category;

[0082] An attack category corresponding to a regular expression matching the request information is selected as the attack category of the log to which the request information belongs.

[0083] This embodiment identifies the status code and start and end time of the log; determines whether the log is normal or attacked according to the status code; and counts the number of normal and attacked logs.

[0084] The types of attacks on the system include xss (cross-site scripting attack), comm (command execution attack), sqlinj (SQL injection attack), file (file inclusion attack) and webshell (script Trojan horse).

[0085] This embodiment predefines regular expressions of the above-mentioned several attack categories, matches the request information (request) of the log with the regular expression of each attack category, and if the match is successful, determines that the attack category is the attack category of the log, and marks the log.

[0086] S3: Divide all system logs according to attack categories to obtain multiple types of system logs.

[0087] After determining the attack category of each system log, this embodiment classifies logs belonging to the same attack category as one category to obtain multiple categories of system logs.

[0088] S4: Cluster each type of system log respectively to obtain multiple cluster centers of each type of system log as representative logs of each type of system log.

[0089] This embodiment determines multiple cluster centers of a certain type of log by performing multiple rounds of clustering on each type of system log; for each type of system log, when performing each round of clustering, the shortest distance between each system log sample in the type of system log and the existing cluster center of the type of system log is calculated; the fitness of each system log is calculated based on the shortest distance between each system log and the existing cluster center; the proportion of the fitness of each system log to the total fitness of the type of system log is calculated as the individual probability of each system log; based on the individual probability of each system log, the cumulative probability of each system log is calculated and determined; based on the cumulative probability of each system log, the cluster center of this round of clustering is determined.

[0090] Specifically, this embodiment uses an improved clustering analysis method (K-means) to cluster each type of system log.

[0091] The detailed steps of clustering a certain type of system logs using the improved cluster analysis method (K-means) include:

[0092] S41: First randomly select a system log in this type of system log as the first cluster center .

[0093] S42: Calculate the shortest distance D(x) between each system log in the system log category and the current existing cluster center (i.e., the distance between the system log and the nearest cluster center); the larger the D(x), the greater the probability of being selected as the cluster center;

[0094] S43: Use the roulette wheel method to select new cluster centers;

[0095] S44: Repeat steps S42-S43 until k cluster centers are selected.

[0096] The process of selecting new cluster centers using the roulette wheel method includes:

[0097] S431: Calculate the fitness of each system log in this type of system log , , where n is the data sample size;

[0098] Fitness: In the roulette wheel method, each individual corresponds to a specific area on the wheel, the size of which is determined by the fitness function of the individual. The higher the fitness, the wider the area where the individual is located, and the greater the probability of being selected.

[0099] S432: Calculate the individual probability of each system log;

[0100]

[0101] In the formula, is the individual probability of system log i;

[0102] is the fitness of system log i;

[0103] N is the number of system logs in a type of system log;

[0104] It represents the total fitness of a type of system log, and N is the number of system logs in a type of system log.

[0105] S433: Calculate the cumulative probability of each system log ;

[0106]

[0107] is the individual probability of system log j.

[0108] S434: Generate a uniformly distributed pseudo-random number r in the interval [0,1];

[0109] S435: If Select the first system log; otherwise select system log k such that: Established;

[0110] S436: Repeat steps S434 and S435 for a total of n times.

[0111] The improved cluster analysis method (K-means) proposed in this embodiment reduces the risk of falling into a local optimal solution in the selection of cluster centers, and has better clustering effect and faster convergence speed; in target detection applications, the improved k-means changes the way the initial cluster centers are generated, and uses the roulette wheel method to select the cluster centers of system logs. The gap between the initial anchor frames is increased so that the clusters cover different system logs more evenly. In this way, the gap between the initial anchor frames is increased, which helps to adapt to more diverse system logs, thereby improving the matching degree and accuracy; if the spacing between the anchor frames is too small, multiple anchor frames may predict the same system log, reducing the accuracy of the prediction. The anchor frame will be more adapted to the overall distribution, and a multi-scale anchor frame with better matching degree will be obtained, thereby improving the detection performance of the model.

[0112] S5: Extract the access path feature value vector of each representative log.

[0113] The process of extracting the access path feature value vector representing the log includes:

[0114] Determine the access path of the representative log;

[0115] Divide the access path representing the log into words to obtain multiple word groups;

[0116] Calculate the importance of each phrase;

[0117] A phrase whose importance is greater than a set value is selected as the access path feature value vector of the representative log.

[0118] Among them, the word frequency and inverse document frequency of each phrase are calculated;

[0119] Multiply the term frequency and inverse document frequency of each phrase to obtain the importance of each phrase.

[0120] In order to improve feature space mapping, reduce the difficulty of dimensionality, and improve extraction efficiency, this embodiment first uses N-grams to extract phrases, and then uses the TF-IDF algorithm to calculate the importance of phrases in order to determine the importance of words in the access path.

[0121] N-grams are used to divide the access path representing the log into words. The process of obtaining multiple word groups includes:

[0122] The access path representing the log is preprocessed by removing stop words, extracting stems, and other preprocessing operations to obtain a word sequence suitable for n-gram statistical preprocessing.

[0123] According to the set n value, the preprocessed word sequence is split into several phrases by sliding windows. The process of splitting the word sequence by sliding windows includes: starting from the beginning of the word sequence each time, taking n consecutive words as a phrase (n-gram), then moving back one word position, and continuing to take the next n-gram until the entire word sequence is traversed.

[0124] Count the number of times each n-gram appears in the access path and calculate its frequency. The frequency of each phrase is equal to the number of times the phrase appears divided by the number of all phrases in the access path.

[0125] The statistically obtained n-grams and their frequencies are stored for later use. Usually, data structures such as hash tables or dictionaries can be used to achieve efficient storage and query.

[0126] The process of calculating the importance of a phrase using the TF-IDF algorithm includes:

[0127] Calculate the term frequency TF of each phrase. The term frequency refers to the frequency of a phrase appearing in a document. It indicates the importance of a phrase in a document and is calculated using the following formula:

[0128]

[0129] Standard formula:

[0130]

[0131] in:

[0132] w is a phrase.

[0133] d is the document containing the phrase w.

[0134] is the number of occurrences of phrase i in the representative log, and j refers to all phrases in a representative log.

[0135] It represents the number of phrases after the log is split by n-gram.

[0136] Calculate the inverse document frequency (IDF) of each phrase. IDF measures the importance of a phrase in the entire document collection. The larger the IDF value, the less common the phrase is in the entire document collection (a representative log of all network failures in the log), and therefore the more important it is in the document. IDF is usually calculated using the following formula:

[0137]

[0138] Standard formula:

[0139]

[0140] in:

[0141] w is a phrase.

[0142] D is a type of representative log that represents all network failures.

[0143] : is the number of representative logs containing phrase i.

[0144] S6: Use all access path feature value vectors representing the logs to train the constructed log anomaly detection model. After the training is completed, a trained log anomaly detection model is obtained; use the trained log anomaly detection model to perform anomaly detection on the system log.

[0145] The log anomaly detection model in this embodiment is constructed using a single-classification support vector machine.

[0146] The single-class support vector product maps the data to a high-dimensional feature space through a kernel function, and uses a hyperplane to separate the majority data from the novel data. The most commonly used is the Gaussian kernel function, also known as the RBF (Radial Basis Function) kernel.

[0147] The single-class support vector product solves the problem of optimizing the objective function, which is the following formula:

[0148]

[0149] Constrained by:

[0150]

[0151] In the formula, is the weight vector coefficient, for The second-order norm of is the penalty coefficient, is a slack variable, i is a certain log, m is the total number of logs, is the ith eigenvector.

[0152] The specific implementation steps of single-class support vector product are:

[0153] Calculate the signed distance from sample X to the splitting hyperplane;

[0154] Calculate the soft boundary of sample X;

[0155] After calculating the soft boundary of sample X, return the label (label is whether it is abnormal);

[0156] Get the estimator training parameters;

[0157] Returns the label of sample X.

[0158] The single-class support vector product is used to illustrate that the single-class support vector machine is trained through log data to obtain a classification model, and finally the classification model is used to distinguish normal or abnormal data.

[0159] When training the constructed log anomaly detection model using the access path feature value vectors representing all logs, the grid search method is used to select the optimal model hyperparameters of the log anomaly detection model. By traversing the given parameter grid, each set of parameters is cross-validated, and the optimal parameter combination is selected according to the scoring criteria.

[0160] Cross-validation is to evaluate the performance of the model by dividing the dataset into multiple subsets, cyclically using one part as the test set and the other parts as the training set; then the confusion matrix is ​​performed to measure the accuracy of the classification model's prediction results, including indicators such as true positive examples, false positive examples, true negative examples, and false negative examples; finally, the accuracy, precision, recall rate, and F1 score are calculated to evaluate the performance indicators of the classification model, and the model is optimized based on F1.

[0161] The tuning criteria used are:

[0162] Precision: The ratio of correctly predicted positive data to predicted positive data, P, P = TP / (TP + FP).

[0163] Recall: The ratio R of the predicted correct positive data to the actual positive data, R = TP / (TP + FN).

[0164] F1 score: The harmonic mean F of precision and recall, F = 2 / (1 / P + 1 / R) = 2 * P * R / (P + R).

[0165] Among them, TP is a true positive example, which means that the prediction is a positive example and the actual one is also a positive example; FN is a false negative example, which means that the prediction is not a positive example, but the actual one is a positive example; TN is a true negative example, which means that the prediction is not a positive example, but the actual one is not a positive example; FP is a false positive example, which means that the prediction is a positive example, but the actual one is not a positive example.

[0166] When optimizing the log anomaly detection model: first, use feature engineering to process the original features, including feature selection, feature transformation, feature construction, etc., to improve model performance; secondly, use Grid Search to find the best model hyperparameter settings by systematically searching for multiple parameter combinations; then control the model complexity through methods such as L1 regularization and L2 regularization to prevent overfitting; finally, use integrated learning to strengthen the model through the Bagging algorithm.

[0167] In the model evaluation method, since the model parameter nu and kernel parameter gamma have an important impact on the performance of the support vector machine (SVM), the model parameters and kernel parameters are adjusted. The model parameters are the upper bound of the proportion of sample points that are misclassified and the lower bound of the proportion of support vectors in the sample points, which are used to control the error tolerance of the model, while the kernel parameters define the range of influence of a single training sample on the result, and are used to define the width of the RBF (radial basis function) kernel, thereby affecting the degree of fit of the model to the data. In order to adjust these two parameters and find the best parameter combination to obtain a higher F1 value, the kernel parameter range is locked in {1e -8 , 1}, the model parameters are locked between {0.001, 0.2}, and grid search is used to find the optimal combination. Finally, the parameter model of kernel parameters and model parameters with the highest F1 value is taken and saved.

[0168] Among them, gamma is the parameter of the kernel function (POLY / RBF / SIGMOID); nu is the parameter of SVM.

[0169] In SVM theory, the use of appropriate kernel functions can transform linear unclassifiable problems in low-dimensional space into linear separable problems in high-dimensional space with the help of kernel functions, and then the optimal boundary of classification can be found in high-dimensional space.

[0170] The radial basis function is a scalar function that is radially symmetric and is usually defined as a monotonic function of the radial distance between the sample and the data center. The RBF kernel is a commonly used kernel function. It is the most commonly used kernel function in support vector machine classification. The commonly used Gaussian radial basis function is as follows:

[0171]

[0172] in, is the squared Euclidean distance between two eigenvectors; is the center of the kernel function, usually one or more data points in the training data; It is a free parameter, which is the width parameter of the function and controls the radial range of the function. is the current test sample i. f(i) measures the degree of association between the test sample and the training sample. Under the same conditions, the larger σ is, The more it fits the training sample x′, the smaller σ is, The more it does not match the training sample x′. describes the extent to which a single training sample affects the results, and The smaller it is, the easier it is to overfit, because only training samples at the boundary have an effect on the decision boundary, and vice versa. The larger it is, the easier it is to underfit and the boundary tends to be smoother.

[0173] In this embodiment, the process of performing anomaly detection on system logs using the trained log anomaly detection model includes:

[0174] Extract access path feature value vector of system log;

[0175] Through the trained log anomaly detection model, the access path feature value vector of the system log is identified to obtain the system log anomaly recognition result.

[0176] The machine learning-based log anomaly detection method disclosed in the present embodiment determines and marks the attack category of each system log after obtaining multiple system logs, and divides all system logs based on the determined attack category; for each type of divided system log, a representative log is determined by a clustering method; then, an access path feature value vector of each representative log is extracted; the access path feature value vectors of all representative logs are used to train the constructed log anomaly detection model; since the selected representative log is the system log that best represents a type of attack category, when the access path feature value vectors of the representative log are used to train the constructed log anomaly detection model, the training accuracy and training efficiency of the log anomaly detection model are improved; when the trained log anomaly detection model is used to perform anomaly detection on the system log, the accuracy of the log anomaly detection is improved.

[0177] Example 2

[0178] In this embodiment, a log anomaly detection system based on machine learning is disclosed, including:

[0179] Log acquisition module, used to obtain multiple system logs;

[0180] An attack category determination module is used to determine and mark the attack category of each system log;

[0181] The log classification module is used to classify all system logs according to attack categories to obtain multiple types of system logs;

[0182] A representative log extraction module is used to cluster each type of system log respectively, and obtain multiple cluster centers of each type of system log as representative logs of each type of system log;

[0183] A feature extraction module is used to extract a feature value vector of an access path representing each log;

[0184] The model training module is used to train the constructed log anomaly detection model by using the access path feature value vectors representing all logs, and after the training is completed, a trained log anomaly detection model is obtained;

[0185] The anomaly detection module is used to perform anomaly detection on system logs using the trained log anomaly detection model.

[0186] The present invention also discloses a computer device, which includes:

[0187] a processor adapted to execute a computer program;

[0188] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the log anomaly detection method based on machine learning disclosed in Example 1 is implemented.

[0189] The present invention also discloses a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor and executing the log anomaly detection method based on machine learning disclosed in Example 1.

[0190] The method disclosed in Example 1 can be directly embodied as a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0191] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0192] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A log anomaly detection method based on machine learning, characterized in that: include: Get multiple system logs; Identify and label the attack category of each system log; Divide all system logs according to attack categories to obtain multiple types of system logs; Cluster each type of system log respectively, and obtain multiple cluster centers of each type of system log as representative logs of each type of system log; Extracting the access path feature value vector of each representative log, wherein the process of extracting the access path feature value vector of the representative log includes: determining the access path of the representative log; dividing the access path of the representative log into words to obtain multiple phrases; calculating the word frequency and inverse document frequency of each phrase; multiplying the word frequency and inverse document frequency of each phrase to obtain the importance of each phrase; selecting the phrase whose importance is greater than a set value as the access path feature value vector of the representative log; The constructed log anomaly detection model is trained using all access path feature value vectors representing the logs. After the training is completed, a trained log anomaly detection model is obtained. Use the trained log anomaly detection model to perform anomaly detection on system logs.

2. The log anomaly detection method based on machine learning as claimed in claim 1, characterized in that: Extract request information from each system log; Match the extracted request information with the regular expression of each attack category; An attack category corresponding to a regular expression matching the request information is selected as the attack category of the log to which the request information belongs.

3. The log anomaly detection method based on machine learning as claimed in claim 1, characterized in that: By performing multiple rounds of clustering on each type of system log, multiple cluster centers of a certain type of log are determined; for each type of system log, in each round of clustering, the shortest distance between each system log sample in this type of system log and the existing cluster center of this type of system log is calculated; based on the shortest distance between each system log and the existing cluster center, the fitness of each system log is calculated; Calculate the proportion of the fitness of each system log to the total fitness of the system logs of this type as the individual probability of each system log; According to the individual probability of each system log, the cumulative probability of each system log is calculated and determined; according to the cumulative probability of each system log, the cluster center of this round of clustering is determined.

4. The log anomaly detection method based on machine learning as claimed in claim 3, characterized in that: The total fitness of a certain type of system log is the sum of the fitness of all system logs in this type of system log.

5. The log anomaly detection method based on machine learning as claimed in claim 1, characterized in that: The log anomaly detection model is constructed using a single-classification support vector machine.

6. The log anomaly detection system based on machine learning is characterized by: include: Log acquisition module, used to obtain multiple system logs; An attack category determination module is used to determine and mark the attack category of each system log; The log classification module is used to classify all system logs according to attack categories to obtain multiple types of system logs; A representative log extraction module is used to cluster each type of system log respectively, and obtain multiple cluster centers of each type of system log as representative logs of each type of system log; The feature extraction module is used to extract the access path feature value vector of each representative log, wherein the process of extracting the access path feature value vector of the representative log includes: determining the access path of the representative log; dividing the access path of the representative log into words to obtain multiple phrases; calculating the word frequency and inverse document frequency of each phrase; multiplying the word frequency and inverse document frequency of each phrase to obtain the importance of each phrase; selecting the phrase whose importance is greater than a set value as the access path feature value vector of the representative log; The model training module is used to train the constructed log anomaly detection model by using the access path feature value vectors representing all logs, and after the training is completed, a trained log anomaly detection model is obtained; The anomaly detection module is used to perform anomaly detection on system logs using the trained log anomaly detection model.

7. An electronic device, characterized in that: The device comprises: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the log anomaly detection method based on machine learning according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the log anomaly detection method based on machine learning according to any one of claims 1 to 5.

Images