IPS log de-overlapping and merging method based on machine learning

Abstract

The invention provides an IPS log deduplication merging method based on machine learning. The IPS log deduplication merging method comprises the following steps of A, data collection and preparation; b, data feature processing; c, dividing a data set; d, selecting a classification model; e, model training; f, performing model evaluation; g, testing and deploying the model; h, de-duplicating and combining the logs; according to the IPS log deduplication and combination method, the model is fitted through the machine learning technology, and the accuracy can be optimized through continuous iteration; manual log deduplication and combination work can be greatly reduced, and the labor cost is reduced; according to the method, the data collected through real attacks has better universality, and meanwhile, the data can also be used for research and application of other projects in the later period; the model provided by the invention has multiple labels, the conditions of merging and de-duplication can be changed as required, and the de-duplication and merging efficiency of logs is improved.

Description

Technical Field

[0001] The present application relates to the technical field of IPS log de-duplication, and in particular to an IPS log de-duplication and merging method based on machine learning. Background Art

[0002] With the improvement of information technology, in order to better prevent network attacks, the Agricultural Bank of China has introduced many network security devices. The number of alarm logs generated by IPS in these devices is large, and the workload is large when analyzing them every month. It is observed that the same type of attacks launched by the same attacker will generate a large number of logs. The main reasons are different attack target IPs (scanning traversal), different attack paths (directory traversal), different attack times (replay attacks), different attack source IPs (second dial IP), etc. If these same type of attack alarms can be merged, it will be easier for administrators to conduct research and analysis on such attacks, and a clearer attacker portrait can be formed. Therefore, it is necessary to design an IPS log de-duplication and merging method based on machine learning. Summary of the invention

[0003] The present application provides an IPS log deduplication and merging method based on machine learning to solve the problem of large analysis workload in the prior art, realizes model fitting through machine learning technology, and can optimize the accuracy through continuous iteration; it can greatly reduce the manual deduplication and merging work of logs.

[0004] The present application provides an IPS log deduplication and merging method based on machine learning, comprising the following steps:

[0005] A. Data collection and preparation;

[0006] B. Data feature processing;

[0007] C. Dataset division;

[0008] D. Select a classification model;

[0009] E. Model training;

[0010] F. Model evaluation;

[0011] G. Model testing and deployment;

[0012] H. Log deduplication and merging;

[0013] I. Model optimization.

[0014] Preferably, the step A specifically comprises the following steps:

[0015] a. Collect data: Collect a large amount of real IPS log data to ensure that the data is representative and covers multiple intrusion types;

[0016] b. Clean the data: process missing values, remove duplicates, format timestamps and IP address information;

[0017] c. Label data: Label each log to distinguish whether they belong to the same type of attack or different types of attacks;

[0018] d. Feature extraction: Extract the features required by the machine learning model from the logs, including timestamp, source IP and target IP, attack type, attack severity level, port, and protocol.

[0019] Preferably, the step B specifically comprises the following steps:

[0020] a. Text feature processing: Use natural language processing technology to process the free text fields of logs, and use TF-IDF and word vector technology to convert logs into numerical features;

[0021] b. Numerical feature processing: standardize or normalize numerical features to ensure that the numerical differences between different feature dimensions do not cause bias to the model;

[0022] c. Feature selection: Use statistical or model-based methods to select the features that contribute most to the classification task.

[0023] Preferably, the specific process of data set division in step C is as follows:

[0024] a. Divide the data set into training set, validation set and test set, with the ratio of training set, validation set and test set being 7:2:1;

[0025] b. Cross-validation: Use K-fold cross-validation to improve the generalization ability of the model, K=5 or K=10.

[0026] Preferably, the classification model in step D includes a machine learning model and a deep learning model, wherein the machine learning model selects a support vector machine, a random forest, a gradient boosting decision tree, XGBoost, a LightGBM ensemble learning model or a neural network model.

[0027] Preferably, the model training method in step E is as follows:

[0028] a. Define the loss function: select a suitable loss function and determine the optimizer; the loss function uses cross entropy loss and the optimizer uses Adam or SGD;

[0029] b. Hyperparameter tuning: Use grid search or random search methods to adjust the model's hyperparameters, including learning rate, tree depth, and regularization parameters;

[0030] c. Model training: Use the training set to train the model and record the changes in training loss and accuracy.

[0031] Preferably, in step F, the model performance is evaluated on the validation set using accuracy, precision, recall, and F1 score evaluation indicators; if the model performs poorly, the features are adjusted, the model architecture is changed, or more hyperparameter tuning is performed.

[0032] Preferably, the specific process of step G is as follows:

[0033] a. Perform a final test on the model on the test set to evaluate its generalization ability;

[0034] b. If the results are satisfactory, the model will be integrated into the IPS log analysis system for real-time log deduplication and merging.

[0035] Preferably, step H includes log deduplication and log merging, wherein log deduplication: training the model to identify duplicate log records, and judging whether they are duplicate entries based on timestamps, attack features and other key fields; log merging: for multiple logs belonging to the same event, the model merges them into one record through clustering or classification methods.

[0036] Preferably, the model optimization in step I is specifically as follows:

[0037] a. Continuously monitor the performance of the model and regularly update the model based on new data;

[0038] b. Use incremental learning or online learning methods to dynamically adjust the model to adapt to changing intrusion patterns.

[0039] Beneficial effects:

[0040] (1) The IPS log deduplication and merging method adopted by the present invention fits the model through machine learning technology, and can optimize the accuracy through continuous iteration; it can greatly reduce the manual deduplication and merging work of logs, thereby reducing labor costs.

[0041] (2) The data collected by the present invention through real attacks has better versatility, and the data can also be used for research and application of other projects in the future.

[0042] (3) The model of the present invention has multiple tags, and the conditions for merging and deduplication can be changed as required, thereby improving the efficiency of log deduplication merging.

[0043] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 This is the workflow of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification, claims and drawings of this application are intended to cover non-exclusive inclusions.

[0048] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase "embodiments" in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0050] See also Figure 1 , the present application provides an IPS log deduplication and merging method based on machine learning, comprising the following steps:

[0051] A. Data collection and preparation;

[0052] B. Data feature processing;

[0053] C. Dataset division;

[0054] D. Select a classification model;

[0055] E. Model training;

[0056] F. Model evaluation;

[0057] G. Model testing and deployment;

[0058] H. Log deduplication and merging;

[0059] I. Model optimization.

[0060] In the present invention, step A specifically comprises the following steps:

[0061] a. Collect data: Collect a large amount of real IPS log data to ensure that the data is representative and covers multiple intrusion types;

[0062] b. Clean the data: process missing values, remove duplicates, format timestamps and IP address information;

[0063] c. Label data: Label each log to distinguish whether they belong to the same type of attack or different types of attacks;

[0064] d. Feature extraction: Extract the features required by the machine learning model from the logs, including timestamp, source IP and target IP, attack type, attack severity level, port, and protocol.

[0065] The data collection and preparation method adopted by the present invention can realize the collection, cleaning, labeling and feature processing of IPS log data, has high work efficiency, and facilitates the subsequent further processing of IPS log data.

[0066] In the present invention, step B specifically comprises the following steps:

[0067] a. Text feature processing: Use natural language processing technology to process the free text fields of logs, and use TF-IDF and word vector technology to convert logs into numerical features. The word vector technology uses Word2Vec or BERT. Among them, natural language processing technology is a field of computer science, artificial intelligence, and linguistics that focuses on the interaction between computers and human (natural) language. It can realize various theories and methods for effective communication between humans and computers using natural language, and can achieve efficient processing of text.

[0068] b. Numerical feature processing: standardize or normalize numerical features to ensure that the numerical differences between different feature dimensions do not cause bias to the model;

[0069] c. Feature selection: Use statistical methods or model-based methods (Random Forest or Lasso) to select the features that contribute most to the classification task.

[0070] In the present invention, the specific process of data set division in step C is as follows:

[0071] a. Divide the data set into training set, validation set and test set, with the ratio of training set, validation set and test set being 7:2:1;

[0072] b. Cross-validation: Use K-fold cross-validation to improve the generalization ability of the model, K=5 or K=10.

[0073] The generalization ability of a model refers to its performance on data that it has never seen before, that is, its ability to adapt to new data. In machine learning, the goal of a model is to accurately predict or classify unknown data. If a model can only perform well on training data but not on new data, then the generalization ability of the model is poor.

[0074] Improve the generalization ability of the model, that is, the ability to perform well on data that has never been seen. For a machine learning algorithm, generalization ability is an important indicator for evaluating its performance. Usually, the data set is divided into a training set and a test set. The training set is used to train the model, and then the test set is used to test the generalization ability of the model.

[0075] In the present invention, the classification model in step D includes a machine learning model and a deep learning model, wherein the machine learning model selects a support vector machine, a random forest, a gradient boosting decision tree, XGBoost, a LightGBM ensemble learning model or a neural network model. If the amount of data is large enough, a deep learning model is used: a sequence model based on RNN / LSTM can process time series IPS logs, and a Transformer model can be used for complex log parsing and classification.

[0076] In the present invention, the model training method in step E is as follows:

[0077] a. Define the loss function: select a suitable loss function and determine the optimizer; the loss function uses cross entropy loss and the optimizer uses Adam or SGD;

[0078] b. Hyperparameter tuning: Use grid search or random search methods to adjust the model's hyperparameters, including learning rate, tree depth, and regularization parameters;

[0079] c. Model training: Use the training set to train the model and record the changes in training loss and accuracy.

[0080] Through the above steps, model training can enable machine learning algorithms to learn patterns from data, and through continuous optimization and adjustment, improve the prediction accuracy of the model, and ultimately achieve effective use in various application scenarios. Model training is the core link in machine learning and deep learning. The purpose is to enable the model to learn useful patterns and features from data, so as to make accurate predictions or decisions on new and unknown data.

[0081] In the present invention, in step F, the model performance is evaluated on the validation set, using accuracy, precision, recall, and F1 score evaluation indicators; if the model performs poorly, the features are adjusted, the model architecture is changed, or more hyperparameter tuning is performed.

[0082] The present invention adopts holdout cross validation and K-fold cross validation to obtain a reliable estimate of the performance error of the model. K-fold cross validation repeats the holdout method k times on k training subsets. K-fold cross validation randomly divides the training set into k parts without repetition, k-1 of which are used for training and the remaining one is used for testing. The process is repeated k times to obtain the evaluation of the model performance by k models.

[0083] In the present invention, the specific process of step G is as follows:

[0084] c. Perform a final test on the model on the test set to evaluate its generalization ability;

[0085] d. If the results are satisfactory, the model will be integrated into the IPS log analysis system for real-time log deduplication and merging.

[0086] In the present invention, step H includes log deduplication and log merging, wherein log deduplication: training the model to identify duplicate log records, and judging whether they are duplicate entries based on timestamps, attack features and other key fields; log merging: for multiple logs belonging to the same event, the model merges them into one record through clustering or classification methods.

[0087] In addition, in the present invention, the model optimization in step I is specifically as follows:

[0088] a. Continuously monitor the performance of the model and regularly update the model based on new data;

[0089] b. Use incremental learning or online learning methods to dynamically adjust the model to adapt to changing intrusion patterns.

[0090] The present invention adopts visualization technology to monitor the performance of the model. The present invention adopts matplotlib and seaborn for data visualization and model result analysis.

[0091] The data collected by the present invention through real attacks has better versatility, and the data can also be used for research and application of other projects in the future; the model of the present invention has multiple labels, and the conditions for merging and deduplication can be changed as required, thereby improving the deduplication and merging efficiency of logs.

[0092] In summary, the IPS log deduplication and merging method adopted in the present invention fits the model through machine learning technology, and can optimize the accuracy through continuous iteration; it can greatly reduce the manual deduplication and merging work of logs, thereby reducing labor costs.

[0093] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for deduplication and merging of IPS logs based on machine learning, characterized in that: The following steps are involved: A. Data collection and preparation; B. Data feature processing; C. Dataset division; D. Select a classification model; E. Model training; F. Model evaluation; G. Model testing and deployment; H. Log deduplication and merging; I. Model optimization.

2. According to the machine learning-based IPS log deduplication and merging method of claim 1, it is characterized in that: The step A specifically comprises the following steps: a. Collect data: Collect a large amount of real IPS log data to ensure that the data is representative and covers multiple intrusion types; b. Clean the data: process missing values, remove duplicates, format timestamps and IP address information; c. Label data: Label each log to distinguish whether they belong to the same type of attack or different types of attacks; d. Feature extraction: Extract the features required by the machine learning model from the logs, including timestamp, source IP and target IP, attack type, attack severity level, port, and protocol.

3. According to the machine learning-based IPS log deduplication and merging method of claim 1, it is characterized in that: The step B specifically comprises the following steps: a. Text feature processing: Use natural language processing technology to process the free text fields of logs, and use TF-IDF and word vector technology to convert logs into numerical features; b. Numerical feature processing: standardize or normalize numerical features to ensure that the numerical differences between different feature dimensions do not cause bias to the model; c. Feature selection: Use statistical or model-based methods to select the features that contribute most to the classification task.

4. According to the machine learning-based IPS log deduplication and merging method of claim 1, it is characterized in that: The specific process of data set division in step C is as follows: a. Divide the data set into training set, validation set and test set, with the ratio of training set, validation set and test set being 7:2:1; b. Cross-validation: Use K-fold cross-validation to improve the generalization ability of the model, K=5 or K=10.

5. According to the machine learning-based IPS log deduplication and merging method of claim 1, it is characterized in that: The classification model in step D includes a machine learning model and a deep learning model, wherein the machine learning model selects a support vector machine, a random forest, a gradient boosting decision tree, XGBoost, a LightGBM ensemble learning model or a neural network model.

6. The IPS log deduplication and merging method based on machine learning according to claim 1 is characterized in that: The model training method in step E is as follows: a. Define the loss function: select a suitable loss function and determine the optimizer; the loss function uses cross entropy loss and the optimizer uses Adam or SGD; b. Hyperparameter tuning: Use grid search or random search methods to adjust the model's hyperparameters, including learning rate, tree depth, and regularization parameters; c. Model training: Use the training set to train the model and record the changes in training loss and accuracy.

7. The IPS log deduplication and merging method based on machine learning according to claim 1 is characterized in that: In step F, the model performance is evaluated on the validation set using accuracy, precision, recall, and F1 score evaluation indicators; if the model performs poorly, the features are adjusted, the model architecture is changed, or more hyperparameter tuning is performed.

8. The IPS log deduplication and merging method based on machine learning according to claim 1, characterized in that: The specific process of step G is as follows: a. Perform a final test on the model on the test set to evaluate its generalization ability; b. If the results are satisfactory, the model will be integrated into the IPS log analysis system for real-time log deduplication and merging.

9. The IPS log deduplication and merging method based on machine learning according to claim 1, characterized in that: The step H includes log deduplication and log merging, wherein log deduplication: training the model to identify duplicate log records, and judging whether it is a duplicate entry based on timestamp, attack characteristics and other key fields; log merging: for multiple logs belonging to the same event, the model merges them into one record through clustering or classification methods.

10. The IPS log deduplication and merging method based on machine learning according to claim 1, characterized in that: The model optimization in step I is specifically as follows: a. Continuously monitor the performance of the model and regularly update the model based on new data; b. Use incremental learning or online learning methods to dynamically adjust the model to adapt to changing intrusion patterns.

Images