An alarm log aggregation optimization system based on machine learning

Abstract

The present invention discloses an alarm log aggregation optimization system based on machine learning, including: a machine learning module, which is used to perform machine learning training according to local input logs and alarm aggregation fields set by users, and obtain a trained machine learning model table; a task management module, which is used to start the machine learning module in a specific situation; a log processing module, which is used to read the trained machine learning model table and log file, and enable the currently enabled model in the trained machine learning model table, mark the log file, and store the marked log in a database; an alarm aggregation module, which aggregates log files according to a set aggregation range and generates alarms; a user interaction module, which is used to operate and modify alarm rules and fields, and view alarm data. The present invention can realize custom aggregation, and the aggregation accuracy is high, and the compatibility and adaptability are strong.

Description

Technical Field

[0001] The present invention relates to the field of log processing, and in particular to an alarm log aggregation optimization system based on machine learning. Background Art

[0002] With the development of the information age, a large amount of log data will be monitored in real time, and the analyzed data will be stored in the project system and statistically analyzed. In order to facilitate network engineers, security experts, and operation and maintenance personnel to efficiently and quickly analyze and locate problems in the current environment based on the time range, log aggregation is quite necessary, because it is impossible to require manual analysis of hundreds of millions of data, which is obviously unrealistic.

[0003] Alarm log aggregation is a common management strategy to improve efficiency and reduce noise. The following is some background information about alarm log aggregation:

[0004] 1. The number of log aggregations has increased dramatically: As the scale and complexity of IT systems continue to increase, the number of logs generated by monitoring systems is also increasing. A large number of duplicate, redundant, and insignificant logs have brought tremendous pressure to operation and maintenance personnel, so log aggregation has become a necessary means.

[0005] 2. Accurate contextual information is needed: It is often difficult to obtain the full picture and contextual information of the problem through only a single log. Log aggregation can integrate related logs, provide more accurate contextual information, and help the operation and maintenance team better understand and analyze the problem.

[0006] 3. Improve fault handling efficiency: Log aggregation can help the operation and maintenance team locate and handle faults more quickly and accurately. By aggregating and filtering logs, operation and maintenance personnel can solve problems more specifically and avoid getting lost in massive logs.

[0007] Log aggregation is a strategy to deal with the flood of logs and improve operation and maintenance efficiency, and is gaining wide application and attention. In order to cope with increasingly complex scenarios, simple logs such as fuzzy matching and exact matching are increasingly unable to meet the needs (for example, multiple attributes of a log actually have a mutual aggregation feature after being converted into corresponding values, such as a certain IP in a similar range, such as 11.10.21.x, in a specific port range of 88-91, there are many logs with a flow of about 100m in a short period of time. Such logs obviously have similarities, and any change of value will result in another scenario. At this time, it is obviously unrealistic to write an algorithm flow for each scenario with code, not to mention that the correlation of some data is even more difficult to detect). At this time, machine learning as a new matching mode is becoming more and more important.

[0008] The prior art generally has the following defects:

[0009] 1. In current alarm aggregation projects, users often focus on data with similar features within a certain time range, and these features are often not met by simple aggregation algorithms. For example, if several attributes are the same within a certain period of time, they are aggregated.

[0010] 2. In alarm aggregation, users often modify the attributes they are concerned about, such as the log source IP, destination IP, log category, and log alarm level during this period, and the source IP, destination IP, and event category after a period of time. He may also want to use different types of machine learning models (such as supervised learning and unsupervised learning). This means that in the case of high user customization, the model trained by the machine in advance will become invalid and the original model cannot be reused.

[0011] 3. Based on the previous point, alarm aggregation usually hopes to quickly observe the changed results after the conditions are changed, but large-scale data machine learning models are often slow to create and cannot meet the needs. Summary of the invention

[0012] In order to solve the above problems, the present invention proposes an alarm log aggregation optimization system based on machine learning, which can not only reasonably apply machine learning to the case of log alarm aggregation, but also enable users to solve the delay problem of temporary training models under the condition of highly customizable rules.

[0013] The system includes:

[0014] Log processing module, machine learning module, user interaction module, task management module and alarm aggregation module;

[0015] The machine learning module is used to perform machine learning training according to the local input log and the alarm aggregation field set by the user to obtain a trained machine learning model table;

[0016] The task management module is used to start the machine learning module under specific circumstances;

[0017] The log processing module is used to read the trained machine learning model table and log files, enable the currently enabled model in the trained machine learning model table, mark the log files, and store the marked logs in the database;

[0018] The alarm aggregation module aggregates log files according to a set aggregation range and generates an alarm;

[0019] The user interaction module is used to operate and modify alarm rules and fields and view alarm data.

[0020] The beneficial effects provided by the present invention are: according to one's own needs, one can arbitrarily specify the focus field to allow the machine learning program to assist in the aggregation of the same type of logs, and can switch between unsupervised learning and supervised learning. When the number of security logs and program log data is 20,000 per second, when the user updates the alarm aggregation field and the mode is unsupervised learning, the machine training model is generated within one minute, and the alarm log generated under the new rules is generated, and the similarity of the logs under the aggregated alarm gradually increases in the following time, and the model prediction accuracy becomes higher. And after the user sets it to the previously used aggregation field again, the user-perceived machine learning training time is close to 0. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the schematic diagram of the system structure;

[0022] Figure 2 Task management module scheduling flow chart;

[0023] Figure 3 Machine learning module flow chart;

[0024] Figure 4 Flowchart of the alarm aggregation module. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0026] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the system structure of the present invention;

[0027] The present invention provides an alarm log aggregation optimization system based on machine learning, comprising:

[0028] Log processing module, machine learning module, user interaction module, task management module and alarm aggregation module;

[0029] The machine learning module is used to perform machine learning training according to the local input log and the alarm aggregation field set by the user to obtain a trained machine learning model table;

[0030] It should be noted that the machine learning training includes supervised learning and unsupervised learning.

[0031] The specific process of the machine learning module is as follows:

[0032] Get the log file from the local storage log;

[0033] Obtain the alarm aggregation specified field set by the user from the user interaction module;

[0034] Convert the specified field into a numeric type and input it into the machine learning model for model training;

[0035] Get the trained model and store it in the local HDFS system, and update the machine learning model table of the corresponding database.

[0036] The task management module is used to start the machine learning module under specific circumstances;

[0037] It should be noted that the specific process of the task management module is as follows:

[0038] At a preset time point every day, the machine learning module is started to update the thermal model; the thermal model specifically refers to: a machine learning model corresponding to the attribute of particular interest set through the user interaction module. The validity period of the thermal model is T days, where T is a preset value. After T days, the thermal model becomes invalid and is no longer updated at the preset time point every day.

[0039] Monitor the user interaction module and start the machine learning module when an alarm rule or field modification occurs.

[0040] The log processing module is used to read the trained machine learning model table and log files, enable the currently enabled model in the trained machine learning model table, mark the log files, and store the marked logs in the database;

[0041] The alarm aggregation module aggregates log files according to a set aggregation range and generates an alarm;

[0042] It should be noted that the specific working process of the alarm aggregation module is as follows: according to the time range specified by the user, logs with the same tag are aggregated into one alarm.

[0043] The user interaction module is used to operate and modify alarm rules and fields and view alarm data.

[0044] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the system structure of the present invention.

[0045] Logs sent from various data sources are collected by the log collection program and pushed to the TOPIC topic corresponding to KAFKA;

[0046] The log processing module has three inputs and one output;

[0047] The first input is the collected logs; the second input is the model metadata; the third input is the currently enabled machine learning model;

[0048] The second input comes from the postgres database; the third input comes from the HDFS system;

[0049] After being processed by the log processing module, the log processing module is responsible for enhancing the attributes of the logs and storing them in the ELASTICSEARCH log index (audit_log). At the same time, the machine algorithm model is used to input log attributes for labeling, which facilitates the subsequent aggregation of logs.

[0050] exist Figure 1 In the log processing module ( Figure 1 The corresponding module in is called program) reads the model metadata from the postgres database to obtain the model stored in hdfs; it should be noted that the postgres database stores relevant information such as training model metadata and alarm aggregation fields;

[0051] The output of the log processing module is the labeled log; the labeled log is output to the ELASTICSEARCH system for local storage.

[0052] Further references Figure 1 , Figure 1 The machine learning module has two inputs and two outputs;

[0053] The first input is the local log obtained from the ELASTICSEARCH system; the second input is the control signal of the task management module;

[0054] The first output is the trained model, which is output to the postgres database and used to update the model training table. The second output is also the trained model, which is stored in the HDFS system.

[0055] Further, refer to Figure 1 The alarm aggregation module includes an input and an output; an input is an alarm aggregation field and an aggregation range from the postgres database, and an output is a generated alarm, which is output to the ELASTICSEARCH database.

[0056] refer to Figure 1 , Figure 1 The user interaction module is used to operate and modify alarm rules and fields, and view alarm data.

[0057] The Spark machine learning module will read the logs stored in the library and the fields specified by the user in the library for setting alarm aggregation, such as source IP, destination IP, event type, and then convert them into numerical types (if the user adopts a supervised learning solution, a specified target attribute needs to be additionally specified as a label), and then be handed over to the corresponding machine learning for training. After a certain period of time, a model is generated and stored in the local HDFS system, and the status of the corresponding database machine learning model table is updated to training completed.

[0058] As an implementation example, for example, the attributes source IP, destination IP, and event type used in this example correspond to the first, fourth, and fifth attributes, and the data with id 145 and the model name 145Kmean are inserted. If a training operation with the same attribute is performed later, the attribute of the last update time of the data with id 145 is modified.

[0059] It should be noted that the task management module is the key to optimizing the machine learning aggregation method. In addition to launching a learning task at dawn every day to update the hot model of machine learning (hot model: the model corresponding to the concerned attribute set by the current alarm system, such as the 145Kmean model corresponding to the aforementioned data No. 145. A model will be identified as a hot model within 30 days after its generation, and the model will be trained and updated with the log data of the past year at dawn every day, regardless of whether the user modifies the concerned attribute. Otherwise, it is a cold model and is not updated daily).

[0060] In order to meet users' real-time requirements (most of the time users hope that the alarm can give a result quickly), it will also monitor the user interaction program. Once the user modifies the aggregate field of concern through the program, it will query whether there is a specified model available. If not, it will first use a small amount of recent data (such as log data in the past 1 hour) to quickly train a model for use by the log processing program. At the same time, it will also use 24 hours, 30 days, and 1 year of data for training, and overwrite the previous model to improve accuracy, so that users cannot perceive the delay caused by the model training process and can get timely feedback on data alarms.

[0061] The log aggregation module aggregates logs with the same tag into an alarm based on the time range specified by the user. The alarm is the data that the user needs to see. The user can see the aggregated alarm through the drop-down box.

[0062] The user interaction module is actually the data used by users to operate and view the alarm page. Users can modify the alarm aggregation field on the page, and the program will notify the task management program so that it can schedule machine learning tasks.

[0063] It should be noted that if the user does not specify any fields, the original log is used by default for fuzzy aggregation. Specifically, after the NLP word segmentation algorithm, Word2Vec is used to create a word vector matrix, which is then input into the machine learning program for unsupervised learning.

[0064] Please refer to Figure 2 , Figure 2 It is the scheduling flow chart of the task management module;

[0065] The user interaction module triggers the task management module;

[0066] Query the postgres alarm rule base and query the rule information;

[0067] Get the fields and the corresponding machine learning algorithms, and piece together the corresponding machine learning model ID (such as 145KMEAN mentioned above);

[0068] Check whether there is a corresponding model in the trained model table according to the machine learning model ID. If there is, check whether the model with the corresponding ID is a hot model. If so, end the task. Otherwise, call the machine learning module, start the machine learning task, and use the data volume of the log for the past hour. Set the subsequent three scheduled tasks (with data volumes of 24 hours, 1 month, and 1 year, respectively) and end the task.

[0069] Please refer to Figure 3 , Figure 3 It is a flowchart of the machine learning module;

[0070] Read the enabled log rules in the database;

[0071] Start the task according to the time range information sent by the task management program;

[0072] Read the logs within the specified time range to the local computer;

[0073] Determine whether the aggregation rule is empty. If so, use the NLP algorithm to segment the text of the original information field of the log, and then use the word2Vec algorithm to convert the segmented text into a word vector matrix; input the word vector matrix to the Spark machine learning module to train the model default1 by the KMEANS trainer, and store the trained model default1 in the HDFS system; insert it into the postgresql model metadata table with the key value default1, and update the model status to enabled, and end.

[0074] Otherwise, further determine whether it is supervised learning. If so, convert the rule-specified attributes into vector matrices and the label attributes into numeric types; input the converted attributes and labels into the machine learning module, train the model with the MLPC trainer, and name it 145MLPC; store the model 145MLPC in HDFS; insert it into the postgresql model metadata table with the key value 145MLPC, and update the model status to enabled, and end

[0075] If it is unsupervised learning, the converted attributes and labels are input into the machine learning module, and the model is trained by the KMEANS trainer and named 145KEMEANS; the model 145KEMEANS is stored in HDFS; it is inserted into the postgresql model metadata table with the key value of 145KEMEANS, and the model status is updated to enabled, and then it ends.

[0076] Please refer to Figure 4 , Figure 4 It is a flowchart of the alarm aggregation module;

[0077] Read the enabling rules of the database;

[0078] Determine whether the rule is empty. If so, query the log data with the log identifier DEFAULT1 within the time range; aggregate the logs with the same log category identifier to generate an alarm, and generate a specified ID (such as ID: 145KMEAN|2023-08-01 14:00:00-2023-08-0114:05:00); insert it into the ELASTICSEARCH database according to the ID. If the ID already exists in the database, overwrite it, otherwise insert it;

[0079] If the rule is not empty, construct the corresponding log identifier (for example, 145KMEANS) according to the rule; query the log data with the specified log identifier within the time range; aggregate the logs with the same log category identifier to generate an alarm, and generate a specified ID; insert it into the ELASTICSEARCH database according to the ID, if the ID already exists in the database, overwrite it, otherwise insert it and end.

[0080] The implementation of the present invention adopts Spark+Kafka+ElasticSearch+HDFS+big data cluster as tools; the database adopts elasticsearch and postgres database.

[0081] Finally, the beneficial effect of the present invention is that: according to one's own needs, one can arbitrarily specify the focus field to allow the machine learning program to assist in the aggregation of the same type of logs, and can switch between unsupervised learning and supervised learning. When the number of security logs and program log data is 20,000 per second, when the user updates the alarm aggregation field and the mode is unsupervised learning, the machine training model is generated within one minute, and the alarm log generated under the new rules is generated. In the following time, the log similarity under the aggregated alarm gradually increases, and the model prediction accuracy becomes higher. And after the user sets it to the previously used aggregation field again, the user-perceived machine learning training time is close to 0.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An alarm log aggregation optimization system based on machine learning, characterized by: include: Log processing module, machine learning module, user interaction module, task management module and alarm aggregation module; The machine learning module is used to perform machine learning training according to the local input log and the alarm aggregation field set by the user to obtain a trained machine learning model table; The task management module is used to start the machine learning module under specific circumstances; The specific process of the task management module is as follows: At a preset time point every day, starting the machine learning module to update the thermal model; Monitor the user interaction module, and start the machine learning module when an alarm rule or field modification occurs; The thermal model specifically refers to: a machine learning model corresponding to a special concern attribute set through a user interaction module; The log processing module is used to read the trained machine learning model table and log files, enable the currently enabled model in the trained machine learning model table, mark the log files, and store the marked logs in the database; The alarm aggregation module aggregates log files according to a set aggregation range and generates an alarm; The user interaction module is used to operate and modify alarm rules and fields and view alarm data.

2. The alarm log aggregation optimization system based on machine learning as claimed in claim 1, characterized in that: The specific process of the machine learning module is as follows: Get the log file from the local storage log; Obtain the alarm aggregation specified field set by the user from the user interaction module; Convert the specified field into a numeric type and input it into the machine learning model for model training; Get the trained model and store it in the local HDFS system, and update the machine learning model table of the corresponding database.

3. The alarm log aggregation optimization system based on machine learning as claimed in claim 1, characterized in that: The thermal model is valid for T days, where T is a preset value. After T days, the thermal model becomes invalid and is no longer updated at a preset time point every day.

4. The alarm log aggregation optimization system based on machine learning as claimed in claim 1, characterized in that: The specific working process of the alarm aggregation module is as follows: according to the time range specified by the user, the logs with the same tag are aggregated into one alarm.

5. The alarm log aggregation optimization system based on machine learning as claimed in claim 1, characterized in that: The machine learning training includes supervised learning and unsupervised learning.

Images