A real-time anomaly detection system in a big data environment

Abstract

This invention discloses a real-time anomaly detection system for large-scale data environments, relating to the field of network data monitoring technology. The system implements a streaming process in feature extraction and classification prediction models. When generating feature traffic data, CICFlowMeter introduces a message middleware and proposes a master-slave architecture using Elasticsearch and MySQL for massive data anomaly retrieval. It combines the high-performance full-text search of Elasticsearch with the highly reliable persistent storage of MySQL, leveraging the advantages of each search storage engine. A dedicated synchronization module is designed for data update synchronization, and a Kubernetes server is built. After updating the classification model, Kubernetes performs rolling updates to ensure the business is always online. To prevent excessive traffic congestion and avoid traffic loss in the message queue, Kubernetes can also monitor excessive system CPU usage (i.e., excessive anomaly detection traffic data) and excessive anomaly detection thread usage, enabling horizontal migration of the system to ensure no data loss and improve system real-time performance.

Description

Technical Field

[0001] This invention relates to the field of network data monitoring technology, and more specifically, to a real-time anomaly detection system in a large data volume environment. Background Technology

[0002] Network anomalies typically refer to situations where network operations deviate from normal network behavior. The causes of these anomalies are varied, including network overload, worm intrusions, routing policy modifications, and distributed denial-of-service (DDoS) attacks. Abnormal network traffic is the most common threat among network anomalies. Abnormal network traffic can lead to reduced central network speeds or even network paralysis, causing severe damage to the network environment.

[0003] In recent years, machine learning has achieved high accuracy in anomaly detection and classification. While its application to real-world production environments has yielded significant success in terms of functionality, performance challenges remain, one of which is timeliness. Traditional supervised learning algorithms classify data by using a pre-trained classification model to predict its classification. This presents two challenges: if the access gateway sets an excessively long persistence period for traffic data into PCAP packets, real-time detection efficiency is significantly reduced; if the period is too short, the system must load the model from disk after each packetization, a frequent occurrence that leads to low detection efficiency.

[0004] Regarding model optimization, when a new labeled dataset arrives, we optimize the model and re-upload it to the system. This process first causes the entire system to stop and then the model is replaced and re-uploaded. This process may cause the system to be unavailable for a period of time. During the unavailable period, traffic data will be lost and cannot be detected.

[0005] In addition, after the detection work is completed, traditional relational databases also face significant challenges in terms of time performance when querying anomaly detection results in a large data environment. Although traditional databases have made many optimizations in querying, such as optimizing indexes and master-slave separation, the effect is minimal when the data volume reaches tens of millions, which may cause the system interface to freeze for a long time or even lose connection to the database and become unusable.

[0006] Based on the above description, we found that the current network anomaly detection methods have the following three main shortcomings:

[0007] 1) In the process of feature extraction and classification prediction, CICFlowMeter needs to calculate all the data of the traffic data packet and write the data to the disk before making predictions when generating feature traffic data, which results in low efficiency and poor real-time performance.

[0008] 2) Additional data synchronization operations are performed in the main business, resulting in lower data retrieval and storage performance;

[0009] 3) When traffic is high, if the processing speed cannot match the traffic, it may result in data loss and poor performance. Summary of the Invention

[0010] The present invention provides a real-time anomaly detection system in a large data environment, which can alleviate the above-mentioned problems.

[0011] To alleviate the above problems, the technical solution adopted by the present invention is as follows:

[0012] A real-time anomaly detection system for large data volumes, the anomaly detection process of the real-time anomaly detection system includes the following steps:

[0013] S1. Start the real-time anomaly detection system and load the machine learning model simultaneously;

[0014] S2. Pack the traffic of the system access side router gateway to obtain traffic data packets;

[0015] S3. Calculate the characteristic data of the traffic data packets using CICFlowMeter software and send them to the message queue;

[0016] S4. The machine learning model obtains feature data from the message queue and uses a multi-detection thread approach to perform anomaly detection on multiple feature data simultaneously, obtains the anomaly detection results, and writes them to the message queue.

[0017] S5. The real-time statistics module obtains the anomaly detection results from the message queue, performs data statistics, and obtains the anomaly statistics results.

[0018] S6. The real-time statistics module sends the anomaly detection results and anomaly statistics results to the MySQL cluster. The MySQL cluster adds the anomaly detection results and anomaly statistics results to the persistence operation and synchronizes the newly added and changed data to the Elasticsearch cluster.

[0019] The S7 and Elasticsearch clusters perform data queries based on newly added or changed data and return the query results to the data dashboard of the real-time statistics module.

[0020] In a preferred embodiment of the present invention, in step S3, the CICFlowMeter software sends a piece of feature data that has not yet been sent to the message queue to the message queue for each feature data calculated.

[0021] In a preferred embodiment of the present invention, in step S3, the real-time anomaly detection system obtains the CICFlowmeter software service by calling it, and the internal persistence strategy of the CICFlowmeter software is modified to send the calculated feature data to the message queue.

[0022] In a preferred embodiment of the present invention, when the real-time anomaly detection system is started, the machine learning model is directly loaded into the system's running memory. The machine learning model subscribes to the corresponding traffic feature topic in the message queue and obtains the feature data corresponding to the topic.

[0023] In a preferred embodiment of the present invention, step S6, which involves synchronizing data between the MySQL cluster and the Elasticsearch cluster, specifically includes the following steps:

[0024] S61. Monitor the binlog of the MySQL cluster and obtain changed data;

[0025] S62. Based on the operation type of the changed data, convert the changed data into objects of different types;

[0026] S63. Initialize the database table, operation type and different types of objects corresponding to the changed data as binlog transmission service objects;

[0027] S64. Convert the binlog transport service object to JSON type and send it to the data synchronization topic in the message queue;

[0028] S65. Initialize the add / delete / update list, classify the operation types of data objects received from the data synchronization topic of the message queue, i.e., the operation types of binlog transmission service objects, and add them to the list corresponding to the operation type;

[0029] S67. Based on the different operation types of the changed data, generate different request methods and send them to Elasticsearch to achieve data synchronization.

[0030] In a preferred embodiment of the present invention, in step S7, Elasticsearch is used for massive data retrieval, and the specific steps are as follows:

[0031] S71. Generate a data request body based on the query conditions input from the user interface;

[0032] S72. Send the data request body to the ES cluster to obtain the data response body;

[0033] S73. If the response is successful, extract the primary key index set corresponding to the data from the data response body and then execute step S74. Otherwise, perform a full table query in the MySQL cluster based on the data request body and then jump to step S75.

[0034] S74. Perform a reverse table query on the MySQL cluster based on the primary key index set;

[0035] S75. Return the retrieved query results to the user.

[0036] In a preferred embodiment of the present invention, the real-time anomaly detection system uses Docker-Kubernetes to batch and continuously update the machine learning model based on the query results.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] A process for implementing streaming processing in feature extraction and classification prediction models is proposed. When generating feature traffic data, CICFlowMeter no longer needs to calculate all the data in the traffic data packet and write the data to disk before making a prediction. Instead, it introduces a message queue middleware to reduce the disk writing process and advance the prediction time, thereby further improving real-time performance.

[0039] This paper proposes a master-slave architecture using Elasticsearch and MySQL for massive data anomaly retrieval. By combining the high-performance full-text search capabilities of Elasticsearch with the highly reliable persistent storage of MySQL, the advantages of each search and storage engine are utilized. At the same time, a dedicated synchronization module is designed for data update synchronization, which avoids additional data synchronization operations in the main business, thereby greatly improving data retrieval and storage performance.

[0040] Setting up a Kubernetes server is for the process of updating and bringing back the core detection component offline in high-traffic scenarios. To prevent excessive traffic loss in the message queue, Kubernetes is used for rolling updates after the classification model is updated in the system, ensuring that the business is always online. At the same time, Kubernetes can also monitor the system CPU usage, which means too much anomaly detection traffic data. If too many anomaly detection threads are used, the system can also perform horizontal migration to ensure that no data is lost and improve the system's real-time performance.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is an architecture diagram of a real-time anomaly detection system;

[0044] Figure 2 This is a flowchart of the message queue production side;

[0045] Figure 3 This is a flowchart of the message queue consumer side;

[0046] Figure 4 This is a flowchart of the binlog monitoring process;

[0047] Figure 5 It updates the ES flowchart;

[0048] Figure 6 This is a flowchart of the anomaly retrieval process. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0050] like Figure 1 , Figure 2 and Figure 3 As shown, this invention provides a real-time anomaly detection system for large data volumes. The anomaly detection process of the real-time anomaly detection system includes the following steps:

[0051] S1. Start the real-time anomaly detection system and load the machine learning model simultaneously, instead of waiting for the detection algorithm to predict feature data before starting. This reduces the need to frequently start the machine learning model and improves detection efficiency.

[0052] S2. Pack the traffic from the system access side router gateway to obtain traffic data packets.

[0053] S3. Calculate the characteristic data of the traffic data packets using CICFlowMeter software and send them to the message queue;

[0054] S4. The machine learning model retrieves feature data from the message queue and uses a multi-detection thread approach to simultaneously perform anomaly detection on multiple feature data, obtains the anomaly detection results, and writes them to the message queue.

[0055] S5. The real-time statistics module obtains the anomaly detection results from the message queue and performs data statistics to obtain the anomaly statistics results.

[0056] S6, the real-time statistics module sends the anomaly detection results and anomaly statistics results to the MySQL cluster. The MySQL cluster adds the anomaly detection results and anomaly statistics results to persistence and synchronizes the newly added and changed data to the Elasticsearch cluster.

[0057] The S7 and Elasticsearch clusters perform data queries based on newly added or changed data and return the query results to the data dashboard module.

[0058] In this invention, packaged traffic data packets are used by the third-party software CICFlowMeter to generate corresponding feature data for the traffic packets. Simultaneously, the feature data persistence strategy of CICFlowMeter is rewritten, no longer writing to disk, but instead connecting to a message queue and subscribing to topics. A topic is analogous to a database table, and subscribing to a topic is similar to a producer sending data to a specified database table, and a consumer retrieving data from that table. Each calculated feature data point is directly sent to the message queue instead of being written to a disk CSV file, reducing disk write time and eliminating the need to wait for all feature data of the traffic packet to be written before classification and prediction, thus reducing latency.

[0059] In this invention, when the real-time anomaly detection system is started, the machine learning model is directly loaded into the system's running memory. The machine learning model subscribes to the corresponding traffic feature topics in the message queue and obtains the feature data corresponding to that topic, such as... Figure 3 As shown, the anomaly detection algorithm subscribes to topics generated by corresponding features and introduces a thread pool. When high-intensity traffic arrives, multiple detection threads can run simultaneously, improving detection speed and efficiency, reducing latency, and finally saving the classification prediction results to the database.

[0060] In this invention, the process of filtering anomaly detection results from feature data prediction results combines the characteristics of a relational database master-slave architecture. Elasticsearch, a full-text search engine, is used for data querying (the master business), while MySQL handles data creation, deletion, and modification operations (the slave business). Figure 4 and Figure 5 As shown, the specific steps for synchronizing and retrieving data between Elasticsearch and MySQL are as follows:

[0061] S61. Monitor the binlog of the MySQL cluster and obtain changed data;

[0062] S62. Based on the operation type of the changed data, convert the changed data into objects of different types;

[0063] S63. Initialize the database table, operation type and different types of objects corresponding to the changed data as binlog transmission service objects;

[0064] S64. Convert the binlog transport service object to JSON type and send it to the data synchronization topic in the message queue;

[0065] S65. Initialize the add / delete / update list, classify the operation types of data objects received from the data synchronization topic of the message queue, i.e., the operation types of binlog transmission service objects, and add them to the list corresponding to the operation type;

[0066] S67. Based on the different operation types of the changed data, generate different request methods and send them to Elasticsearch to achieve data synchronization.

[0067] In this invention, a data synchronization service is used to consume the incremental Binlog log content in the message queue. Figure 5 To update Elasticsearch (ES) based on the data changes obtained from MySQL, a list of corresponding operation types is generated according to the object type. Then, based on the object table to which the data object in the list belongs, the ES client request method is executed to update ES, thereby achieving data synchronization.

[0068] In this invention, Elasticsearch is used for massive data retrieval, such as... Figure 6 As shown, the specific steps are as follows:

[0069] S71. Generate a data request body based on the query conditions input from the user interface;

[0070] S72. Send the data request body to the ES cluster to obtain the data response body;

[0071] S73. If the response is successful, extract the primary key index set corresponding to the data from the data response body and then execute step S74. Otherwise, perform a full table query in the MySQL cluster based on the data request body and then jump to step S75.

[0072] S74. Perform a reverse table query on the MySQL cluster based on the primary key index set;

[0073] S75. Return the retrieved query results to the user.

[0074] In this invention, when retrieving traffic anomaly data, Elasticsearch server is used as the primary query method, and MySQL query is used as a fallback strategy. The process is as follows: Figure 6 As shown. Based on the query conditions input by the user, an Elasticsearch query request body is generated and sent to the Elasticsearch server. If the response is successful, the primary key index set of the corresponding range of data is obtained, and a table lookup query is performed on the MySQL database server. If the query response fails, the query is performed directly in MySQL based on the SQL query statement.

[0075] In this invention, the method for rolling updates of machine learning models includes: using GitLab as a code management repository, establishing a Docker private repository, packaging the latest code into an image and pushing it to the private repository, and using Kubernetes to perform rolling releases through container images to ensure that there is always a system copy running online, that is, the business is always running, and the system business will not become unavailable due to updating the model.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time anomaly detection system for large data volumes, characterized in that, The anomaly detection process of a real-time anomaly detection system includes the following steps: S1. Start the real-time anomaly detection system and load the machine learning model simultaneously; S2. Pack the traffic of the system access side router gateway to obtain traffic data packets; S3. The characteristic data of the traffic data packet is calculated by the CICFlowMeter software and sent to the message queue; and for each characteristic data calculated by the CICFlowMeter software, a characteristic data that has not been sent to the message queue is sent to the message queue; the real-time anomaly detection system obtains the CICFlowmeter software service by calling it, and the internal persistence strategy of the CICFlowmeter software is modified to send the calculated characteristic data to the message queue. S4. The machine learning model obtains feature data from the message queue and uses a multi-detection thread approach to perform anomaly detection on multiple feature data simultaneously, obtains the anomaly detection results, and writes them to the message queue. S5. The real-time statistics module obtains the anomaly detection results from the message queue, performs data statistics, and obtains the anomaly statistics results. S6. The real-time statistics module sends the anomaly detection results and anomaly statistics results to the MySQL cluster. The MySQL cluster adds the anomaly detection results and anomaly statistics results to the persistence operation and synchronizes the newly added and changed data to the Elasticsearch cluster. Data synchronization between a MySQL cluster and an Elasticsearch cluster includes the following steps: S61. Monitor the binlog of the MySQL cluster and obtain changed data; S62. Based on the operation type of the changed data, convert the changed data into objects of different types; S63. Initialize the database table, operation type and different types of objects corresponding to the changed data as binlog transmission service objects; S64. Convert the binlog transport service object to JSON type and send it to the data synchronization topic in the message queue; S65. Initialize the add / delete / update list, classify the operation types of data objects received from the data synchronization topic of the message queue, i.e., the operation types of binlog transmission service objects, and add them to the list corresponding to the operation type; S67. Based on the different operation types of the changed data, generate different request methods and send them to Elasticsearch to achieve data synchronization; The S7 Elasticsearch cluster performs data queries based on newly added or changed data and returns the query results to the data dashboard of the real-time statistics module. The specific steps are as follows: S71. Generate a data request body based on the query conditions input from the user interface; S72. Send the data request body to the ES cluster to obtain the data response body; S73. If the response is successful, extract the primary key index set corresponding to the data from the data response body and then execute step S74. Otherwise, perform a full table query in the MySQL cluster based on the data request body and then jump to step S75. S74. Perform a reverse table query on the MySQL cluster based on the primary key index set; S75、According to the query result, return to the user; The real-time anomaly detection system loads the machine learning model into the system running work memory directly when starting the system, and subscribes to the corresponding traffic feature topic in the message queue, and obtains the feature data corresponding to the topic.

2. The real-time anomaly detection system in a big data environment of claim 1, wherein, ​

Images