Network log analysis system and method

Abstract

A network log analysis system includes a load balancing module that load balances and forwards online Syslog logs to various processing servers; a data acquisition module that receives Syslog logs and outputs Syslog data; a data caching module that receives and caches Syslog data; a data processing module that reads cached data in real time, parses the meaning of basic log fields, and outputs parsed results and original log data; a data storage module that receives and stores parsed results and original log data when no alarm conditions are triggered; and a streaming alarm processing module that receives parsed results and original log data, sets alarm configuration rules, filters and performs statistical analysis on the original log data, and triggers an alarm when the statistical analysis results match the alarm configuration rules when an alarm condition is triggered. The network log analysis system can achieve load balancing among the acquisition and processing nodes, merge, converge, and reduce noise in alarm information to reduce the impact of duplicate alarms.

Description

Technical Field

[0001] The present invention relates to the technical field of network log data processing, and in particular to a network log analysis system and method. Background Art

[0002] A server is a computer system dedicated to providing data, resources, or services. It primarily responds to requests from clients (such as personal computers, mobile phones, and applications) and provides the necessary support. When a client connects to a server, detailed records of the user's internet usage and system responses are stored on the server. By analyzing network logs, we can expand the system's data usage scenarios, optimize the system's alerting and analysis capabilities, and better mine the data's value. This helps us understand the system's operating status, detect abnormal behavior, and optimize performance. This helps operations and maintenance personnel identify potential security risks and potential failures, thereby improving system performance. For multi-departmental organizations, multiple servers are often deployed to support the daily work of a large number of personnel. Due to the internet usage patterns of different departments and the online habits of different staff members, each server has different processing capabilities. Consequently, during the network log data collection process, some server resources may be underutilized, while others may be overwhelmed. This can lead to high processing pressure on individual system nodes, making them prone to failures, which in turn impacts the normal operation of the system and the effective analysis of network logs. Furthermore, existing network log analysis systems suffer from complex alarm configurations, making it difficult to merge, converge, or reduce the noise associated with alarm information. Consequently, they contain a high number of duplicate alarms. When the system is experiencing a high volume of alarms, tens of thousands of alarms can be recorded daily. This can lead to excessive server load and network congestion, resulting in service anomalies, delayed user access, page load failures, or complete service interruptions, impacting the user experience. Furthermore, increased log storage pressure within the system and the inherent operation of the log analysis system can cause the log analysis function to terminate abnormally. Summary of the Invention

[0003] Based on this, it is necessary to address the above shortcomings and provide a network log analysis system and method that can achieve load balancing of each collection and processing node, merge, converge and reduce noise of alarm information to reduce the impact of duplicate alarms.

[0004] A network log analysis system, comprising:

[0005] The load balancing module is used to load balance the UDP protocol Syslog logs received by the server and forward them to each processing server;

[0006] The data collection module is used to receive the Syslog logs received by each processing server and output the Syslog data;

[0007] The data cache module is used to receive and cache the Syslog data output by the data acquisition module;

[0008] The data processing module is used to read the cached data of the data cache module in real time, parse the meaning of basic log fields, and output the parsing results and original log data;

[0009] A data storage module is used to receive and store the parsing results and original log data when the data processing module does not trigger the alarm condition;

[0010] The streaming processing alarm module is used to receive the parsing results and original log data when the data processing module triggers the alarm condition, set the alarm configuration rules and filter the original log data according to the alarm configuration rules, perform statistical analysis on the filtered data, and trigger the alarm when the statistical analysis results adapt to the alarm configuration rules.

[0011] In one embodiment, the server includes a master node server and a backup node server, wherein the master node server is used to bind to the VIP node and receive Syslog logs, and distribute the Syslog logs to each processing server. The master node server, the backup node server and each processing server together form a load balancing pool;

[0012] The load balancing module includes:

[0013] The LVS load balancing submodule is used to distribute the syslog data received by the network card of the master node server or the backup node server to each processing server through the load balancing algorithm;

[0014] The KL submodule is used to monitor the health status of the master node server, backup node server and each processing server through the VRRP protocol, and when the master node server fails, it will drift the node VIP to the backup node server; when the processing server fails, it will remove the faulty processing server from the load balancing pool.

[0015] In one embodiment, the alarm configuration rules include alarm name, data range, detection time window, alarm condition, aggregation field, alarm threshold, and aggregation period; the streaming alarm module filters the original log data through the alarm conditions, and the alarm conditions include a single keyword or multiple keywords or a custom combination of multiple conditions.

[0016] In one embodiment, the streaming alarm processing module generates a workflow for each alarm name, the workflow including calculation of the detection time window, alarm quantity statistics, and output of data in a merged manner, and each alarm is an independent calculation process.

[0017] In one embodiment, the process of triggering an alarm includes:

[0018] The alarm condition is saved to the data processing module, and the stream processing alarm module generates a workflow corresponding to the alarm condition;

[0019] The data processing module parses the logs and continuously sends the parsing results and original log data to the data storage module. When the log data meets the alarm conditions, the parsing results and original log data are sent to the streaming processing alarm module.

[0020] The streaming processing alarm module receives logs and matches them to the corresponding workflow for streaming processing;

[0021] When the first log entry arrives, the current time is recorded, and statistics are grouped and accumulated by aggregation fields. The statistical data is checked every detection time window to see if it meets the alarm threshold conditions.

[0022] After the alarm threshold is met, an alarm ID and event ID are generated, and it is recorded whether the current time exceeds the aggregation period, and the alarm information is sent to the database.

[0023] In one embodiment, the network log analysis system further includes:

[0024] Cluster scheduling module, used to schedule node task processing of data cache module and streaming processing alarm module;

[0025] Self-monitoring module, used to monitor the resource usage and process survival status of each module;

[0026] The Web configuration presentation module is used to provide Syslog log retrieval, query, statistics and analysis pages, and display the result data.

[0027] In one embodiment, the analysis of Syslog logs includes log asset analysis, log trend analysis, network device basic analysis, network device login analysis, network device change analysis, Linux basic analysis, Linux login analysis, Linux change analysis, Windows login analysis, WAF attack analysis, and IPS / IDS attack analysis.

[0028] The present invention also discloses a network log analysis method, which is implemented using the above network log analysis system and includes the following steps:

[0029] S1, load balances the UDP protocol Syslog logs received by the server and forwards them to each processing server;

[0030] S2. Receive the Syslog logs received by each processing server and output the Syslog data;

[0031] S3, receives and caches Syslog data;

[0032] S4: Read cached data in real time and parse the meaning of basic log fields, outputting the parsing results and original log data;

[0033] S5. When the log data does not trigger an alarm condition, store the parsing results and the original log data; when the log data triggers an alarm condition, set the alarm configuration rules and filter the original log data according to the alarm configuration rules, perform statistical analysis on the filtered data, and trigger an alarm when the statistical analysis results adapt to the alarm configuration rules.

[0034] In one embodiment, the server includes a master node server and a backup node server, wherein the master node server is used to bind to the VIP node and receive Syslog logs, and distribute the Syslog logs to each processing server. The master node server, the backup node server and each processing server together form a load balancing pool;

[0035] Step S1 includes:

[0036] S11, distribute the syslog data received by the network card of the master node server or the standby node server to each processing server through the load balancing algorithm;

[0037] S12. Monitor the health status of the master node server, backup node server, and each processing server through the VRRP protocol. When the master node server fails, the node VIP will be drifted and switched to the backup node server. When the processing server fails, the faulty processing server will be removed from the load balancing pool.

[0038] In one of the embodiments, the alarm configuration rules include alarm name, data range, detection time window, alarm condition, aggregation field, alarm threshold, and aggregation period;

[0039] Step S5 includes:

[0040] S51, saving the alarm condition and generating a workflow corresponding to the alarm condition;

[0041] S52, continuously storing the parsing results and original log data, and when the log data meets the alarm conditions, matching the log to the corresponding workflow for stream processing;

[0042] S53: When the first log entry is received, the current time is recorded, the statistics are grouped and accumulated by the aggregation field, and the statistical data are checked every detection time window to see if they meet the alarm threshold conditions;

[0043] S54: After the alarm threshold is met, an alarm ID and event ID are generated, and whether the current time exceeds the aggregation period is recorded, and the alarm information is sent to the database.

[0044] The network log analysis system and method of the present invention balance the load of received Syslog logs and forward them to each processing server, making full use of the resources of each server, avoiding server failure problems caused by excessive processing pressure on a single server, achieving high availability of the system, ensuring the normal operation of the system and effective analysis of network logs; filtering and statistical analysis of log data that triggers alarm conditions through a streaming processing alarm module, and merging, converging, and reducing noise on alarm information using a streaming processing method, reducing the number of alarms, reducing server load, alleviating network congestion and server anomalies, solving user access delays, page loading failures, or complete service interruptions caused by an excessive number of alarms, improving the user's Internet experience, and at the same time reducing the log storage pressure in the system, ensuring the normal operation of the log analysis system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the module structure of a network log analysis system according to an embodiment of the present invention;

[0046] Figure 2 This is a diagram of the data collection load balancing architecture of a network log analysis system in one embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0048] Please combine Figure 1 and Figure 2The present invention discloses a network log analysis system that can achieve load balancing of each collection and processing node, merge, converge, and reduce noise of alarm information to reduce the impact of repeated alarms. The network log analysis system relies on a cluster composed of multiple servers for data processing, and the cluster includes multiple servers and multiple processing servers. Specifically, the network log analysis system of this embodiment includes a load balancing module 100, a data collection module 200, a data cache module 300, a data processing module 400, a data storage module 500, a streaming processing alarm module 600, a cluster scheduling module 700, a self-monitoring module 800, a Web configuration presentation module 900, and a database 910. Among them, the load balancing module 100 is used to load balance the UDP protocol Syslog logs received by the server and forward them to each processing server. The data collection module 200 is used to receive Syslog logs from each processing server and output the Syslog data. Data collection module 200 can be one of Rsyslog (a high-performance Syslog server for Linux that supports TCP / UDP, TLS encryption, log filtering, and forwarding), Syslog-ng (which supports structured logs (such as JSON) and multi-destination forwarding (databases and message queues)), Logstash (an input plugin that receives logs through the syslog input plugin and supports parsing and enrichment before outputting to ES, Hadoop, etc.), or Fluentd (a Syslog plugin that collects logs through the in_syslog plugin and integrates with Kubernetes and cloud-native environments). Data caching module 300 is used to receive and cache the Syslog data output by data collection module 200. In this embodiment, data caching module 300 can be one of Redis, Kafka, a local file buffer (such as the disk queue of rsyslog or syslog-ng), or a dedicated log agent (such as Fluentd or Logstash). The data processing module 400 is used to read the cached data of the data cache module 300 in real time and parse the meaning of the basic log fields, and output the parsing results and the original log data. The data processing module 400 can be a log parser or a field mapper. The meaning of the basic log fields includes information such as the log level, time, and the associated asset organization, responsible person, etc. After the data processing module 400 parses the meaning of the basic log fields, it outputs the parsing results and the original log data to the data storage module 500 or the streaming processing alarm module 600. The data storage module 500 is used to receive and store the parsing results and the original log data when the data processing module 400 does not trigger the alarm condition.The stream processing alarm module 600 is used to receive parsing results and raw log data when an alarm condition is triggered by the data processing module 400, set alarm configuration rules, filter the raw log data according to the alarm configuration rules, perform statistical analysis on the filtered data, and trigger an alarm when the statistical analysis results match the alarm configuration rules. The cluster scheduling module 700 is used to schedule node task processing for the data cache module 300 and the stream processing alarm module 600. The cluster scheduling module 700 can be implemented using Apache Mesos (a common cluster resource manager) or YARN (Hadoop's resource management system), or a dedicated scheduler such as a custom scheduler, queue-based scheduling, or a hybrid scheduling strategy. The self-monitoring module 800 is used to monitor the resource utilization and process liveness of each module. The self-monitoring module 800 can monitor each module through distributed tracing methods such as Jaeger and Zipkin, alarm tools such as Alertmanager and PagerDuty, regular heartbeat packet transmission to monitor system activity, or metrics collection tools such as Prometheus and Telegraf. The web configuration presentation module 900 provides Syslog retrieval, query, statistics, and analysis pages, and displays the resulting data. The database 910 receives and stores alarm information sent by the streaming alarm module 600 and transmits it to the web configuration presentation module 900. The web configuration presentation module 900 can be a PC display or other display device.

[0049] In this embodiment, the network log analysis system is configured with an alarm condition. This condition is written into the data processing module 400 and serves as a processing branch of the data processing module 400. Specifically, while parsing the meaning of the basic log fields, the data processing module 400 determines whether the log data meets the alarm condition, thereby outputting the processed log data to the corresponding module. When the alarm condition is triggered during processing, the data processing module 400 sends the parsed log information and the original log to the streaming alarm processing module 600.

[0050] Syslog is a standardized protocol and log format used to transmit and store system management, security event, and operation logs between network devices, servers, and applications. It is simple and widely compatible, making it a universal logging language in the field of operations and maintenance. The UDP protocol is a connectionless, lightweight transport layer protocol on the Internet that enables rapid data transmission. In this embodiment, UDP is used to transmit Syslog logs, which not only enables rapid transmission of network log data but also expands the scope of application of network log analysis systems.

[0051] In one embodiment, the server includes a master node server and a backup node server (one master node server and one backup node server). The master node server is used to bind to a VIP (a virtual IP node providing services to the outside world) node and receive Syslog logs, distributing Syslog logs to each processing server. The number of processing servers is three, and the data acquisition module 200 is arranged in the processing server. The master node server, the backup node server, and each processing server together form a load balancing pool. In this embodiment, the master node server refers to a scheduling server in this system that is used to receive syslog logs on a long-term basis and distribute them to each processing server and is bound to the VIP node. It is used to distribute the log data received by the virtual IP node (VIP node) providing services to the outside world to the three processing servers at the back end. The backup node server is used to bind the VIP node to the backup node server by the load balancing module 100 when the master node server goes down. The backup node server takes over the master node server's reception of syslog logs and performs scheduling work to ensure the continuous and stable operation of the system. The master node server and the backup node server are used for scheduling services, and each processing server is used for data processing.

[0052] In this embodiment, the load balancing module 100 includes an LVS load balancing submodule and a KL submodule. The LVS (Linux Virtual Server) load balancing submodule is a high-performance, open-source, Layer 4 (transport layer) load balancing module 100 based on the Linux kernel. It achieves traffic balancing, high availability, and scalability by distributing client requests to multiple backend servers (real servers). The LVS load balancing submodule is used to distribute syslog data received by the network interface cards (NICs) of the primary or backup servers to each processing server using a load balancing algorithm. Specifically, the LVS load balancing submodule uses DR (Direct Routing) mode to distribute syslog data received by the server NICs to the processing servers using load balancing algorithms such as round-robin and weighting. The data collection module 200 of the processing server receives and processes the data. DR mode is a high-performance Layer 4 (transport layer) load balancing solution commonly used in Linux Virtual Server (LVS) architectures. Its core feature is that data packets are returned directly from the real servers to the client, eliminating the load balancer (director) from becoming a performance bottleneck. It is suitable for high-concurrency, low-latency scenarios.

[0053] In this embodiment, the LVS load balancing submodule is a load balancer (Director). The load balancer is responsible for receiving client requests and forwarding the requests to the back-end server (processing server) according to the scheduling algorithm. The load balancer is configured with VIP (a virtual IP that provides services to the outside world) and DIP (an IP that communicates with the back-end server (processing server)). In this embodiment, the final processing end of the LVS load balancing submodule is the processing server that receives the syslog log. The processing server is the server that actually processes the request. Each processing server is also configured with a VIP (a virtual IP that provides services to the outside world), but ARP suppression technology is needed to avoid IP conflicts. In this case, the processing server can directly respond to the client request, and the response data packet does not pass through the load balancer. ARP suppression technology reduces unnecessary ARP (Address Resolution Protocol) traffic in the network through cache control, request filtering, or corresponding optimization, thereby improving network efficiency and achieving network optimization.

[0054] The KL (Keepalived) submodule is a high-availability processing module based on the VRRP (Virtual Router Redundancy Protocol) protocol. It primarily implements automatic failover and load balancing for service failures. In network logs, Keepalived log entries typically relate to its health checks, VRRP state transitions, virtual IP management, and other activities. In this embodiment, the KL submodule is used to achieve high availability for each server in the system. It implements active / standby failover using the VRRP protocol and ensures the availability of processing servers through health checks. Specifically, the KL submodule monitors the health of the primary, standby, and processing servers using VRRP. If a primary server fails, it will shift the node VIP to the standby server, effectively implementing active / standby failover. If a processing server fails, it will remove the failed processing server from the load balancing pool. Thus, through the coordinated efforts of the LVS load balancing submodule and the KL submodule, load balancing is achieved across all acquisition and processing nodes while enabling automatic failover between LVS active and standby nodes, improving system reliability.

[0055] Please further combine Figure 1 and Figure 2During the operation of the load balancing module 100, network devices or servers first send Syslog logs to the VIP node (i.e., the primary node server) of the network log analysis system. The LVS master node (primary node server), configured with a VIP (a virtual IP address providing external services), distributes the Syslog logs to the data processing server nodes using an algorithm. The LVS load balancing submodule distributes the requests to the processing servers, while the KL submodule monitors the status of the LVS primary and backup nodes (primary and backup node servers) and backend servers. If the LVS master node (primary node server) fails, the KL submodule switches the VIP node to the backup node (backup node server). If a processing server fails, the KL submodule removes it from the load balancing pool.

[0056] In one embodiment, the alarm configuration rules include an alarm name, data range, detection time window, alarm condition, aggregation field, alarm threshold, and aggregation period. The streaming alarm processing module 600 filters the raw log data based on the alarm conditions, accumulates the data that meets the conditions, and triggers an alarm. Alarm conditions can include a single keyword, multiple keywords, or a custom combination of multiple conditions. During network log analysis, the streaming alarm processing module 600 generates a workflow for each alarm name. The time range window, trigger count, and merge method in the alarm configuration rules all correspond to a portion of the streaming processing logic in the workflow. After receiving log data, the streaming alarm processing module 600 matches the log data according to the alarm configuration rules. Because the received log data contains a mixture of conditions from multiple alarm rules, it needs to be filtered again. If a log entry meets multiple alarm conditions, the streaming alarm processing module 600 can create a copy of the log entry and simultaneously execute multiple processes. After distinguishing the name to which the alarm belongs, the designated alarm workflow is entered. The workflow includes calculation of the detection time window, alarm quantity statistics, and output of data in a merged manner. Each alarm is an independent calculation process.

[0057] In this embodiment, the process of triggering an alarm includes:

[0058] The alarm condition is saved to the data processing module 400, and at the same time, the stream processing alarm module 600 generates a workflow corresponding to the alarm condition.

[0059] The data processing module 400 parses the log and continuously sends the parsing results and original log data to the data storage module 500 . When the log data meets the alarm condition, the parsing results and original log data are sent to the streaming alarm module 600 .

[0060] The stream processing alarm module 600 receives logs and matches the logs to corresponding workflows for stream processing.

[0061] When the first log entry arrives, the current time is recorded, and statistics are grouped and accumulated by aggregation fields. The statistical data is checked every detection time window to see if it meets the alarm threshold conditions.

[0062] After the alarm threshold is met, an alarm ID and event ID are generated, and it is recorded whether the current time exceeds the aggregation period, and the alarm information is sent to the database 910.

[0063] The following describes the process of triggering an alarm with a specific example.

[0064] First, set the alarm configuration rules as follows: Alarm name: keyword error alarm; Data range: last 5 minutes; Detection time window: every 1 minute; Alarm condition: keyword "error"; Aggregation field: asset IP; Alarm threshold: 5 times; Aggregation period: 1 day.

[0065] The alarm process of network log analysis is as follows:

[0066] Step 1: Save the alarm condition "error" to the data processing module 400. At the same time, the streaming alarm module 600 generates a processing flow of "keyword error alarm".

[0067] Step 2: The data processing module 400 parses the log and continuously sends the parsing results and the original log data to the data storage module 500. When the log data meets the keyword "error", the parsing results and the original log data are sent to the streaming alarm module 600.

[0068] Step 3: The streaming alarm module 600 receives the log and matches the log to the "error alarm" process, then the log enters the "error alarm" process for processing.

[0069] Step 4: When the first log entry arrives, record the current time, group and accumulate the statistics by "Asset IP," and check every minute to see if the statistics meet the alarm threshold. Each check is based on data entered within the previous 5 minutes. If four subsequent log entries with the same asset IP address arrive, that totals five, thus meeting the alarm threshold five times.

[0070] Step 5: After the alarm threshold is met, an alarm ID and event ID are generated, and whether the current time exceeds the aggregation period is recorded. The alarm information is then sent to database 910. In this embodiment, the alarm information includes information such as the aggregation field, alarm count, start time, and end time. Within the aggregation period, one event ID includes multiple different alarm IDs, and the event details for one event ID include the time, IP address, and number of each alarm. Thus, within one aggregation period, the alarm displayed on the web configuration presentation module 900 is a single event, and the event details include the time, IP address, and number of each alarm, thus achieving noise reduction for alarms.

[0071] In one embodiment, the database 910, data cache module 300, and streaming alarm module 600 are each equipped with a cluster scheduling module 700. This cluster scheduling module 700 is responsible for scheduling task processing for the master node (the master node, used to assign tasks to worker nodes) and the worker node (the worker node, used to run specific services) of these modules. In this embodiment, both the data cache module 300 and the streaming alarm module 600 use ZooKeeper for cluster management and scheduling. That is, the cluster scheduling module 700 is a ZooKeeper module. ZooKeeper is a distributed coordination service that provides consistency, reliability, and efficient management capabilities for distributed systems. ZooKeeper's scheduling process includes:

[0072] 1) Client request: The client sends a request to the Zookeeper cluster, such as reading or writing data.

[0073] 2) Request forwarding: A request first arrives at a ZooKeeper server (leader or follower). If a follower receives the request, it forwards the write request to the leader. The leader coordinates all write operations (such as create, update, and delete) in the ZooKeeper cluster and is the master node of the ZooKeeper server. Followers, representing the majority of the nodes in the ZooKeeper cluster, handle client read requests and participate in leader election and proposal voting. They are the slave nodes of the ZooKeeper server.

[0074] 3) Leader processing: After receiving a write request, the leader generates a transaction proposal and broadcasts it to all followers.

[0075] 4) Follower confirmation: After receiving the proposal, the follower writes it to the local log and sends a confirmation (ack) to the leader.

[0076] 5) Committing the transaction: When the leader receives confirmation from the majority of followers, it commits the transaction and notifies all followers to commit.

[0077] 6) Respond to the client: The leader or the follower that receives the request returns the result to the client.

[0078] 7) Data synchronization: Followers regularly synchronize data with leaders to ensure consistency.

[0079] In one embodiment, the Web configuration presentation module 900 can be a display screen of a PC or other display devices. The display interface of the Web configuration presentation module 900 supports visual methods such as "drag and drop, drop-down box selection" to implement multi-keyword alarm rule configuration. The streaming alarm processing module 600 issues streaming tasks based on various alarm rules. These tasks filter the original logs of the equipment by matching a single keyword or multiple keywords, or a custom combination of multiple conditions, and then perform statistics and analysis on the data that meets the conditions. After hitting the adapted alarm rule, the alarm push is triggered to achieve alarm merging, convergence and noise reduction. The alarms of the same asset are merged into one alarm event, and only one alarm is issued for one asset per day. The earliest time and the latest time can also be analyzed. Similarly, before the alarm rules are adapted, the retrieval and analysis of multiple keywords combined can help understand the log information of each device under abnormal conditions, help establish log information models for various types of equipment under abnormal conditions, and provide data basis for early warning.

[0080] In this embodiment, the analysis of Syslog logs on the display interface of the Web configuration presentation module 900 includes comprehensive analysis, network device analysis, host device analysis, and security device attack analysis. Among them, the comprehensive analysis includes log asset analysis and log trend analysis. The network device analysis includes network device basic analysis, network device login analysis, and network device change analysis. The host device analysis includes Linux basic analysis, Linux login analysis, Linux change analysis, and Windows login analysis. The security device attack analysis includes WAF attack analysis and IPS / IDS attack analysis.

[0081] Specifically, log asset analysis includes statistics by asset category, asset department, top 10 EMERG assets, top 10 ALERT assets, top 10 CRIT assets, top 10 ERROR assets, top 10 WARN assets, top 10 INFO assets, and top 10 DEBUG assets. Log trend analysis includes trends in total log volume, EMERG volume, ALERT volume, CRIT volume, ERROR volume, WARN volume, INFO volume, and DEBUG volume. Network device basic analysis includes statistics on network log volume by asset IP, network log volume by asset category, today's real-time network log volume trend, 30-day network log volume trend, firewall log level statistics, recent 30-day firewall log volume trend, switch log level statistics, recent 30-day switch log volume trend, router log level statistics, recent 30-day router log volume trend, NAT log level statistics, recent 30-day NAT log volume trend, DNS log level statistics, and recent 30-day DNS log volume trend. Network device login analysis includes today's successful login trends, today's failed login trends, successful login trends over the past 30 days, failed login trends over the past 30 days, successful login user statistics, failed login user statistics, successful login source IP statistics, failed login source IP statistics, successful login asset name statistics, and failed login asset name statistics. Network device change analysis includes today's change operation trends, past 30-day change operation trends, change user statistics, asset name change statistics, and change operation category statistics. Linux basic analysis includes today's real-time Linux log volume trends, past 30-day Linux log volume trends, Linux log host IP statistics, Linux log level statistics, Linux operation user statistics, Linux high-risk operation command distribution, and Linux severity level process statistics. Linux login analysis includes today's successful login trends, today's failed login trends, past 30-day successful login trends, past 30-day failed login trends, successful login user statistics, failed login user statistics, successful login source IP statistics, failed login source IP statistics, successful login asset name statistics, and failed login asset name statistics. Linux change analysis includes today's change operation trends, past 30-day change operation trends, change user statistics, asset name change statistics, and change operation category statistics. Windows login analysis includes today's successful login trend, today's failed login trend, recent 30-day successful login trend, recent 30-day failed login trend, successful login user statistics, failed login user statistics, successful login source IP statistics, failed login source IP statistics, successful login asset name statistics, failed login asset name statistics.WAF attack analysis includes today's real-time WAF log volume trends, past 30-day WAF log volume trends, WAF log device IP statistics, WAF attack source IP statistics, WAF attacked destination IP statistics, WAF attack type statistics, WAF attacked site statistics, WAF attacked URL statistics, WAF attack interception status statistics, WAF attack protocol statistics, and WAF attack level statistics. IPS / IDS attack analysis includes today's real-time IPS / IDS log volume trends, past 30-day IPS / IDS log volume trends, IPS / IDS log device IP statistics, IPS / IDS attack source IP statistics, IPS / IDS attacked destination IP statistics, IPS / IDS attack type statistics, IPS / IDS attack event statistics, IPS / IDS attack interception status statistics, and IPS / IDS attack protocol statistics.

[0082] In addition, the present invention also discloses a network log analysis method, which is implemented using the above-mentioned network log analysis system and includes the following steps:

[0083] S1. Load balance the UDP protocol Syslog logs received by the server and forward them to each processing server.

[0084] In this embodiment, the network log analysis method relies on a cluster composed of multiple servers for data processing, and the cluster includes multiple servers and multiple processing servers. The servers include a master node server and a backup node server (one master node server and one backup node server). The master node server is used to bind to the VIP (virtual IP that provides services to the outside world) node and receive Syslog logs, and distribute the Syslog logs to each processing server. The master node server, the backup node server and each processing server together form a load balancing pool. The master node server refers to a scheduling server in this system that is used to receive syslog logs for a long time and distribute them to each processing server and is bound to the VIP node. It is used to distribute the log data received by the virtual IP node (VIP node) that provides services to the outside world to the three processing servers at the back end; the backup node server is used to bind the VIP node to the backup node server when the master node server goes down, and the backup node server takes over the master node server to receive syslog logs and perform scheduling work to ensure the continuous and stable operation of the system.

[0085] Step S1 includes:

[0086] S11. Distribute the syslog data received by the network card of the master node server or the standby node server to each processing server through a load balancing algorithm.

[0087] S12. Monitor the health status of the master node server, backup node server, and each processing server through the VRRP protocol. When the master node server fails, the node VIP will be drifted and switched to the backup node server. When the processing server fails, the faulty processing server will be removed from the load balancing pool.

[0088] In this embodiment, LVS load balancing is combined with Keepalived to achieve load balancing across all collection and processing nodes while also enabling automatic switchover between active and standby LVS nodes, improving system reliability. Specifically, LVS (Linux Virtual Server) load balancing is a high-performance, open-source, four-layer (transport layer) solution based on the Linux kernel. It distributes client requests to multiple backend servers (real servers) to achieve traffic balancing, high availability, and scalability. In this embodiment, the LVS master node uses DR mode, distributing syslog data received by the server network interface cards to the processing servers using load balancing algorithms such as round-robin and weighting. Keepalived is a high-availability solution based on the VRRP (Virtual Router Redundancy Protocol) protocol, designed to achieve automatic failover and load balancing. Keepalived uses VRRP to implement active / standby switchover and ensures the availability of processing servers through health checks. The load balancing process can be found in the operating process of the load balancing module 100 in the network log analysis system and will not be further described here.

[0089] S2. Receive the Syslog logs received by each processing server and output the Syslog data.

[0090] Syslog is a standardized protocol and log format used to transmit and store system management, security event, and operation logs between network devices, servers, and applications. It is simple and widely compatible, making it a universal logging language in the field of operations and maintenance. The UDP protocol is a connectionless, lightweight transport layer protocol on the Internet that enables rapid data transmission. In this embodiment, UDP is used to transmit Syslog logs, which not only enables rapid transmission of network log data but also expands the scope of application of network log analysis systems.

[0091] In this embodiment, receiving Syslog logs from the processing server and outputting Syslog data are achieved through one of Rsyslog (a high-performance Syslog server under Linux, supporting TCP / UDP, TLS encryption, log filtering and forwarding), Syslog-ng (supporting structured logs (such as JSON), multi-destination forwarding (database 910, message queue)), Logstash (Input plug-in, receiving logs through the syslog input plug-in, supporting parsing and enrichment before output to ES, Hadoop, etc.), and Fluentd (Syslog plug-in, collecting logs through the in_syslog plug-in, integrating Kubernetes and cloud native environments).

[0092] S3 receives and caches Syslog data. Syslog data caching is implemented using Redis, Kafka, a local file buffer (such as the disk queue of rsyslog or syslog-ng), or a dedicated log agent (such as Fluentd / Logstash).

[0093] S4. Read cached data in real time and parse the meaning of basic log fields, outputting the parsed results and original log data. The parsing of the meaning of basic log fields is achieved through a log parser or field mapper.

[0094] In step S4, before reading the cached data, an alarm condition is configured. In this way, while parsing the meaning of the basic log fields, it is possible to determine whether the log data meets the alarm condition, so that when the alarm condition is triggered, the parsed log information and the original log are streamed.

[0095] S5. When the log data does not trigger the alarm condition, the parsing results and the original log data are stored; when the log data triggers the alarm condition, the alarm configuration rules are set and the original log data is filtered according to the alarm configuration rules, the filtered data is statistically analyzed, and the alarm is triggered when the statistical analysis results adapt to the alarm configuration rules. The alarm configuration rules include the alarm name, data range, detection time window, alarm condition, aggregation field, alarm threshold, and aggregation period. During streaming processing, the original log data is filtered by the alarm condition, the data that meets the conditions is accumulated, and the alarm is triggered. The alarm conditions include a single keyword or multiple keywords or a custom combination of multiple conditions. When analyzing network logs, a workflow is generated for each alarm name. The workflow includes the calculation of the detection time window, the statistics of the number of alarms, and the output of data in a merged manner. Each alarm is an independent calculation process. The time range window, the number of triggers, and the merge method in the alarm configuration rules all correspond to a part of the streaming processing logic in the workflow. Step S5 includes:

[0096] S51. Save the alarm condition and generate a workflow corresponding to the alarm condition.

[0097] S52: Continuously store the parsing results and original log data. When the log data meets the alarm condition, match the log to the corresponding workflow for stream processing.

[0098] S53. When the first log is entered, the current time is recorded, and the statistics are grouped and accumulated according to the aggregation field. The statistical data is checked every detection time window to see whether it meets the alarm threshold condition.

[0099] S54: After the alarm threshold is met, an alarm ID and event ID are generated, and whether the current time exceeds the aggregation period is recorded, and the alarm information is sent to the database.

[0100] The following describes the process of triggering an alarm with a specific example.

[0101] First, set the alarm configuration rules as follows: Alarm name: keyword error alarm; Data range: last 5 minutes; Detection time window: every 1 minute; Alarm condition: keyword "error"; Aggregation field: asset IP; Alarm threshold: 5 times; Aggregation period: 1 day.

[0102] The alarm process of network log analysis is as follows:

[0103] Step 1: Save the alarm condition "error" and generate a processing flow for "keyword error alarm".

[0104] Step 2: Parse the log and continuously save the parsing results and original log data. When the log data meets the keyword "error", match the log to the "error alarm" process for streaming processing.

[0105] Step 3: When the first log entry arrives, record the current time, group and accumulate the statistics by "Asset IP," and check every minute to see if the statistics meet the alarm threshold. Each check is based on data entered within the previous 5 minutes. If four subsequent log entries with the same asset IP address arrive, that totals five, thus meeting the alarm threshold five times.

[0106] Step 4: After the alarm threshold is met, an alarm ID and event ID are generated, and whether the current time exceeds the aggregation period is recorded. The alarm information is then sent to the database. In this embodiment, the alarm information includes information such as the aggregation field, alarm count, start time, and end time. Within the aggregation period, one event ID can include multiple different alarm IDs, and the event details for one event ID include the time, IP address, and number of alarms for each alarm. Thus, within one aggregation period, the alarm displayed on the page is a single event, and the event details include the time, IP address, and number of alarms for each alarm, thus achieving noise reduction for the alarms.

[0107] The network log analysis system and method of the present invention are implemented to load balance the received Syslog logs and forward them to each processing server, making full use of the resources of each server, avoiding server failure problems caused by excessive processing pressure on a single server, achieving high availability of the system, ensuring the normal operation of the system and effective analysis of network logs; the stream processing alarm module 600 is used to filter and statistically analyze the log data that triggers the alarm condition, and adopts a stream processing method to merge, converge, and reduce the noise of the alarm information, thereby reducing the number of alarms, reducing the server load, alleviating network congestion and server anomalies, and solving the problems of user access delays, page loading failures or complete service interruptions caused by too many alarms, improving the user's Internet experience, and at the same time reducing the log storage pressure in the system, ensuring the normal operation of the log analysis system.

[0108] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A network log analysis system, characterized in that: include: The load balancing module is used to load balance the UDP protocol Syslog logs received by the server and forward them to each processing server; The data collection module is used to receive the Syslog logs received by each processing server and output the Syslog data; The data cache module is used to receive and cache the Syslog data output by the data acquisition module; The data processing module is used to read the cached data of the data cache module in real time, parse the meaning of basic log fields, and output the parsing results and original log data; A data storage module is used to receive and store the parsing results and original log data when the data processing module does not trigger the alarm condition; The streaming alarm module is used to receive the parsing results and original log data when the data processing module triggers an alarm condition, set alarm configuration rules and filter the original log data according to the alarm configuration rules, perform statistical analysis on the filtered data, and trigger an alarm when the statistical analysis results match the alarm configuration rules; If a log meets multiple alarm conditions, you can copy the log in the streaming alarm module and perform multiple processes simultaneously. After distinguishing the alarm name, enter the specified alarm workflow; Each processing server is configured with a VIP, and ARP suppression technology is required to avoid IP conflicts. The processing server can directly respond to client requests, and the response data packets do not pass through the load balancer; The streaming alarm module filters the original log data according to the alarm conditions, accumulates the data that meets the conditions, and triggers the alarm; The process of triggering an alarm includes: The alarm condition is saved to the data processing module, and the stream processing alarm module generates a workflow corresponding to the alarm condition; The data processing module parses the logs and continuously sends the parsing results and original log data to the data storage module. When the log data meets the alarm conditions, the parsing results and original log data are sent to the streaming processing alarm module. The streaming processing alarm module receives logs and matches them to the corresponding workflow for streaming processing; When the first log entry arrives, the current time is recorded, and statistics are grouped and accumulated by aggregation fields. The statistical data is checked every detection time window to see if it meets the alarm threshold conditions. After the alarm threshold is met, an alarm ID and event ID are generated, and whether the current time exceeds the aggregation period is recorded, and the alarm information is sent to the database; The load balancing module includes an LVS load balancing submodule and a KL submodule. The KL submodule is used to remove the faulty processing server from the load balancing pool when a processing server fails. The alarm configuration rules include alarm name, data range, detection time window, alarm condition, aggregation field, alarm threshold, and aggregation period; alarm conditions include a single keyword or a custom combination of multiple keywords or multiple conditions, data range: last 5 minutes; detection time window: every 1 minute; aggregation field: asset IP; alarm threshold: 5 times; aggregation period: 1 day.

2. The network log analysis system according to claim 1, characterized in that The server includes a master node server and a backup node server. The master node server is used to bind the VIP node and receive the Syslog log, and distribute the Syslog log to each processing server. The master node server, the backup node server and each processing server together form a load balancing pool; The LVS load balancing submodule is used to distribute the syslog data received by the network card of the master node server or the backup node server to each processing server through the load balancing algorithm; The KL submodule is used to monitor the health status of the master node server, backup node server and each processing server through the VRRP protocol, and when the master node server fails, it will drift and switch the node VIP to the backup node server.

3. The network log analysis system according to claim 1, characterized in that: The stream processing alarm module generates a workflow for each alarm name. The workflow includes calculation of the detection time window, alarm quantity statistics, and output of data in a merged manner. Each alarm is an independent calculation process.

4. The network log analysis system according to claim 1, wherein: Also includes: Cluster scheduling module, used to schedule node task processing of data cache module and streaming processing alarm module; Self-monitoring module, used to monitor the resource usage and process survival status of each module; The Web configuration presentation module is used to provide Syslog log retrieval, query, statistics and analysis pages, and display the result data.

5. The network log analysis system according to claim 4, characterized in that: Syslog analysis includes log asset analysis, log trend analysis, network device basic analysis, network device login analysis, network device change analysis, Linux basic analysis, Linux login analysis, Linux change analysis, Windows login analysis, WAF attack analysis, and IPS / IDS attack analysis.

6. A network log analysis method, implemented using the network log analysis system according to any one of claims 1 to 5, characterized in that: The following steps are involved: S1, load balances the UDP protocol Syslog logs received by the server and forwards them to each processing server; S2. Receive the Syslog logs received by each processing server and output the Syslog data; S3, receives and caches Syslog data; S4: Read cached data in real time and parse the meaning of basic log fields, outputting the parsing results and original log data; S5. When the log data does not trigger an alarm condition, store the parsing results and the original log data; when the log data triggers an alarm condition, set alarm configuration rules and filter the original log data according to the alarm configuration rules, perform statistical analysis on the filtered data, and trigger an alarm when the statistical analysis results match the alarm configuration rules; Step S1 includes: S11, distribute the syslog data received by the network card of the master node server or the standby node server to each processing server through the load balancing algorithm; S12. Monitor the health status of the master node server, backup node server, and each processing server through the VRRP protocol. When the master node server fails, the node VIP is drifted to the backup node server. When a processing server fails, the faulty processing server is removed from the load balancing pool. Step S5 includes: S51, saving the alarm condition and generating a workflow corresponding to the alarm condition; S52, continuously storing the parsing results and original log data, and when the log data meets the alarm conditions, matching the log to the corresponding workflow for stream processing; S53: When the first log entry is received, the current time is recorded, the statistics are grouped and accumulated by the aggregation field, and the statistical data are checked every detection time window to see if they meet the alarm threshold conditions; S54: After the alarm threshold is met, an alarm ID and event ID are generated, and whether the current time exceeds the aggregation period is recorded, and the alarm information is sent to the database.

7. The network log analysis method according to claim 6, characterized in that: The server includes a master node server and a backup node server. The master node server is used to bind the VIP node and receive Syslog logs, and distribute the Syslog logs to each processing server. The master node server, the backup node server and each processing server together form a load balancing pool.

8. The network log analysis method according to claim 6, characterized in that: The alarm configuration rules include alarm name, data range, detection time window, alarm condition, aggregation field, alarm threshold, and aggregation period.

Images