Safety information event management system and method based on Graylog

Abstract

The invention discloses a safety information event management system and method based on Graylog. The system comprises a log access module, a log index module, a field analysis module, a stream distribution module, an event detection and alarm module and a notification module. The log access module is used for receiving log data from a plurality of log sources, the log index module is used for establishing index sets for different applications and executing storage and alternation, and the field analysis module is used for carrying out multi-stage analysis on logs to extract fields related to operation behaviors, network addresses, application names and security levels; the stream distribution module routes logs to corresponding stream channels according to a preset rule, and the event detection and alarm module generates events according to the rule and a threshold value and generates alarm information through a template. According to the method, refined analysis, classification and alarm pushing of the log data are realized, the event detection accuracy and the notification reliability are improved, and the method is suitable for network security monitoring and operation and maintenance scenes.

Description

Technical Field

[0001] This invention relates to the field of information security and log analysis technology, and in particular to a security information event management system and method based on Graylog. Background Technology

[0002] With the continuous improvement of informatization and networking, enterprises and institutions generate a massive amount of log data from various business systems, servers, and network devices every day, sourced from diverse locations. These logs contain rich information on security incidents, and timely identification and analysis will play a crucial role in anomaly detection, intrusion prevention, and operational management. However, existing log management platforms primarily focus on centralized log collection and retrieval, and generally have the following shortcomings when addressing security incident management: First, log formats are diverse, and traditional platforms have limited ability to parse unstructured logs or nested structures, which makes it impossible to accurately extract key information and affects subsequent rule judgment and automated analysis.

[0003] Secondly, the indexing and storage strategies lack specificity and are difficult to independently divide and flexibly rotate according to different applications or components, which reduces retrieval efficiency and increases the system storage and maintenance burden.

[0004] Third, the event alarm mechanism is relatively simple, mostly relying on simple thresholds or keyword matching, which can easily lead to false alarms or missed alarms, making it difficult to support complex security operation and maintenance needs.

[0005] Fourth, the notification method is not flexible enough, it cannot achieve differentiated push based on event priority, and it lacks secure access control to external instant messaging platforms, which may lead to delays, omissions or even leaks of alarm information.

[0006] Based on the above problems, there is an urgent need for a security information event management system that can form an organic and collaborative process in log collection, parsing, distribution, event detection and notification, so as to achieve refined processing of log data and efficient response to security events, thereby improving the overall security operation and maintenance level.

[0007] Therefore, existing technologies still need to be improved. Summary of the Invention

[0008] In view of the deficiencies of the prior art in log analysis capability, index management flexibility, alarm generation accuracy, and notification delivery reliability, the purpose of the present application is to provide a Graylog-based security information event management system and method. The system forms a full-link processing flow from log collection to alarm delivery through the setting of log access, index set, field analysis, stream distribution, event detection and alarm, and notification pushing function modules, thereby realizing efficient access and structured processing of multi-source logs, improving the accuracy and real-time performance of event detection, and ensuring the safe and reliable delivery of alarm information through a multi-channel notification mechanism, to effectively make up for the defects of the prior art.

[0009] The technical solutions of the present application are as follows: The present application provides a Graylog-based security information event management system, comprising: A log access module for receiving log data from multiple log sources; A log index module for establishing an index set for different applications or components and performing partitioned storage and rotation of log data; A field analysis module for analyzing and extracting multiple structured fields from received logs; A stream distribution module for distributing parsed logs to corresponding stream channels according to predefined rules; An event detection and alarm module for generating events in the stream channels according to rules and thresholds and mapping event fields to alarm templates; A notification module for sending alarms generated based on the templates to an external instant messaging platform and supporting security restrictions and formatted display.

[0010] In one embodiment, the log access module simultaneously supports log input based on text format and structured format.

[0011] In one embodiment, the index set of the log index module independently corresponds to different applications or components, and performs data storage, sharding, and rotation according to a preset strategy.

[0012] In one embodiment, the field analysis module processes log content through a multi-level analysis strategy and extracts fields related to operation behavior, network address, application name, and security level.

[0013] In one embodiment, the stream distribution module classifies logs based on the field analysis results and routes logs to different stream channels according to security priority.

[0014] In one embodiment, the event detection and alarm module aggregates and processes similar events, and classifies alarm information according to event severity.

[0015] In one embodiment, the notification module is configured with a Webhook-based notification unit that supports access restrictions and template-based content generation.

[0016] In one embodiment, the notification module further includes a backup mechanism that automatically switches to other preset channels to complete alarm sending when the preferred notification channel is unavailable.

[0017] Another aspect of the present application also provides a Graylog-based security information event management method, comprising: Receiving logs from multiple log sources; Establishing an index set for different applications or components and setting storage and rotation strategies; Parsing the logs and extracting multiple structured fields; Routing the parsed logs to corresponding flow channels according to rules; Generating events in the flow channels and mapping event fields to alarm templates; Sending alarms according to the templates through external instant messaging platforms.

[0018] In one embodiment, it also includes triggering a backup channel to complete alarm delivery when the alarm sending fails.

[0019] In summary, the present application proposes a Graylog-based security information event management system and method to address the problems of insufficient parsing capability, lack of index flexibility, single alarm mechanism, and unreliable notification method of existing log management platforms. The system realizes a full-link closed loop from data collection to alarm notification through the coordinated work of log access, index set division, field parsing, flow distribution, event detection and alarm, and notification pushing modules. Compared with the prior art, the present application not only significantly improves the refinement of log processing and the accuracy of event detection, but also ensures the safety and reliability of alarm delivery through a multi-channel notification mechanism. Therefore, the present application has obvious advantages in reducing operation and maintenance complexity and improving security response efficiency, and has wide application and promotion value.

[0020] Compared with the prior art, the present application has the following advantages and benefits: Firstly, by setting independent log index sets and combining partition storage and rotation strategies, the present application realizes the refined management of log data by application or component, avoids the problems of low retrieval efficiency and high storage pressure caused by traditional centralized indexes, and thus improves the scalability and long-term stability of the system.

[0021] Secondly, the field analysis module of the present application adopts a multi-level analysis strategy, which can automatically extract key fields related to operation behavior, network address, application name and security level from multi-source and multi-format log data, and realize the deep structuring of log information. This scheme not only improves the analysis accuracy, but also lays a reliable data foundation for subsequent flow distribution and event generation, which is difficult to achieve simultaneously in the prior art.

[0022] Thirdly, the flow distribution and event detection module of the present application can classify logs based on multi-dimensional rules, and aggregate and classify events, thereby significantly reducing the false positive rate and false negative rate of alarms. Compared with the traditional single threshold matching method, the present application not only improves the accuracy and controllability of alarms, but also enhances the adaptability of the system to complex security scenarios.

[0023] In addition, the notification module of the present application not only supports template-based formatted alarm information generation, but also combines access restrictions, priority control and backup mechanisms to ensure that alarms are quickly delivered under safe and reliable conditions. Even in the case of unavailable main notification channels, the system can still complete alarm pushing through backup channels, effectively improving the reliability of event response.

[0024] In summary, the present application realizes the upgrading of the log management platform to the security information event management system through the organic combination of log collection, analysis, indexing, distribution, event detection and notification, and has unexpected technical advantages, i.e. significantly improving the detection accuracy and response efficiency of security events while reducing system complexity and maintenance cost. BRIEF DESCRIPTION OF DRAWINGS

[0025] The present application will be further described below in conjunction with the drawings and embodiments, in which: Figure 1 The process of the security information event management system based on Graylog provided by the present application; Figure 2 The system architecture diagram of the security information event management system based on Graylog provided by the present application. DETAILED DESCRIPTION

[0026] To make the purpose, technical scheme and effect of the present application more clear and explicit, the present application will be further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. The embodiments of the present application will be introduced below in conjunction with the drawings.

[0027] The security information event management system based on Graylog provided by the present application comprises: A log access module for receiving log data from multiple log sources; The log indexing module is used to create index sets for different applications or components and to partition and rotate log data. The field parsing module is used to parse the received logs and extract multiple structured fields; The stream distribution module is used to distribute the parsed logs to the corresponding stream channels according to predefined rules; The event detection and alarm module is used to generate events in the flow channel according to rules and thresholds and map event fields to alarm templates; The notification module is used to send alarms generated based on the template to an external instant messaging platform, and supports security restrictions and formatted display.

[0028] Specifically, in one embodiment, a Graylog-based security information event management system is provided. Please refer to [link / reference]. Figure 1 , Figure 2 The system is deployed on a high-concurrency log processing framework. First, a log access module aggregates log data from different servers, network devices, and application systems. Then, a log indexing module establishes independent index sets for various applications, ensuring that data from different sources can be partitioned and stored in rotation according to preset time or capacity rules. Before entering the index, log data is automatically identified and extracted by a field parsing module, forming structured fields related to user operations, network connection information, and application running status, creating a unified data description. After parsing, a stream distribution module routes data to the corresponding stream channel according to predefined rules, achieving distributed management of different types of events. Based on this, an event detection and alarm module monitors each stream channel in real time, generates events based on set thresholds, and maps key fields to pre-defined alarm templates. Finally, a notification module sends alarm information to an external instant messaging platform, ensuring the security and accuracy of information transmission through access restrictions and formatted display. This embodiment demonstrates an end-to-end closed-loop process from log collection to alarm notification, ensuring the system's practicality and engineering feasibility.

[0029] In another embodiment, the log access module supports log input based on both text and structured formats.

[0030] Specifically, the log access module can support log input based on traditional text format and structured format at the same time, thereby being compatible with output characteristics of different log sources. For example, part of the network device output log appears in the form of pure text, while the application program may generate records in the form of key-value pairs or nested structure. By configuring a multi-channel input mechanism in the access module, unified access and synchronous processing of the two types of logs can be realized, avoiding dependence on a single format. At the same time, the access module can perform basic verification and timestamp correction in the initial stage of log entering the system, ensuring data consistency in subsequent processing links. The design enables the system to maintain compatibility with historical devices and support efficient input of modern applications.

[0031] In a further embodiment, the index set of the log index module corresponds to different applications or components independently, and performs data storage, sharding and rotation according to a preset strategy.

[0032] Further, in the log index module, the system establishes independent index sets for different applications or components, and specifies a storage strategy for each set. The strategy can include the number of partitions, the number of replicas and the rotation rules, so that log data can be automatically archived and rotated according to time period or capacity threshold. For example, when processing a key business system, the system can establish an index set for its log alone and set a higher storage priority, so as to quickly locate when forensics or traceability is needed. Compared with the traditional unified index mode, this scheme improves the data retrieval efficiency and reduces the overall maintenance complexity of the system.

[0033] In a further embodiment, the field parsing module processes log content through a multi-level parsing strategy and extracts fields related to operation behavior, network address, application name and security level.

[0034] Specifically, in the specific implementation of the field parsing module, the system adopts a multi-level parsing strategy. The first layer of parsing extracts basic fields for common log formats, and the second layer of parsing faces nested structures or complex formats and extracts fields related to operation behavior, source address, target address, application name and priority. Through such a hierarchical strategy, the system can maintain processing speed while taking accuracy into account. The parsed fields are not only written into the event database, but also used for subsequent stream distribution and rule determination. For example, when the log contains multiple layers of nesting, the parsing module can expand and extract key values layer by layer, avoiding information omission. This design ensures the deep structuring of log data and improves the fineness of overall analysis.

[0035] In a further embodiment, the stream distribution module classifies logs based on the field parsing results and routes logs to different stream channels according to security priority.

[0036] Specifically, in the stream distribution module, the system classifies logs according to field parsing results. Administrators can set multi-dimensional conditions in rule configurations, such as application source, event level, or user behavior characteristics. When logs meet specific conditions, the system automatically routes them to corresponding stream channels. Each stream channel can be configured with independent processing strategies, such as event aggregation, frequency limitation, or priority marking. Taking the intrusion detection scenario as an example, when parsed logs meet high-risk event conditions, the stream distribution module can classify them into high-priority channels, so that subsequent modules can respond quickly. This mechanism enables the system to have the ability to differentiateially process complex security events.

[0037] In further embodiments, the event detection and alarm module aggregates similar events for processing, and classifies alarm information according to event severity.

[0038] Specifically, in the event detection and alarm module, the system monitors logs from various stream channels in real time. When accumulated data meets a set threshold or matches a specific pattern, the system automatically generates an event. At the same time of generating the event, the system maps event fields to placeholders in the alarm template according to the mapping relationship, thereby forming a complete alarm message. To avoid redundancy, the system can also aggregate similar events, integrating multiple similar logs occurring within a short period of time into one high-confidence alarm record. This not only reduces the workload of operation and maintenance personnel, but also improves the readability and actual reference value of alarms.

[0039] In further embodiments, the notification module is configured with a Webhook-based notification unit that supports access restriction and template-based content generation.

[0040] Specifically, in the specific implementation of the notification module, the system is configured with a Webhook-based notification unit for pushing alarm information to external instant messaging platforms. During transmission, the system restricts the source of the request to ensure that only trusted network environments can receive alarms. In addition, the notification unit formats and displays alarm information based on a pre-set template, such as highlighting source addresses, target addresses, and event levels, so that recipients can understand key information at the first time. This design ensures the security and operability of alarm transmission.

[0041] In further embodiments, the notification module further includes a backup mechanism that automatically switches to other pre-set channels to complete alarm transmission when the preferred notification channel is unavailable.

[0042] Specifically, in another embodiment, the notification module further includes a backup mechanism. When the preferred notification channel becomes unavailable due to network failure or platform limitations, the system automatically switches to other preset channels to continue sending alarms. For example, if an instant messaging platform is unavailable, the system can automatically switch to email or other platforms to complete the notification. Through this redundancy mechanism, even in emergencies, critical alarm information can still be delivered in a timely manner, improving the overall system reliability.

[0043] In another embodiment, the present invention also provides a security information event management method based on Graylog, comprising: Receive logs from multiple log sources; Create index sets for different applications or components and set storage and rotation strategies; Parse the logs and extract multiple structured fields; The parsed logs are routed to the corresponding streaming channels according to the rules. Generate events in the stream channel and map event fields to alarm templates; Send alerts via an external instant messaging platform according to the template.

[0044] Specifically, in the implementation of this method, the system first receives log data from multiple log sources and establishes index sets for different applications or components. Then, through a parsing step, it extracts structured fields related to operational behavior, network address, application name, and security level, and routes the logs to different streaming channels according to predefined rules. When a log meets a condition, the system generates an event and maps its fields to an alarm template, finally pushing the alarm through an external instant messaging platform. This method forms a complete process from log access to alarm notification, possessing deployability and portability.

[0045] Furthermore, the method also includes triggering a backup channel to complete the alarm transmission when the alarm transmission fails.

[0046] Specifically, in the above method, the system has a backup channel set up when sending alarms. When the primary channel fails, the system can automatically trigger the backup mechanism to ensure the continuity and integrity of alarm transmission. For example, if instant messaging fails, the system will switch to email or other channels to complete the notification, thereby ensuring that event information is not missed. This method significantly improves the robustness of the system in complex network environments without increasing manual intervention.

[0047] In summary, through the above examples, it can be seen that the present application constructs a complete security information event management system around key links such as log access, index management, field analysis, flow distribution, event detection and alarm, and notification pushing. The modules are not isolated, but form an organic and cooperative working link, so that the originally scattered and redundant log data can be structured and analyzed, finely distributed, intelligently alarmed, and reliably notified in a unified platform. Compared with the prior art, the present application not only breaks through in the depth and accuracy of log analysis, but also forms unexpected technical effects in the accuracy of alarm generation and the safety of notification delivery.

[0048] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can improve or change it according to the above description, and all these improvements and changes shall belong to the protection scope of the appended claims of the present application.

Claims

1. A security information event management system based on Graylog, characterized in that, include: The log access module is used to receive log data from multiple log sources; The log indexing module is used to create index sets for different applications or components and to partition and rotate log data. The field parsing module is used to parse the received logs and extract multiple structured fields; The stream distribution module is used to distribute the parsed logs to the corresponding stream channels according to predefined rules; The event detection and alarm module is used to generate events in the flow channel according to rules and thresholds and map event fields to alarm templates; The notification module is used to send alarms generated based on the template to an external instant messaging platform, and supports security restrictions and formatted display.

2. The Graylog-based security information event management system according to claim 1, characterized in that, The log input module supports both text-based and structured log input.

3. The Graylog-based security information event management system according to claim 1, characterized in that, The index set of the log index module independently corresponds to different applications or components, and performs data storage, sharding, and rotation according to a preset strategy.

4. The Graylog-based security information event management system according to claim 1, characterized in that, The field parsing module processes the log content through a multi-level parsing strategy and extracts fields related to operation behavior, network address, application name, and security level.

5. The Graylog-based security information event management system according to claim 1, characterized in that, The stream distribution module classifies logs based on the field parsing results and routes logs to different stream channels according to security priority.

6. The Graylog-based security information event management system according to claim 1, characterized in that, The event detection and alarm module aggregates similar events and classifies alarm information according to the severity of the event.

7. The Graylog-based security information event management system according to claim 1, characterized in that, The notification module is configured with a Webhook-based notification unit, which supports access restrictions and template-based content generation.

8. The Graylog-based security information event management system according to claim 1, characterized in that, The notification module further includes a backup mechanism that automatically switches to other preset channels to complete the alarm transmission when the preferred notification channel is unavailable.

9. A security information event management method based on Graylog, characterized in that, include: Receive logs from multiple log sources; Create index sets for different applications or components and set storage and rotation strategies; Parse the logs and extract multiple structured fields; The parsed logs are routed to the corresponding streaming channels according to the rules. Generate events in the stream channel and map event fields to alarm templates; Send alerts via an external instant messaging platform according to the template.

10. The Graylog-based security information event management method according to claim 9, characterized in that, It also includes triggering a backup channel to complete the alarm transmission when the alarm transmission fails.

Images