A method and device for real-time aggregation and analysis of security logs

Abstract

The present invention discloses a method and device for real-time aggregation analysis of security logs, and relates to the technical field of network security log processing. The method includes: generating a dynamic rule broadcast stream; generating a custom water level according to the event time in the security log; filtering the security log using the filtering rule, and matching and merging the security log that does not hit the filtering rule with the rule ID of the current rule to form a new security log; performing window aggregation calculation on the security log according to the aggregation rule, comparing the count value of the window aggregation calculation with the preset threshold, and obtaining the result of the window aggregation calculation under each aggregation rule; obtaining the final aggregation result according to the result of the window aggregation calculation under each aggregation rule, and judging whether a risk alarm occurs; and processing the risk alarm content. The present invention completes the analysis and aggregation calculation of the security log in real time according to the dynamic rules, and produces the risk alarm that hits the risk rule within the time range of second-level delay.

Description

Technical Field

[0001] The present invention relates to the technical field of network security log data processing, and in particular to a method and device for real-time aggregation and analysis of security logs. Background Art

[0002] In situational awareness systems, real-time analysis and detection capabilities are usually required to quickly identify abnormal behaviors, potential threats, and environmental changes. Such technologies usually include but are not limited to complex event processing technologies such as aggregation analysis, correlation analysis, and classification algorithms. Aggregate processing and analysis of security logs with large amounts of data are generally handled differently according to different rules. Existing processing methods have the following defects:

[0003] (1) Using offline batch tasks to process data, the timeliness of alarm output is delayed by minutes;

[0004] (2) When filtering and parsing different rules, it is necessary to customize the data processing logic according to the changes in the rules. After the rules are changed, the relevant code logic needs to be taken offline for modification and then put back online;

[0005] (3) In the aggregate analysis processing, it is necessary to customize the development time window according to the needs, and store the logs that meet the rules for secondary aggregation and statistical processing. The overall process is complex and cannot support a changing business portfolio in a platform-based manner. Summary of the invention

[0006] In view of the above-mentioned defects or deficiencies in the prior art, the present invention provides a method and device for real-time aggregation and analysis of security logs. The present invention constructs a custom window filtering aggregation platform that can dynamically load changes in parsing rules. The aggregation analysis calculation model in the platform completes the analysis and aggregation calculation of security logs in real time according to dynamic rules, and generates risk alerts that hit risk rules within a time range with a delay of seconds.

[0007] One aspect of the present invention provides a method for real-time aggregation and analysis of security logs, including: using the flink architecture to consume zookeeper rule model data to generate a dynamic rule broadcast stream; using the flink architecture to consume security log data, and generating a custom watermark according to the event time in the security log data; traversing all set rules to process each security log data; wherein, when using each rule to process the security log data, using the filtering rule to filter the security log data, the security log data that does not hit the filtering rule is matched and merged with the rule ID of the current rule to form new security log data, and sent to the downstream data stream; the new security log data is processed according to the rule ID, rule ID, and rule ID. The version batches and aggregation Key values ​​are grouped, and window aggregation calculation is performed on the grouped security log data according to the corresponding aggregation rules in the dynamic rule broadcast stream, and the count value of the window aggregation calculation under each aggregation rule is compared with the preset threshold to obtain the result of the window aggregation calculation under each aggregation rule; when the waterline reaches the window end time, the final aggregation result is obtained according to the result of the window aggregation calculation under each aggregation rule, and then it is determined whether a risk alarm occurs according to the final aggregation result; according to the risk alarm content in the final aggregation result, the number of security logs with alarm risks is updated to the relational database, and the risk alarm content is written to the Kafka message middleware for consumption by downstream business systems.

[0008] Furthermore, the corresponding aggregation rules in the dynamic rule broadcast stream include counting by field, counting by field deduplication, and counting by field condition comparison.

[0009] Furthermore, window aggregation calculation is performed according to field count, including: determining whether there is a record for the field name of the current security log data within the window time, if so, adding 1 to the number of times the field name appears, otherwise creating a record for the field name, and assigning an initial value of 0 to the number of times the field name appears in the record and adding 1.

[0010] Furthermore, window aggregation calculation is performed according to the field deduplication count, including: obtaining the field name of the current security log data within the window time, determining whether there is a Bloom filter corresponding to the field name; if there is no corresponding Bloom filter, creating a new Bloom filter corresponding to the field name, adding the field value corresponding to the field name to the newly created Bloom filter, and assigning an initial value of 0 to the number of times the field name appears and adding 1; if there is a corresponding Bloom filter, determining whether the field value already exists in the Bloom filter; if there is no field value, adding the field value to the Bloom filter, and adding 1 to the number of times the field name corresponding to the field value appears.

[0011] Furthermore, window aggregation calculation is performed according to field condition comparison counts, including: determining whether the field value under the field name of the current security log data within the window time meets the comparison condition in the rule; if the field value meets the comparison condition and a record exists, the number of occurrences of the field name is increased by 1; if the field value meets the comparison condition and no record exists, a record of the field value is created, and the number of occurrences of the field name in the record is assigned an initial value of 0 and increased by 1.

[0012] Furthermore, when the water level reaches the window end time, the final aggregation result is obtained according to the results of the window aggregation calculation under each aggregation rule, and then the step of judging whether a risk alarm occurs is based on the final aggregation result, including: when the window aggregation calculation results of each aggregation rule are logically true, the final aggregation result is logically true; when the window aggregation calculation result of any aggregation rule is logically false, the final aggregation result is logically false; if the final aggregation result is logically false, then this window aggregation calculation ends and no risk alarm is generated; if the final aggregation result is logically true, then this window aggregation calculation generates a risk alarm, and the relevant field contents whose counts reach or exceed the preset threshold constitute the risk alarm content.

[0013] Another aspect of the present invention provides a device for real-time aggregation and analysis of security logs, including: a broadcast stream generation module, configured to consume zookeeper rule model data using the flink architecture to generate a dynamic rule broadcast stream; a watermark generation module, configured to consume security log data using the flink architecture, and generate a custom watermark according to the event time in the security log data; a rule filtering module, configured to traverse all set rules to process each security log data; wherein, when each rule is used to process the security log data, the security log data is filtered using the filtering rule, and the security log data that does not hit the filtering rule is matched and fused with the rule ID of the current rule to form a new security log data, which is sent to the downstream data stream; a window aggregation calculation module, configured to The new security log data is grouped according to the rule ID, the version batch of the rule and the aggregation Key value, and the grouped security log data is subjected to window aggregation calculation according to the corresponding aggregation rule in the dynamic rule broadcast stream, and the count value of the window aggregation calculation under each aggregation rule is compared with the preset threshold value to obtain the result of the window aggregation calculation under each aggregation rule; when the water level reaches the window end time, the final aggregation result is obtained according to the result of the window aggregation calculation under each aggregation rule, and then it is determined whether a risk alarm occurs according to the final aggregation result; the risk alarm processing module is configured to update the number of security logs with alarm risks to the relational database according to the risk alarm content in the final aggregation result, and write the risk alarm content into the Kafka message middleware for consumption by downstream business systems.

[0014] Furthermore, the window aggregation calculation module includes: a field counting unit, which is used to determine whether there is a record of the field name of the current security log data within the window time. If there is a record, the number of times the field name appears is increased by 1, otherwise a record of the field name is created, and the number of times the field name appears in the record is assigned an initial value of 0 and increased by 1; a field deduplication counting unit, which is used to obtain the field name of the current security log data within the window time, determine whether there is a Bloom filter corresponding to the field name, and if there is no corresponding Bloom filter, create a new Bloom filter corresponding to the field name, add the field value corresponding to the field name to the newly created Bloom filter, and add the number of times the field name appears The number of occurrences is assigned an initial value of 0 and incremented by 1; if there is a corresponding Bloom filter, determine whether the field value already exists in the Bloom filter; if not, add the field value to the Bloom filter and increment the number of occurrences of the field name corresponding to the field value by 1; a field condition counting unit is used to determine whether the field value under the field name of the current security log data within the window time satisfies the comparison condition in the rule; if the field value satisfies the comparison condition and there is a record, increment the number of occurrences of the field name by 1; if the field value satisfies the comparison condition and there is no record, create a record of the field value, and assign an initial value of 0 and increment the number of occurrences of the field name in the record.

[0015] Furthermore, the window aggregation calculation module is configured as follows: when the window aggregation calculation results of each aggregation rule are logically true, the final aggregation result is logically true; when the window aggregation calculation result of any aggregation rule is logically false, the final aggregation result is logically false.

[0016] Furthermore, the window aggregation calculation module is configured as follows: if the final aggregation result is logically false, the current window aggregation calculation ends and no risk warning is generated; if the final aggregation result is logically true, the current window aggregation calculation generates a risk warning, and the relevant field contents whose counts reach or exceed the preset threshold constitute the risk warning content.

[0017] The method and device for real-time aggregation and analysis of security logs provided by the present invention have the following beneficial effects:

[0018] (1) Dynamic loading and effectiveness of rules are realized. Business rules can be changed anytime and anywhere and take effect immediately after going online. There is no need to change the code in the background.

[0019] (2) It enables businesses to set window sizes in rules as needed. After the rules take effect, the window timer automatically performs aggregate calculations.

[0020] (3) The aggregation analysis calculation model can be freely combined in three major scenarios: field statistics, field deduplication statistics, and field matching condition statistics, to complete real-time analysis of alarms in complex aggregation statistics scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0022] Figure 1 This is the overall process of the method for real-time aggregation and analysis of security logs provided by an embodiment of the present application Figure 1 ;

[0023] Figure 2 This is the overall process of the method for real-time aggregation and analysis of security logs provided by an embodiment of the present application Figure 2 ;

[0024] Figure 3 is a flow chart of window aggregation analysis calculation provided by one embodiment of the present application;

[0025] Figure 4 It is a structural diagram of a device for real-time aggregation and analysis of security logs provided by an embodiment of the present application;

[0026] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0029] It should be understood that although the terms first, second, third, etc. may be used to describe the acquisition modules in the embodiments of the present invention, the acquisition modules should not be limited to these terms. These terms are only used to distinguish the acquisition modules from each other.

[0030] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0031] It should be noted that the directional words such as "upper", "lower", "left", and "right" described in the embodiments of the present invention are described at the angles shown in the drawings and should not be understood as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is formed "on" or "under" another element, it can not only be formed directly "on" or "under" another element, but also be formed "on" or "under" another element indirectly through an intermediate element.

[0032] The following are the meanings of the technical terms used in the embodiments of the present application:

[0033] 1. Situational Awareness:

[0034] In the field of network security, situational awareness refers to the real-time monitoring and analysis of various elements in the network environment (such as assets, threats, vulnerabilities, etc.) to understand and predict the security status of the network. It involves multiple aspects such as data collection, event detection, analysis and prediction, and visualization, with the aim of providing a comprehensive view of network security to help security teams make better decisions.

[0035] 2. Security Log:

[0036] In the field of network security, there are many types of security devices such as firewalls, WAF, IDS, IPS, HIDS, etc. The logs generated by the operation of these devices are security logs.

[0037] 3. Risk warning:

[0038] Risk alerts refer to risk alert data automatically generated in the field of network security based on real-time analysis and calculation of security logs according to the set alert rules. Relevant security personnel can take security control measures in real time based on risk alerts to prevent security attacks.

[0039] 4. Rule ID:

[0040] A rule ID is a unique identifier that identifies a specific rule. In this patent application, it refers to a unique ID that belongs to an analysis rule of a type of risk warning.

[0041] 5. Batch ID:

[0042] Batch ID refers to different version IDs of the same rule, that is, the version batch identifier of the rule.

[0043] 6. Aggregation Key:

[0044] The aggregation key is a key used in data aggregation operations to group data elements with the same key value together for subsequent aggregation operations, window calculations, etc.

[0045] 7.Flink framework:

[0046] Apache Flink is an open source stream processing framework for real-time data processing.

[0047] 8. Broadcast Stream:

[0048] A broadcast stream is a special type of stream in Flink that allows a smaller dataset (called the broadcast state) to be efficiently distributed to every operator on every parallel instance.

[0049] 9. Water level:

[0050] When processing real-time data, data may arrive late or out of order. The watermark is a timestamp used in Flink to handle such out-of-order events, indicating that all data before this timestamp has arrived.

[0051] 10. Window:

[0052] Window is a mechanism used in Flink to process and group events. In stream processing, data is continuous, and window allows to define a time period or amount of data so that data can be grouped and processed within that time period or amount of data.

[0053] 11. Timer:

[0054] A timer is a mechanism in Flink stream processing that allows registering an event at a certain point in the future (such as a callback to process delayed data). When the timer is triggered, a timestamp and the data associated with it are sent back to Flink's processing flow.

[0055] See also Figure 1 , 2 An embodiment of the present application provides a method for real-time aggregation and analysis of security logs, comprising the following steps:

[0056] Step S101, use the flink architecture to consume the zookeeper rule model data to generate a dynamic rule broadcast stream.

[0057] Specifically, the Flink architecture is used to consume Zookeeper rule model data and generate a dynamic rule broadcast stream. The broadcast stream data model contains a JSON object of the following content: filtering rules, window settings (for example, window size, whether to trigger in advance, etc.) and aggregation rules.

[0058] Step S102: Use the flink architecture to consume security log data and generate a custom watermark based on the event time in the security log data.

[0059] Specifically, use Flink to consume business data (i.e., security logs) and generate a custom watermark based on the event time in the security logs. The watermark in this step is the timer that drives the window below. Assume that the window in this embodiment is a 10-minute rolling window, 11:50:00--11:59:59, which is used to aggregate and calculate security logs that come in within 10 minutes. When the watermark reaches 12:00, the window will trigger the final calculation.

[0060] Step S103, traverse all the set rules to process each security log data; wherein, when using each rule to process the security log data, the security log data is filtered using the filtering rule, and the security log data that does not hit the filtering rule is matched and merged with the rule ID of the current rule to form new security log data, which is sent to the downstream data stream.

[0061] Specifically, each security log is traversed to all set rules, that is, each rule is used to process the security log data, including: filtering the security log data according to the filtering rules, generating the security log data under the new rules after the security log data that misses the filtering rules merges the rule ID, and sends it to the downstream data stream. "Fusion" here refers to combining the security log data and the rule ID that misses the filtering rules to form the security log data under the new rules. The rules in the present embodiment include two major categories: filtering rules and aggregation rules. If the filtering rules are met, the filtering rules are hit, and the current security log and the current rule do not need to be combined, that is, the processing of the log under the rule ends, that is, it is filtered out. If the filtering rules are not hit, then this security log needs to be merged with the current rule.

[0062] For example, the initial security log includes four fields: id: 001, name: test, type: SQL injection attack, level: high risk. The maintained aggregation calculation rules include: {ruleId: rule 1, filter rule: type level is lower than high risk, aggregation rule: within 1 minute window, count by ID} {ruleId: rule 2, filter rule: type level is higher than high risk, aggregation rule: within 1 minute window, count by ID} {ruleId: rule 3, filter rule: type level equals high risk, aggregation rule: within 1 minute window, count by ID}} After traversing the rules, the security log does not hit rule 1 and rule 2. After merging the rule IDs, the new "security log under the rule" generated is two. Rule 3 hits "type level equals high risk" and is filtered out. Then the new "security log under rules" generated after integrating the rule ID is as follows, that is, combining the useful fields of the security log data and the rule ID together: id:001, name:test, type:sql injection attack, level:high risk, rule ID: rule 1; id:001, name:test, type:sql injection attack, level:high risk, rule ID: rule 2.

[0063] Step S104, grouping the new security log data according to the rule ID, rule version batch and aggregation Key value, performing window aggregation calculation on the grouped security log data according to the corresponding aggregation rule in the dynamic rule broadcast stream, comparing the count value of the window aggregation calculation under each aggregation rule with the preset threshold, and obtaining the result of the window aggregation calculation under each aggregation rule; when the water level reaches the window end time, obtaining the final aggregation result according to the result of the window aggregation calculation under each aggregation rule, and then judging whether a risk alarm occurs according to the final aggregation result.

[0064] Specifically, the new security log data is first grouped according to the rule ID, rule version batch and aggregation Key value through the KeyBy function in the flink architecture; then a custom window is generated according to the window size set in the dynamic rule broadcast stream, and a window trigger timer is set; then, window aggregation calculation is performed on the grouped security log data according to the corresponding aggregation rule in the dynamic rule broadcast stream.

[0065] Furthermore, the aggregation rules include counting by field, counting by field deduplication, and counting by field condition comparison.

[0066] See also Figure 3, window aggregation calculation is performed according to field count, including: judging whether there is a record of the field name of the current security log data within the window time, if there is a record, the number of occurrences of the field name is increased by 1, otherwise a record of the field name is created, and the number of occurrences of the field name in the record is assigned an initial value of 0 and increased by 1. In other words, according to field count, when a security log data is received, the field name count value in the security log data is increased by 1.

[0067] See also Figure 3 , window aggregation calculation is performed according to the field deduplication count, including: obtaining the field name of the current security log data within the window time, judging whether there is a Bloom filter corresponding to the field name, if there is no corresponding Bloom filter, then creating a new Bloom filter corresponding to the field name, adding the field value corresponding to the field name to the newly created Bloom filter, and assigning an initial value of 0 to the number of times the field name appears and adding 1; if there is a corresponding Bloom filter, then judging whether the field value already exists in the Bloom filter, if the field value does not exist, then adding the field value to the Bloom filter, and adding 1 to the number of times the field name corresponding to the field value appears. Among them, the Bloom filter is a probabilistic data structure with extremely high space efficiency, which is used to detect whether an element is in a set. In this embodiment, each field name corresponds to a Bloom filter, and the Bloom filter stores all the field values ​​of this field name. When a new field value comes (that is, different from the existing field value), the count value of the corresponding field name is increased by 1, and the data of the new field value is added to the Bloom filter.

[0068] For example, the aggregation rule is "the security log event type count after deduplication is greater than 3 within 10 minutes, and the security log risk level count after deduplication is greater than 2". This aggregation rule generates three security logs: id: 001, name: test1, type: SQL injection attack, level: high risk, rule ID: rule 1; id: 002, name: test2, type: cross-site scripting (XSS) attack, level: medium risk, rule ID: rule 1; id: 003, name: test3, type: SQL injection attack, level: high risk, rule ID: rule 1;

[0069] In the above case, there are two Bloom filters, namely: Bloom filter A for security log event type and Bloom filter B for security log risk level. The values ​​in A are two "sql injection attack" and "cross-site scripting (XSS) attack", and the values ​​in B are also two "high risk" and "medium risk". The counting results of Bloom filter A and Bloom filter B are "2, 2". In other words, field deduplication counting means that when the field value of the new security log data is a new field value that does not exist, the counting result of the field name is increased by 1.

[0070] See also Figure 3 , window aggregation calculation is performed according to the field condition comparison count, including: judging whether the field value under the field name of the current security log data within the window time meets the comparison condition in the rule, if the field value meets the comparison condition and there is a record, then the number of occurrences of the field name is increased by 1, if the field value meets the comparison condition and there is no record, then create a record of the field value, and assign an initial value of 0 to the number of occurrences of the field name in the record and increase it by 1. In other words, the field condition count means that when the field value of the new security log data meets the requirements of the screening condition, the count result of the field name is increased by 1.

[0071] In addition, window aggregation calculations also include statistics on the number of risk log records, statistics on risk log associated ID records, and so on.

[0072] Furthermore, the count value of the window aggregation calculation under each aggregation rule is compared with the preset threshold value. When it exceeds the preset threshold value or is lower than the preset threshold value, the result of the window aggregation calculation under each aggregation rule is obtained, and the result can be expressed as logical true or logical false. When the window aggregation calculation results of each aggregation rule are all logically true, the final aggregation result is logically true; when the window aggregation calculation result of any aggregation rule is logically false, the final aggregation result is logically false; if the final aggregation result is logically false, the current window aggregation calculation ends and no risk warning is generated; if the final aggregation result is logically true, the current window aggregation calculation generates a risk warning, and the relevant field content whose count reaches or exceeds the preset threshold constitutes the risk warning content.

[0073] Step S105, based on the risk warning content in the final aggregation result, the number of security logs with warning risks is updated to the relational database, and the risk warning content is written to the Kafka message middleware for consumption by downstream business systems.

[0074] Specifically, the risk alert content includes but is not limited to the unique alert ID, the risk log associated with the alert, the number of risk logs associated with the alert, the window to which the alert belongs, the rule ID associated with the alert, the batch ID associated with the alert, etc. The number of security logs with alert risks is updated to the relational database, the risk alert content is converted into json format data, and the json format data is written to the Kafka message middleware for consumption by downstream business systems. The business system consumes Kafka data for page function visualization.

[0075] See also Figure 4Another embodiment of the present invention further provides a device 200 for real-time aggregation analysis of security logs, comprising a broadcast stream generation module 201, a watermark generation module 202, a rule filtering module 203, a window aggregation calculation module 204, and a risk warning processing module 205. The device 200 for real-time aggregation analysis of security logs can execute the method for real-time aggregation analysis of security logs in the above method embodiment.

[0076] Specifically, the device 200 for real-time aggregation and analysis of security logs includes:

[0077] The broadcast stream generation module 201 is configured to consume the zookeeper rule model data using the flink architecture to generate a dynamic rule broadcast stream;

[0078] The watermark generation module 202 is configured to consume the security log data using the flink architecture and generate a custom watermark according to the event time in the security log data;

[0079] The rule filtering module 203 is configured to traverse all set rules to process each security log data; wherein, when each rule is used to process the security log data, the security log data is filtered using the filtering rule, and the security log data that does not hit the filtering rule is matched and merged with the rule ID of the current rule to form new security log data, which is sent to the downstream data stream;

[0080] The window aggregation calculation module 204 is configured to group the new security log data according to the rule ID, the version batch of the rule and the aggregation Key value, perform window aggregation calculation on the grouped security log data according to the corresponding aggregation rule in the dynamic rule broadcast stream, compare the count value of the window aggregation calculation under each aggregation rule with the preset threshold value, and obtain the result of the window aggregation calculation under each aggregation rule; when the water level reaches the window end time, obtain the final aggregation result according to the result of the window aggregation calculation under each aggregation rule, and then determine whether a risk alarm occurs according to the final aggregation result;

[0081] The risk warning processing module 205 is configured to update the number of security logs with warning risks to the relational database according to the risk warning content in the final aggregation result, and write the risk warning content into the Kafka message middleware for consumption by downstream business systems.

[0082] Furthermore, the window aggregation calculation module 204 includes:

[0083] A field counting unit is used to determine whether there is a record of the field name of the current security log data within the window time. If there is a record, the number of occurrences of the field name is increased by 1; otherwise, a record of the field name is created, and the number of occurrences of the field name in the record is given an initial value of 0 and increased by 1;

[0084] A field deduplication counting unit is used to obtain the field name of the current security log data within the window time, determine whether there is a Bloom filter corresponding to the field name, and if there is no corresponding Bloom filter, create a new Bloom filter corresponding to the field name, add the field value corresponding to the field name to the newly created Bloom filter, and assign an initial value of 0 to the number of times the field name appears and add 1; if there is a corresponding Bloom filter, determine whether the field value already exists in the Bloom filter, and if there is no field value, add the field value to the Bloom filter, and add 1 to the number of times the field name corresponding to the field value appears;

[0085] The field condition counting unit is used to determine whether the field value under the field name of the current security log data within the window time meets the comparison condition in the rule. If the field value meets the comparison condition and there is a record, the number of occurrences of the field name is increased by 1. If the field value meets the comparison condition and there is no record, a record of the field value is created, and the number of occurrences of the field name in the record is assigned an initial value of 0 and increased by 1.

[0086] Furthermore, the window aggregation calculation module 204 is configured as follows:

[0087] When the window aggregation calculation results of each aggregation rule are logically true, the final aggregation result is logically true; when the window aggregation calculation result of any aggregation rule is logically false, the final aggregation result is logically false.

[0088] Furthermore, the window aggregation calculation module 204 is configured as follows:

[0089] If the final aggregation result is logically false, the current window aggregation calculation ends and no risk warning is generated; if the final aggregation result is logically true, the current window aggregation calculation generates a risk warning, and the relevant field contents whose counts reach or exceed the preset threshold constitute the risk warning content.

[0090] It should be noted that the device 200 for real-time aggregation and analysis of security logs provided in this embodiment corresponds to a technical solution that can be used to execute each method embodiment. Its implementation principle and technical effect are similar to the method and will not be repeated here.

[0091] Figure 5A schematic diagram of the structure of an electronic device 300 provided in another embodiment of the present invention is provided, and the electronic device is used to implement the method of real-time aggregation analysis of security logs in the method embodiment. The electronic device 300 in the embodiment of the present invention may include but is not limited to a terminal or server such as a firewall, WAF, IDS, IPS, HIDS, etc. Figure 5 The electronic device 300 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0092] like Figure 5 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303 to implement the method of the embodiment of the present invention. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0093] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0094] The above description is only a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but also should cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) to form a technical solution.

Claims

1. A method for real-time aggregation and analysis of security logs, characterized in that: include: Use the flink architecture to consume zookeeper rule model data to generate dynamic rule broadcast streams; Use the Flink architecture to consume security log data and generate a custom watermark based on the event time in the security log data; Traverse all the set rules to process each security log data; when using each rule to process the security log data, use the filtering rule to filter the security log data, and the security log data that does not hit the filtering rule is matched and merged with the rule ID of the current rule to form new security log data, which is sent to the downstream data stream; The new security log data is grouped according to the rule ID, the version batch of the rule and the aggregation Key value, and the grouped security log data is subjected to window aggregation calculation according to the corresponding aggregation rule in the dynamic rule broadcast stream, and the count value of the window aggregation calculation under each aggregation rule is compared with the preset threshold value to obtain the result of the window aggregation calculation under each aggregation rule; when the water level reaches the window end time, the final aggregation result is obtained according to the result of the window aggregation calculation under each aggregation rule, and then whether a risk alarm occurs is determined according to the final aggregation result; According to the risk warning content in the final aggregation result, the number of security logs with warning risks is updated to the relational database, and the risk warning content is written to the Kafka message middleware for consumption by downstream business systems; The corresponding aggregation rules in the dynamic rule broadcast stream include counting by field, counting by field deduplication, and counting by field condition comparison; Window aggregation calculation is performed according to field count, including: determining whether there is a record of the field name of the current security log data within the window time, if there is a record, adding 1 to the number of times the field name appears, otherwise creating a record of the field name, and assigning an initial value of 0 to the number of times the field name appears in the record and adding 1; Window aggregation calculation is performed according to the field deduplication count, including: obtaining the field name of the current security log data within the window time, judging whether there is a Bloom filter corresponding to the field name, if there is no corresponding Bloom filter, creating a new Bloom filter corresponding to the field name, adding the field value corresponding to the field name to the newly created Bloom filter, and assigning an initial value of 0 to the number of times the field name appears and adding 1; if there is a corresponding Bloom filter, judging whether the field value already exists in the Bloom filter, if there is no field value, adding the field value to the Bloom filter, and adding 1 to the number of times the field name corresponding to the field value appears; Window aggregation calculation is performed according to field condition comparison counts, including: judging whether the field value under the field name of the current security log data within the window time meets the comparison condition in the rule; if the field value meets the comparison condition and there is a record, the number of occurrences of the field name is increased by 1; if the field value meets the comparison condition and there is no record, a record of the field value is created, and the number of occurrences of the field name in the record is assigned an initial value of 0 and increased by 1.

2. A method for real-time aggregation and analysis of security logs according to claim 1, characterized in that: When the water level reaches the window end time, the steps of obtaining a final aggregation result according to the results of the window aggregation calculation under each aggregation rule, and then determining whether a risk alarm occurs according to the final aggregation result include: When the window aggregation calculation results of each aggregation rule are logically true, the final aggregation result is logically true; when the window aggregation calculation result of any aggregation rule is logically false, the final aggregation result is logically false; if the final aggregation result is logically false, the current window aggregation calculation ends and no risk warning is generated; if the final aggregation result is logically true, the current window aggregation calculation generates a risk warning, and the relevant field contents whose counts reach or exceed the preset threshold constitute the risk warning content.

3. A device for real-time aggregation and analysis of security logs, characterized in that: include: The broadcast stream generation module is configured to consume the ZooKeeper rule model data using the Flink architecture to generate dynamic rule broadcast streams; The watermark generation module is configured to consume security log data using the Flink architecture and generate a custom watermark based on the event time in the security log data; The rule filtering module is configured to traverse all set rules to process each security log data; wherein, when using each rule to process the security log data, the security log data is filtered using the filtering rule, and the security log data that does not hit the filtering rule is matched and merged with the rule ID of the current rule to form new security log data, which is sent to the downstream data stream; The window aggregation calculation module is configured to group the new security log data according to the rule ID, the version batch of the rule and the aggregation Key value, perform window aggregation calculation on the grouped security log data according to the corresponding aggregation rule in the dynamic rule broadcast stream, compare the count value of the window aggregation calculation under each aggregation rule with the preset threshold value, and obtain the result of the window aggregation calculation under each aggregation rule; when the water level reaches the window end time, obtain the final aggregation result according to the result of the window aggregation calculation under each aggregation rule, and then judge whether a risk alarm occurs according to the final aggregation result; The risk warning processing module is configured to update the number of security logs with warning risks to the relational database according to the risk warning content in the final aggregation result, and write the risk warning content to the Kafka message middleware for consumption by downstream business systems; The window aggregation calculation module includes: A field counting unit is used to determine whether there is a record of the field name of the current security log data within the window time. If there is a record, the number of occurrences of the field name is increased by 1; otherwise, a record of the field name is created, and the number of occurrences of the field name in the record is given an initial value of 0 and increased by 1; A field deduplication counting unit is used to obtain the field name of the current security log data within the window time, determine whether there is a Bloom filter corresponding to the field name, and if there is no corresponding Bloom filter, create a new Bloom filter corresponding to the field name, add the field value corresponding to the field name to the newly created Bloom filter, and assign an initial value of 0 to the number of times the field name appears and add 1; if there is a corresponding Bloom filter, determine whether the field value already exists in the Bloom filter, and if there is no field value, add the field value to the Bloom filter, and add 1 to the number of times the field name corresponding to the field value appears; A field condition counting unit is used to determine whether the field value under the field name of the current security log data within the window time satisfies the comparison condition in the rule. If the field value satisfies the comparison condition and a record exists, the number of occurrences of the field name is increased by 1. If the field value satisfies the comparison condition and no record exists, a record of the field value is created, and the number of occurrences of the field name in the record is given an initial value of 0 and increased by 1. The window aggregation calculation module is configured such that: when the window aggregation calculation results of each aggregation rule are logically true, the final aggregation result is logically true; when the window aggregation calculation result of any aggregation rule is logically false, the final aggregation result is logically false.

4. The device for real-time aggregation and analysis of security logs according to claim 3, characterized in that: The window aggregation calculation module is configured as follows: If the final aggregation result is logically false, the current window aggregation calculation ends and no risk warning is generated; if the final aggregation result is logically true, the current window aggregation calculation generates a risk warning, and the relevant field contents whose counts reach or exceed the preset threshold constitute the risk warning content.

Images