Log data processing method, system and device

Abstract

The present application provides a log data processing method, system, and device. The method comprises the following steps: a distribution engine obtains traffic log data and parses log feature information from the log data; a target rule engine that is interested in the log data is determined based on the log feature information; the target rule engine determines whether the log feature information exists in a first rule bucket; if the first rule bucket is determined to contain the log feature information, the target rule engine determines whether the second rule bucket contains the log feature information; if the second rule bucket does not contain the log feature information, the value of the counter corresponding to the log feature information in the first rule bucket is incremented by 1; if the first rule bucket is determined to contain the log feature information, the target rule engine determines whether the log feature information exists in the second rule bucket; if the second rule bucket does not contain the log feature information, the target rule engine determines whether the log feature information exists in the second rule bucket; if the second rule bucket does not contain the log feature information, the target rule engine determines whether the log feature information exists in the second rule bucket; if the second rule bucket does not contain the log feature information, the target rule engine determines whether the log feature information exists in the first ...

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log data processing method, system, and device. Background Art

[0002] The security management platform, based on security big data, acquires, understands, evaluates, and presents factors that could cause changes in network dynamics, as well as predicts future development trends. This platform enhances the ability to identify, understand, analyze, and respond to security threats from a holistic perspective. Through intelligent analysis and coordinated responses, combined with machine learning and artificial intelligence, it drives closed-loop decision-making within the security brain and effectively implements security capabilities. The currently available abnormal traffic module uses Spark to periodically extract log records of interest from massive amounts of raw logs, aggregate and filter them, and generate corresponding security events. It focuses on traffic logs and audit logs. Due to the massive volume of log data and the serial execution of rules, as the number of rules increases, subsequent rules will inevitably be unable to be processed within the Spark task cycle, resulting in a decrease in security analysis capabilities. Therefore, how to quickly identify abnormal data that meets abnormal traffic rules within the massive amount of data from multiple log sources is a key research topic in the security industry.

[0003] The current Spark log analysis method has the following drawbacks: On the one hand, each rule requires querying, grouping, aggregating, filtering, and other operations on massive amounts of data in memory, which may cause rule detection to become stuck. At the same time, serial processing may cause subsequent rules to time out, thereby reducing security detection capabilities. On the other hand, multiple source logs may affect detection performance. For example, DNS tunnel detection involves traffic session logs and DNS audit logs. Traffic session logs contain upstream and downstream traffic sizes, and audit logs contain DNS domain names. Joining multiple tables will cause significant performance loss.

[0004] Therefore, how to efficiently analyze log data to avoid rule detection jams and improve security detection capabilities is one of the technical issues worth considering. Summary of the Invention

[0005] In view of this, the present application provides a log data processing method, system and device for efficiently analyzing log data to avoid rule detection jams and improve security detection capabilities.

[0006] Specifically, this application is implemented through the following technical solutions:

[0007] According to a first aspect of the present application, a log data processing method is provided, which is applied to a log data processing system, wherein the log data processing system includes a distribution engine and multiple rule engines, and the method includes:

[0008] The distribution engine obtains traffic log data and parses log feature information from the log data;

[0009] The distribution engine determines, based on the log feature information, a target rule engine that is interested in the log data, where the target rule engine is at least one of the multiple rule engines;

[0010] The target rule engine determines whether the log feature information exists in the first rule bucket;

[0011] If the target rule engine determines that the log feature information exists in the first rule bucket, it determines whether the log feature information exists in the second rule bucket; if the log feature information does not exist in the second rule bucket, it increases the value of the counter corresponding to the log feature information in the first rule bucket by 1;

[0012] If the target rule engine determines that the log feature information does not exist in the first rule bucket, the target rule engine increases the value of the counter corresponding to the log feature information in the first rule bucket by 1;

[0013] If the target rule engine determines that the value of the counter is greater than a set threshold, it determines that there is an abnormality in the traffic.

[0014] According to a second aspect of the present application, a log data processing method is provided, which is applied to a rule engine. The method includes:

[0015] Obtaining log feature information distributed by a distribution engine, wherein the distribution engine parses the log feature information from the acquired traffic log data and distributes the log feature information after confirming that the rule engine is interested in the log data based on the log feature information;

[0016] Determine whether the log feature information exists in the first rule bucket;

[0017] If it is determined that the log characteristic information exists in the first rule bucket, then determine whether the log characteristic information exists in the second rule bucket; if the log characteristic information does not exist in the second rule bucket, then increase the value of the counter corresponding to the log characteristic information in the first rule bucket by 1;

[0018] If it is determined that the log feature information does not exist in the first rule bucket, the value of the counter corresponding to the log feature information in the first rule bucket is increased by 1;

[0019] If it is confirmed that the value of the counter is greater than the set threshold, it is confirmed that there is an abnormality in the flow.

[0020] According to a third aspect of the present application, a log data processing system is provided, the system comprising a distribution engine and multiple rule engines, wherein:

[0021] The distribution engine is configured to obtain traffic log data and parse log feature information from the log data; determine a target rule engine that is interested in the log data based on the log feature information, where the target rule engine is at least one of the multiple rule engines;

[0022] The target rule engine is used to determine whether the log feature information exists in the first rule bucket; if it is determined that the log feature information exists in the first rule bucket, then determine whether the log feature information exists in the second rule bucket; if the log feature information does not exist in the second rule bucket, then add 1 to the value of the counter corresponding to the log feature information in the first rule bucket; if it is determined that the log feature information does not exist in the first rule bucket, then add 1 to the value of the counter corresponding to the log feature information in the first rule bucket; if it is confirmed that the value of the counter is greater than the set threshold, then it is confirmed that there is an abnormality in the traffic.

[0023] According to a fourth aspect of the present application, a log data processing device is provided, which is provided in a rule engine, and the device includes:

[0024] An acquisition module is used to acquire log characteristic information distributed by a distribution engine, wherein the distribution engine parses the log characteristic information from the acquired traffic log data and distributes the log characteristic information after confirming that the rule engine is interested in the log data based on the log characteristic information;

[0025] A first judgment module is used to judge whether the log feature information exists in the first rule bucket;

[0026] a second judgment module, configured to judge whether the log characteristic information exists in the second rule bucket if the first judgment module judges that the log characteristic information exists in the first rule bucket;

[0027] a processing module, configured to increase the value of the counter corresponding to the log feature information in the first rule bucket by 1 if the second judgment module determines that the log feature information does not exist in the second rule bucket;

[0028] The processing module is further configured to increase the value of the counter corresponding to the log feature information in the first rule bucket by 1 if the first judgment module determines that the log feature information does not exist in the first rule bucket;

[0029] The flow identification module is used to confirm that the flow is abnormal if it is confirmed that the value of the counter is greater than a set threshold.

[0030] According to the fifth aspect of the present application, an electronic device is provided, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores a computer program that can be executed by the processor, and the processor is prompted by the computer program to execute the method provided in the first aspect of the embodiment of the present application.

[0031] According to the sixth aspect of the present application, a machine-readable storage medium is provided, which stores a computer program. When called and executed by a processor, the computer program prompts the processor to execute the method provided in the first aspect of the embodiment of the present application.

[0032] Beneficial effects of the embodiments of the present application:

[0033] The distribution engine can distribute the log feature information in the log data to each target rule engine that is interested in the log data. In this way, the target rule engine can process the log feature information in parallel instead of the serial processing of the existing technology, thereby greatly improving the processing efficiency of the log data and effectively saving the processing time of the log data, thereby avoiding the occurrence of detection rules being stuck, and to a certain extent improving the detection capability of security detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a log data processing logic diagram provided by the existing technology;

[0035] Figure 2 This is a flow chart of a log data processing method provided by an embodiment of the present application;

[0036] Figure 3 This is an architectural diagram of a log data processing system provided by an embodiment of the present application;

[0037] Figure 4 This is a matching diagram of a distribution engine matching a target rule engine provided by an embodiment of the present application;

[0038] Figure 5 This is a schematic diagram of the structure of a rule bucket corresponding to an SMB intranet lateral diffusion rule provided in an embodiment of the present application;

[0039] Figure 6 This is a structural diagram of a log data processing device provided in an embodiment of the present application;

[0040] Figure 7 This is a hardware structure diagram of an electronic device that implements a log data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the present application.

[0042] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the corresponding listed items.

[0043] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0044] The traditional log data processing process can refer to Figure 1 As shown, first, traffic logs and audit logs for a period of time are obtained from the log data source and then stored in memory. For each rule, the processing flow of the rule is to query, aggregate, and filter from the memory through Spark SQL to obtain data that meets the rule conditions, and finally obtain security events. The inventors found that because the rules are analyzed serially, each rule must perform operations such as querying, aggregation, and filtering from the memory, which will cause greater log processing performance loss due to the number of rules. When the rule logic involves processing traffic and audit logs simultaneously, it is necessary to perform a Join operation on multi-source logs based on log information such as user name and user IP address, which will also bring about a huge performance loss.

[0045] In view of this, the present application proposes a log data processing method, in which a distribution engine obtains traffic log data and parses log feature information from the log data; the distribution engine determines a target rule engine that is interested in the log data based on the log feature information; the target rule engine determines whether the log feature information exists in the first rule bucket; if the target rule engine determines that the log feature information exists in the first rule bucket, it determines whether the log feature information exists in the second rule bucket; if the second rule bucket does not contain the log feature information, it increments the value of the counter corresponding to the log feature information in the first rule bucket by 1; if the target rule engine determines that the log feature information does not exist in the first rule bucket, it increments the value of the counter corresponding to the log feature information in the first rule bucket by 1; if the target rule engine confirms that the counter value is greater than a set threshold, it confirms that the traffic is abnormal. Using the above method, the distribution engine distributes the log data to the target rule engine of interest, and then the target rule engine processes the log data, instead of the serial analysis method in the prior art. This greatly improves the processing efficiency of log data, avoids the situation where rule detection is stuck, and improves security detection capabilities to a certain extent.

[0046] The log data processing method provided by this application is described in detail below.

[0047] See also Figure 2 , Figure 2 This is a flow chart of a log data processing method provided by this application, which can be applied to Figure 3 In the architecture diagram of the log data processing system shown in FIG, in actual application, the log data processing system includes a distribution engine and multiple rule engines; when the distribution engine and the rule engine implement the method, they may include the following steps:

[0048] S201: The distribution engine obtains traffic log data and parses log feature information from the log data.

[0049] In this step, the distribution engine may obtain the log data of the traffic according to the following process: the distribution engine reads the log data of the traffic from the memory, and the memory caches part of the log data of the traffic.

[0050] Specifically, log data from multiple sources will generally be cached in a storage device first, and then the distribution engine will read a portion of the log data of the traffic from the storage device and cache it in the memory, and then read the log data one by one from the memory, and then parse the log feature information for each log data read. Specifically, the distribution engine will call multiple threads to read the log data representing the above traffic from the memory, and then each thread will parse the log feature information from the read log data. It should be noted that the above storage device can be, but is not limited to, a database, a message middleware, etc. The above database can be, but is not limited to, an ElasticSearch database, a ClickHouse database, etc., and the above message middleware can be, but is not limited to, a Kafka middleware.

[0051] S202: The distribution engine determines a target rule engine that is interested in the log data according to the log feature information.

[0052] In this step, the distribution engine will match the parsed log feature information with each rule, and then obtain the target rule engine corresponding to the rule that matches (interests in the log data). It should be noted that the number of target rule engines determined can be 1 or more, which can be determined based on the log feature information and the specific content of each rule. After the target rule engine is determined, the distribution engine sends the log feature information of the log data to each of the determined target rule engines.

[0053] On this basis, after executing step S202, the log data processing method provided by the present application further includes: deleting the traffic log data from the memory.

[0054] Specifically, after the distribution engine determines the target rule engine that is interested in the log data, the log data is deleted from the memory to reduce the storage pressure of the memory. Unlike the existing technology that reads all the data into the memory and then analyzes it, this application adopts streaming processing. In the process of multi-threaded reading of log data from multiple sources into the memory, it directly analyzes each log data to match the corresponding target rule engine. In this way, there will be no situation where the detection rules are stuck, and at the same time, it will not cause the low security detection capability caused by the subsequent rule timeout. Then, after matching the target rule engine for each log data, the log data is deleted from the memory to provide memory space for subsequent reading of log data from the database.

[0055] It should be noted that when the distribution engine fails to match a suitable target rule engine based on the log feature information, it indicates that there is no rule that is interested in the log data, and no processing is required at this time.

[0056] Optionally, after determining the target rule engine corresponding to each log data entry, the distribution engine can tag the log data entry to identify the target rule engine that is interested in the log data. It should be noted that each log data entry only needs to be parsed once, and multiple target rule engines can consume the log data, that is, the matched target rule engines can share the log feature information of the log data.

[0057] It's important to note that each rule corresponds to a rule engine. When performing security checks, there may be a large number of rules, and accordingly, there will be a corresponding number of rule engines. Rules may include, but are not limited to, SMB brute force rules, SMB lateral movement rules, SMB lateral spread rules, RDP brute force rules, RDP lateral movement rules, SSH brute force rules, Telnet brute force rules, Redis brute force rules, MySQL brute force rules, MySQL unauthorized access rules, Elasticsearch unauthorized access rules, port scanning rules, ICMP tunneling rules, and high-risk port communication rules. These rules also include intranet and extranet rules. For example, SMB brute force rules include SMB intranet brute force rules and SMB extranet brute force rules. To better understand each rule, we'll use the SMB intranet lateral expansion rule as an example. The SMB intranet lateral expansion rule can be expressed as: within a 5-minute window, the source IP address accesses port 445 of more than 100 different destination IP addresses. It should be noted that the descriptions of other rules are similar, and the specific settings need to be based on the actual scenario.

[0058] In addition, the log feature information corresponding to different rules may be the same or different. In order to maintain the versatility of the log feature information, the log feature information extracted by the distribution engine from the log data may include source IP address, destination IP address, protocol (protocol), destination port (dest_port), upstream traffic (out_bytes) and downstream traffic (in_bytes), etc. The source IP address is generally obtained from the source area field (src_is_out_in) of the log data, the destination IP address is generally obtained from the destination area field (dest_is_out_in) of the log data, and the destination port is obtained from the destination port field of the log data. Similarly, upstream traffic and downstream traffic can be obtained from the upstream traffic field and downstream traffic field of the log data, respectively. Then, based on these extracted log feature information, each rule is matched, and then the target rule that matches is determined, and then the target rule engine corresponding to the target rule is determined. Since the values ​​of each feature in the log feature information of interest to each rule are different, as shown in Table 1, the specific content of the log feature information of interest to each rule is shown. It should be noted that "-" in Table 1 indicates that the rule is not interested in this feature.

[0059] Table 1

[0060]

[0061]

[0062] On this basis, we can match the log feature information extracted from the log data with the rules in Table 1, thereby matching the target rule of interest to the log data, and then determine the target rule engine corresponding to the target rule. For a specific matching diagram, please refer to Figure 4 As shown, Figure 4 A diagram showing the matching of three log feature information to target rule engines is given, distinguished by numbers 1 / 2 / 3. It can be seen that the log feature information corresponding to number 1 matches two target rule engines, namely the SSH intranet lateral diffusion rule engine and the SSH intranet brute force cracking engine, while the log feature information corresponding to number 2 matches one target rule engine, namely the ICMP tunnel rule engine; similarly, the log feature information corresponding to number 3 matches one target rule engine, namely the MySQL unauthorized access rule engine. It should be noted that Figure 4 This is only an example and does not constitute a specific limitation on the use of log feature information to match the target rule engine in this application.

[0063] S203. The target rule engine determines whether the log feature information exists in the first rule bucket; if so, execute step S204; if not, execute step S205.

[0064] In this step, each rule engine has its own first rule bucket and second rule bucket, which are respectively used to cache intermediate data. The intermediate data may include but is not limited to part or all of the log feature information, etc.

[0065] On this basis, after receiving the log characteristic information, the target rule engine can first determine whether the log characteristic information is recorded in the first rule bucket. If the log characteristic information is recorded in the first rule bucket, step S204 needs to be executed. If not, it indicates that the target rule engine has detected log data carrying the log characteristic information for the first time, and step S205 needs to be executed.

[0066] Optionally, the above-mentioned log characteristic information includes the source IP address. On this basis, step S203 can be performed according to the following process: determine whether the source IP address exists in the first rule bucket; if the above-mentioned source IP address exists in the first rule bucket, confirm that the above-mentioned log characteristic information exists in the first rule bucket; if the above-mentioned source IP address does not exist in the first rule bucket, confirm that the above-mentioned log characteristic information does not exist in the first rule bucket.

[0067] Specifically, the above-mentioned first rule bucket is essentially a data structure that can record log feature information, that is, the rule engine is implemented based on the abstract data structure "rule bucket". For example, the first rule bucket can be a map-type data structure, consisting of key-value pairs, the keyword key can be the source IP address in the log feature information, and the value is used to record the value of the counter corresponding to the source IP address, for example, it can be represented by std for counting. Based on this, the key of the first rule bucket can be traversed, and then it can be determined whether the source IP address in the above-mentioned log feature information exists in the first rule bucket. When the source IP address exists, it is confirmed that the above-mentioned log feature information exists in the first rule bucket; if the source IP address does not exist, it is confirmed that the above-mentioned log feature information does not exist in the first rule bucket.

[0068] S204. If the target rule engine determines that the log feature information exists in the first rule bucket, it determines whether the log feature information exists in the second rule bucket; if the log feature information does not exist in the second rule bucket, the value of the counter corresponding to the log feature information in the first rule bucket is increased by 1.

[0069] Specifically, when the above-mentioned log feature information exists in the first rule bucket, it indicates that the first rule bucket has previously recorded other log data carrying the log feature information. At this time, it is necessary to further determine whether the second rule bucket has a record of the log feature information. If the second rule bucket also records the log feature information, it indicates that the current log data has appeared before and does not need to be recorded repeatedly. At this time, the log data can be discarded.

[0070] If the above-mentioned log characteristic information is not recorded in the second rule bucket, indicating that the current log data is new log data, then the value of the counter corresponding to the log characteristic information in the first rule bucket needs to be increased by 1. In addition, when the above-mentioned log characteristic information does not exist in the second rule bucket, the method provided by the present application also includes: the target rule engine sets the value of the flag bit corresponding to the log characteristic information in the second rule bucket to a set value. Specifically, the log characteristic information can be recorded in the second rule bucket, and the value of the flag bit corresponding to the log characteristic information in the second rule bucket is set to a set value. The set value can be, but is not limited to, 1, indicating that the log characteristic information has been recorded.

[0071] Optionally, when the above-mentioned log characteristic information includes the source IP address and the destination IP address, when it is confirmed that the above-mentioned source IP address exists in the first rule bucket, the target rule engine can execute the following process to determine whether the second rule bucket contains the said log characteristic information: the target rule engine determines whether the source IP address and the destination IP address exist in the second rule bucket. If the source IP address and the destination IP address exist in the second rule bucket, it is confirmed that the above-mentioned log characteristic information exists in the second rule bucket; if the above-mentioned source IP address and the destination IP address do not exist in the second rule bucket, it is confirmed that the above-mentioned log characteristic information does not exist in the second rule bucket.

[0072] Specifically, the above-mentioned second rule bucket is essentially also a data structure, which can record log feature information. That is to say, the rule engine is implemented based on the abstract data structure "rule bucket". For example, the second rule bucket can be a map-type data structure, consisting of a key-value pair key-value, and then the keyword key records the source IP address and the destination IP address, and the key value value records the flag. On this basis, the keys of the second rule bucket can be traversed one by one, and then it can be determined whether there is a source IP address and destination IP address in the key of the second rule bucket that are consistent with the source IP address and destination IP address in the above-mentioned log feature information. If so, it indicates that the above-mentioned log feature information exists in the second rule bucket. If not, it is confirmed that the above-mentioned log feature information does not exist in the second rule bucket.

[0073] It should be noted that if the first rule bucket contains log feature information and the second rule bucket contains log feature information, the value of the counter corresponding to the log feature information in the first rule bucket remains unchanged.

[0074] Specifically, when it is confirmed that the source IP address exists in the first rule bucket based on the key in the first rule bucket, it indicates that the target rule engine may have received log data corresponding to the source IP address before. In order to further confirm whether the target rule engine has received log data corresponding to the source IP address before, the target rule engine will determine whether the source IP address and destination IP address in the log feature information exist in the second rule bucket. If the source IP address and destination IP address do not exist in the second rule bucket, it indicates that the log data this time belongs to the same source IP address but a different destination IP address than the log data of the source IP address recorded in the first rule bucket, indicating that this time If the log data is new log data, the source IP address is recorded in the keyword key in the first rule bucket. At this time, the value of the counter corresponding to the source IP address in the first rule bucket needs to be increased by 1, that is, the value (counter value) recorded in the key value value corresponding to the keyword key (the source IP address) in the first rule bucket is increased by 1; at the same time, the source IP address and destination IP address in the log feature information of this log data are recorded in the second rule bucket, that is, the source IP address and destination IP address are added to the keyword key in the second rule bucket, and the value of the flag bit recorded in the key value value corresponding to the keyword key is set to 1.

[0075] In addition, if the above-mentioned source IP address and destination IP address exist in the second rule bucket, it further indicates that the target rule engine has indeed received log data corresponding to the source IP address before. At this time, the value (the value of the counter) recorded in the key value corresponding to the keyword key (the source IP address) in the first rule bucket remains unchanged.

[0076] S205: If the target rule engine determines that the log feature information does not exist in the first rule bucket, it increases the value of the counter corresponding to the log feature information in the first rule bucket by 1.

[0077] In this step, when the target rule engine determines that the above-mentioned log characteristic information is not recorded in the first rule bucket, indicating that the log data this time is new log data, then it is necessary to record the log characteristic information in the first rule bucket, and at the same time, add 1 to the value of the counter corresponding to the log characteristic information. On this basis, it also includes: the target rule engine sets the value of the flag bit corresponding to the log characteristic information in the second rule bucket to a set value. Specifically, the log characteristic information can be recorded in the second rule bucket, and the value of the flag bit corresponding to the log characteristic information in the second rule bucket is set to a set value. The set value can be, but is not limited to, 1, indicating that the log characteristic information has been recorded.

[0078] Optionally, when the above-mentioned log information includes a source IP address, when implementing step S205, if it is determined that the above-mentioned source IP address is not recorded in the first rule bucket, it indicates that the log data this time is obtained for the first time by the target rule engine. At this time, the source IP address is recorded in the keyword key in the first rule bucket, and then the value in the key value value corresponding to the keyword key (the value of the counter) is added by 1.

[0079] S206: If the target rule engine determines that the value of the counter is greater than the set threshold, it is determined that there is an abnormality in the traffic.

[0080] In this step, the target rule engine queries the counter values ​​in the first rule bucket. Specifically, it queries the value already filled in the first rule bucket. If any value exceeds the set threshold, the target rule engine determines that the traffic corresponding to the key-source IP address with a value greater than the set threshold is abnormal. This allows the target rule engine to identify abnormal traffic based on log data.

[0081] By implementing the log data processing method provided by the present application, the distribution engine can distribute the log feature information in the log data to each target rule engine that is interested in the log data. In this way, the target rule engine can process the log feature information in parallel instead of the serial processing of the existing technology, thereby greatly improving the processing efficiency of the log data and effectively saving the processing time of the log data, thereby avoiding the occurrence of the detection rule being stuck, and improving the detection capability of security detection to a certain extent.

[0082] Optionally, after adding 1 to the value of the counter corresponding to the log feature information in the first rule bucket in step S204 or S205, the log data processing method provided in this embodiment may further include: creating a security event template for the log feature information.

[0083] Specifically, a pointer can also be recorded in the value corresponding to each key in the first rule bucket. The pointer is used to point to the security event template. Specifically, the value corresponding to each key is represented by std:pair, and the pair is used to store a pointer (also called a security time object) to point to the security event template. Figure 5 As shown, Figure 5 The key in the first rule bucket corresponding to the rule engine for the SMB intranet lateral diffusion rule is given in the figure. It is used to store the source IP address (srcip), and the value is (int, *), where int is the value of the counter; "*" represents a pointer to the SMB intranet lateral diffusion security event template, which is cached in the memory. In addition, Figure 5The information recorded in the second rule bucket corresponding to the rule engine of the SMB intranet lateral diffusion rule is also given. That is, the key in the second rule bucket is used to record the source IP address + destination IP address, represented by srcip+destip, and the value of the key is used to record the value of the flag bit of the source IP address and destination IP address, that is, Figure 5 The int in the second rule bucket is a flag bit. If the value of the flag bit int is 1, it indicates that the rule engine has received log feature information containing the source IP address and destination IP address.

[0084] On this basis, when the target rule engine confirms that the value of the counter is greater than the set threshold, it also includes: reporting the security event indicated by the security event template.

[0085] Specifically, when the target rule engine confirms that the value of the counter is greater than the set threshold, the created security event template is filled in to obtain a security event, and then the security event is reported. It should be noted that when judging whether the value of the counter is greater than the set threshold, it can be judged whether the value int in the value corresponding to a single key is greater than the set threshold. If it is greater than the set threshold, it indicates that there is an abnormality in the traffic containing the source IP address in the key, and then the security event template indicated by the pointer in the value of the key is filled in to generate a security event, and then the security event is reported so that the operation and maintenance personnel can perform network security detection. In addition, when it is confirmed that the value int in the value corresponding to the key is not greater than the set threshold, it indicates that there is no abnormality in the traffic carrying the source IP address and will not pose a threat to the security of the network, and the security event template indicated by the pointer in the value corresponding to the key is deleted.

[0086] It should be noted that before determining whether the counter value is greater than the set threshold, the following process must be performed: determining whether log data processing within the set time has completed, or determining whether the set timer has expired; when log data processing within the set time has completed, or the set timer has expired, the counter value must be determined to be greater than the set threshold. This ensures the accuracy and rationality of the abnormal traffic identification results. For example, using the SMB lateral movement rule as an example, with a threshold set to 100, when the set timer expires, the SMB lateral movement rule engine determines whether the value of the int in each value in the first rule bucket is greater than 100. If the SMB lateral movement rule engine determines that the int in the value corresponding to source IP address 1 is 101, and 101 is greater than 100, it populates the security event template pointed to by the pointer in the value corresponding to source IP address 1, generates a security event, and then reports the security event. If it is confirmed that the int in the value corresponding to IP address 2 is 99, and 99 is less than 100, it deletes the security event template pointed to by the pointer in the value corresponding to source IP address 2 to free up memory resources.

[0087] So far, the distribution engine of this application uses multi-threading to process log data from multiple sources simultaneously, and adopts streaming analysis to avoid the drawbacks of loading into memory for serial analysis. In addition, based on the log feature information extracted by the distribution engine, it can quickly match the rules of interest to the traffic, so that the rule engine corresponding to the rule of interest can quickly detect the rules based on their respective "rule bucket" data structures, greatly improving the processing performance of abnormal traffic log analysis, which is conducive to the log analysis product to process more abnormal detection rules and enhance the threat detection capabilities of the product. Practice has proved that using the method provided by this application, the stand-alone version can process 80,000 log data per second, far exceeding the product specification of processing 4,000 data per second, which significantly realizes the efficient analysis of log data and greatly improves the detection capabilities of security detection.

[0088] Based on the same inventive concept, the present application also provides a log data processing method implemented on the rule engine side, comprising the following steps:

[0089] Step 1: Obtain log feature information distributed by the distribution engine, wherein the distribution engine parses the log feature information from the acquired traffic log data and confirms that the rule engine is interested in the log data based on the log feature information before distributing the log feature information;

[0090] Step 2: Determine whether the log feature information exists in the first rule bucket;

[0091] Step 3: If it is determined that the log characteristic information exists in the first rule bucket, then determine whether the log characteristic information exists in the second rule bucket; if the log characteristic information does not exist in the second rule bucket, then increase the value of the counter corresponding to the log characteristic information in the first rule bucket by 1;

[0092] Step 4: If it is determined that the log feature information does not exist in the first rule bucket, the value of the counter corresponding to the log feature information in the first rule bucket is increased by 1;

[0093] Step 5: If it is confirmed that the value of the counter is greater than the set threshold, it is confirmed that the flow is abnormal.

[0094] It should be noted that the implementation of the above steps 1 to 5 can refer to the relevant description of steps S203 to S206, and will not be described in detail here.

[0095] Optionally, based on the above embodiment, the above log feature information includes the source IP address and the destination IP address; then

[0096] The following process can be used to perform step 2 to determine whether the log feature information exists in the first rule bucket: determine whether the source IP address exists in the first rule bucket; if the source IP address exists in the first rule bucket, confirm that the log feature information exists in the first rule bucket.

[0097] Similarly, the following process can be used to perform step three to determine whether the second rule bucket contains the log characteristic information: determine whether the second rule bucket contains the source IP address and the destination IP address; if the source IP address and the destination IP address exist in the second rule bucket, confirm that the second rule bucket contains the log characteristic information.

[0098] It should be noted that the implementation of the above two determination steps can refer to the relevant description of steps S203 to S204, and will not be described in detail here.

[0099] Optionally, based on any of the above embodiments, after the rule engine adds 1 to the value of the counter corresponding to the log feature information in the first rule bucket, the rule engine further includes: creating a security event template for the log feature information;

[0100] On this basis, if the rule engine confirms that the value of the counter is greater than the set threshold, it also includes: reporting the security event indicated by the security event template.

[0101] Specifically, the implementation of the above two steps can refer to Figure 2 The relevant descriptions of the target rule engine in the illustrated embodiment will not be described in detail here.

[0102] Optionally, based on any of the above embodiments, the log data processing method provided in this embodiment further includes: if the log feature information exists in the first rule bucket and the log feature information exists in the second rule bucket, then the value of the counter corresponding to the log feature information in the first rule bucket remains unchanged.

[0103] Specifically, the implementation of this step can refer to Figure 2 The relevant descriptions of the target rule engine in the illustrated embodiment will not be described in detail here.

[0104] Optionally, based on any of the above embodiments, if there is no log feature information in the second rule bucket, or there is no log feature information in the above first rule bucket, the log data processing method provided in this embodiment also includes: the value of the flag bit corresponding to the log feature information in the second rule bucket is set to a set value.

[0105] Specifically, the implementation of this step can refer to Figure 2 The relevant descriptions of the target rule engine in the illustrated embodiment will not be described in detail here.

[0106] By implementing the log data processing method provided in the present application, the distribution engine can distribute the log feature information in the log data to each target rule engine that is interested in the log data. In this way, after each target rule engine receives the log feature information, it can process the log feature information in parallel, instead of the serial processing of the existing technology, thereby greatly improving the processing efficiency of the log data, effectively saving the processing time of the log data, and thus avoiding the occurrence of the detection rule being stuck, and to a certain extent improving the detection capability of security detection.

[0107] Based on the same inventive concept, the present application also provides a log data processing system, including a distribution engine and multiple rule engines, wherein:

[0108] The distribution engine is configured to obtain traffic log data and parse log feature information from the log data; determine a target rule engine that is interested in the log data based on the log feature information, wherein the target rule engine is at least one of the multiple rule engines;

[0109] The above-mentioned target rule engine is used to determine whether the log feature information exists in the first rule bucket; if it is determined that the log feature information exists in the first rule bucket, then determine whether the log feature information exists in the second rule bucket; if the log feature information does not exist in the second rule bucket, then add 1 to the value of the counter corresponding to the log feature information in the first rule bucket; if it is determined that the log feature information does not exist in the first rule bucket, then add 1 to the value of the counter corresponding to the log feature information in the first rule bucket; if it is confirmed that the value of the counter is greater than the set threshold, it is confirmed that there is an abnormality in the traffic.

[0110] It should be noted that, in actual applications, the distribution engine and rule engine can be set in the same device or in different devices, and the specific setting can be based on actual conditions.

[0111] Optionally, based on the above embodiment, the distribution engine is specifically configured to read the log data of the traffic from a memory, wherein the memory caches part of the log data of the traffic;

[0112] On this basis, the distribution engine is further configured to delete the log data of the traffic from the memory after determining a target rule engine that is interested in the log data according to the log feature information.

[0113] Optionally, based on any of the above embodiments, the above log feature information includes the source IP address and the destination IP address; then

[0114] The target rule engine is specifically configured to determine whether the source IP address exists in the first rule bucket; if the source IP address exists in the first rule bucket, confirm that the log feature information exists in the first rule bucket;

[0115] Optionally, based on any of the above embodiments, the above target rule engine is specifically used to create a security event template for the log feature information after adding 1 to the value of the counter corresponding to the log feature information in the first rule bucket.

[0116] On this basis, the target rule engine is specifically configured to report the security event indicated by the security event template if it is confirmed that the value of the counter is greater than a set threshold.

[0117] Optionally, the target rule engine is further configured to keep the value of the counter corresponding to the log feature information in the first rule bucket unchanged if the log feature information exists in the first rule bucket and the log feature information exists in the second rule bucket.

[0118] Optionally, the target rule engine is further configured to set the value of the flag bit corresponding to the log feature information in the second rule bucket to a set value if the log feature information does not exist in the second rule bucket, or the log feature information does not exist in the first rule bucket.

[0119] By implementing the log data processing system provided by the present application, the distribution engine can distribute the log feature information in the log data to each target rule engine that is interested in the log data. In this way, after each target rule engine receives the log feature information, it can process the log feature information in parallel, rather than the serial processing of the existing technology, thereby greatly improving the processing efficiency of the log data and effectively saving the processing time of the log data, thereby avoiding the occurrence of the detection rule being stuck, and to a certain extent improving the detection capability of security detection.

[0120] Based on the same inventive concept, the present application also provides a log data processing device corresponding to the above log data processing method. The implementation of the log data processing device can refer to the description of the log data processing method by the rule engine above, which will not be discussed here one by one.

[0121] See also Figure 6 , Figure 6 An exemplary embodiment of the present application provides a log data processing device, which is provided in a rule engine. The device includes:

[0122] An acquisition module 601 is configured to acquire log feature information distributed by a distribution engine, wherein the distribution engine parses the log feature information from the acquired traffic log data and distributes the log feature information after confirming that the rule engine is interested in the log data based on the log feature information.

[0123] A first judgment module 602 is used to judge whether the log feature information exists in the first rule bucket;

[0124] A second judgment module 603 is configured to judge whether the second rule bucket contains the log characteristic information if the first judgment module 602 judges that the first rule bucket contains the log characteristic information;

[0125] a processing module 604 configured to increase the value of a counter corresponding to the log feature information in the first rule bucket by 1 if the second judgment module 603 determines that the log feature information does not exist in the second rule bucket;

[0126] The processing module 604 is further configured to increase the value of the counter corresponding to the log feature information in the first rule bucket by 1 if the first judging module 602 judges that the log feature information does not exist in the first rule bucket;

[0127] The flow identification module 605 is configured to determine that the flow is abnormal if it is determined that the value of the counter is greater than a set threshold.

[0128] Optionally, based on the above embodiment, the log feature information in this embodiment includes the source IP address and the destination IP address;

[0129] The first judgment module 602 is specifically configured to determine whether the source IP address exists in the first rule bucket; if the source IP address exists in the first rule bucket, confirming that the log feature information exists in the first rule bucket;

[0130] The second judgment module 603 is specifically used to determine whether the source IP address and the destination IP address exist in the second rule bucket; if the source IP address and the destination IP address exist in the second rule bucket, it is confirmed that the log feature information exists in the second rule bucket.

[0131] Optionally, based on any of the above embodiments, the log data processing device provided in this embodiment further includes:

[0132] A creation module (not shown in the figure) is used to create a security event template for the log feature information after the processing module 604 adds 1 to the value of the counter corresponding to the log feature information in the first rule bucket;

[0133] The reporting module (not shown in the figure) is used to report the security event indicated by the security event template after the traffic identification module 605 confirms that the value of the counter is greater than the set threshold.

[0134] Optionally, based on any of the above embodiments, the above processing module 604 is also used to keep the value of the counter corresponding to the log feature information in the first rule bucket unchanged if the judgment result of the first judgment module 602 is that the log feature information exists in the first rule bucket, and the judgment result of the above second judgment module 603 is that the log feature information exists in the second rule bucket.

[0135] Optionally, based on any of the above embodiments, the above processing module 604 is also used to set the value of the flag bit corresponding to the log characteristic information in the second rule bucket to a set value if the judgment result of the second judgment module 603 is that the log characteristic information does not exist in the second rule bucket, or if the judgment result of the first judgment module 602 is that the above log characteristic information does not exist in the first rule bucket.

[0136] Based on the same inventive concept, the embodiment of the present application provides an electronic device, such as Figure 7As shown, it includes a processor 701 and a machine-readable storage medium 702, the machine-readable storage medium 702 stores a computer program that can be executed by the processor 701, and the processor 701 is prompted by the computer program to execute the log data processing method on the distribution engine side provided by any embodiment of the present application, and / or, execute the log data processing method on the rule engine side provided by any embodiment of the present application. In addition, the electronic device also includes a communication interface 703 and a communication bus 704, wherein the processor 701, the communication interface 703, and the machine-readable storage medium 702 communicate with each other through the communication bus 704.

[0137] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0138] The communication interface is used for communication between the above electronic device and other devices.

[0139] The memory may include random access memory (RAM), DDR SRAM (Double Data Rate Synchronous Dynamic Random Access Memory), or non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may be at least one storage device located remotely from the processor.

[0140] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0141] In addition, an embodiment of the present application provides a machine-readable storage medium, which stores a computer program. When called and executed by a processor, the computer program prompts the processor to execute the log data processing method provided by the embodiment of the present application.

[0142] As for the electronic device and machine-readable storage medium embodiments, since the method contents involved are basically similar to the aforementioned method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0143] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0144] The implementation process of the functions and effects of each unit / module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0145] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units / modules described as separate components may or may not be physically separated, and the components displayed as units / modules may or may not be physical units / modules, that is, they may be located in one place, or they may be distributed over multiple network units / modules. Some or all of the units / modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0146] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A log data processing method, characterized in that: Applied to a log data processing system, the log data processing system includes a distribution engine and multiple rule engines, and the method includes: The distribution engine obtains traffic log data and parses log feature information from the log data; The distribution engine determines, based on the log feature information, a target rule engine that is interested in the log data, where the target rule engine is at least one of the multiple rule engines; The target rule engine determines whether the log feature information exists in the first rule bucket; If the target rule engine determines that the log feature information exists in the first rule bucket, it determines whether the log feature information exists in the second rule bucket; if the log feature information does not exist in the second rule bucket, it increases the value of the counter corresponding to the log feature information in the first rule bucket by 1; If the target rule engine determines that the log feature information does not exist in the first rule bucket, the target rule engine increases the value of the counter corresponding to the log feature information in the first rule bucket by 1; If the target rule engine determines that the value of the counter is greater than the set threshold, it is determined that the traffic is abnormal; The log feature information includes the source IP address and the destination IP address; The target rule engine determines whether the log feature information exists in the first rule bucket, including The target rule engine determines whether the source IP address exists in the first rule bucket; If the source IP address exists in the first rule bucket, confirming that the log feature information exists in the first rule bucket; The target rule engine determines whether the second rule bucket contains the log feature information, including: The target rule engine determines whether the source IP address and the destination IP address exist in the second rule bucket; If the source IP address and the destination IP address exist in the second rule bucket, it is confirmed that the log feature information exists in the second rule bucket.

2. The method according to claim 1, characterized in that The distribution engine obtains traffic log data, including: The distribution engine reads the log data of the traffic from a memory, wherein the memory caches part of the log data of the traffic; After the distribution engine determines a target rule engine that is interested in the log data according to the log feature information, the method further includes: The log data of the traffic is deleted from the memory.

3. The method according to claim 1, characterized in that After adding 1 to the value of the counter corresponding to the log feature information in the first rule bucket, the method further includes: Creating a security event template for the log feature information; If the target rule engine determines that the value of the counter is greater than the set threshold, the target rule engine further includes: Report the security event indicated by the security event template.

4. The method according to claim 1, wherein Also includes: If the log feature information exists in the first rule bucket and the log feature information exists in the second rule bucket, the value of the counter corresponding to the log feature information in the first rule bucket remains unchanged.

5. The method according to claim 1, wherein If the second rule bucket does not contain the log feature information, or the first rule bucket does not contain the log feature information, the method further includes: The target rule engine sets the value of the flag bit corresponding to the log feature information in the second rule bucket to a set value.

6. A log data processing method, characterized in that: Applied to a rule engine, the method includes: Obtaining log feature information distributed by a distribution engine, wherein the distribution engine parses the log feature information from the acquired traffic log data and distributes the log feature information after confirming that the rule engine is interested in the log data based on the log feature information; Determine whether the log feature information exists in the first rule bucket; If it is determined that the log characteristic information exists in the first rule bucket, then determine whether the log characteristic information exists in the second rule bucket; if the log characteristic information does not exist in the second rule bucket, then increase the value of the counter corresponding to the log characteristic information in the first rule bucket by 1; If it is determined that the log feature information does not exist in the first rule bucket, the value of the counter corresponding to the log feature information in the first rule bucket is increased by 1; If it is confirmed that the value of the counter is greater than the set threshold, it is confirmed that the flow is abnormal; The log feature information includes the source IP address and the destination IP address; Determining whether the log feature information exists in the first rule bucket includes: Determine whether the source IP address exists in the first rule bucket; If the source IP address exists in the first rule bucket, confirming that the log feature information exists in the first rule bucket; Determining whether the second rule bucket contains the log feature information includes: Determine whether the source IP address and the destination IP address exist in the second rule bucket; If the source IP address and the destination IP address exist in the second rule bucket, it is confirmed that the log feature information exists in the second rule bucket.

7. The method according to claim 6, characterized in that After adding 1 to the value of the counter corresponding to the log feature information in the first rule bucket, the method further includes: Creating a security event template for the log feature information; If it is confirmed that the value of the counter is greater than the set threshold, the method further includes: Report the security event indicated by the security event template.

8. The method according to claim 6, characterized in that Also includes: If the log feature information exists in the first rule bucket and the log feature information exists in the second rule bucket, the value of the counter corresponding to the log feature information in the first rule bucket remains unchanged.

9. The method according to claim 6, characterized in that If the second rule bucket does not contain the log feature information, or the first rule bucket does not contain the log feature information, the method further includes: The value of the flag bit corresponding to the log feature information in the second rule bucket is set to a set value.

10. A log data processing system, characterized in that: The system includes a distribution engine and multiple rule engines, wherein: The distribution engine is configured to obtain traffic log data and parse log feature information from the log data; determine a target rule engine that is interested in the log data based on the log feature information, where the target rule engine is at least one of the multiple rule engines; The target rule engine is used to determine whether the log feature information exists in the first rule bucket; if it is determined that the log feature information exists in the first rule bucket, then determine whether the log feature information exists in the second rule bucket; if the log feature information does not exist in the second rule bucket, then increase the value of the counter corresponding to the log feature information in the first rule bucket by 1; if it is determined that the log feature information does not exist in the first rule bucket, then increase the value of the counter corresponding to the log feature information in the first rule bucket by 1; if it is confirmed that the value of the counter is greater than a set threshold, then it is confirmed that the traffic is abnormal; The log feature information includes the source IP address and the destination IP address; Determining whether the log feature information exists in the first rule bucket includes: Determine whether the source IP address exists in the first rule bucket; If the source IP address exists in the first rule bucket, confirming that the log feature information exists in the first rule bucket; Determining whether the second rule bucket contains the log feature information includes: Determine whether the source IP address and the destination IP address exist in the second rule bucket; If the source IP address and the destination IP address exist in the second rule bucket, it is confirmed that the log feature information exists in the second rule bucket.

11. A log data processing device, characterized in that: The device is provided in a rule engine, and includes: An acquisition module is used to acquire log characteristic information distributed by a distribution engine, wherein the distribution engine parses the log characteristic information from the acquired traffic log data and distributes the log characteristic information after confirming that the rule engine is interested in the log data based on the log characteristic information; A first judgment module is used to judge whether the log feature information exists in the first rule bucket; a second judgment module, configured to judge whether the log characteristic information exists in the second rule bucket if the first judgment module judges that the log characteristic information exists in the first rule bucket; a processing module, configured to increase the value of the counter corresponding to the log feature information in the first rule bucket by 1 if the second judgment module determines that the log feature information does not exist in the second rule bucket; The processing module is further configured to increase the value of the counter corresponding to the log feature information in the first rule bucket by 1 if the first judgment module determines that the log feature information does not exist in the first rule bucket; A flow identification module, configured to determine that the flow is abnormal if the value of the counter is greater than a set threshold; The log feature information includes the source IP address and the destination IP address; Determining whether the log feature information exists in the first rule bucket includes: Determine whether the source IP address exists in the first rule bucket; If the source IP address exists in the first rule bucket, confirming that the log feature information exists in the first rule bucket; Determining whether the second rule bucket contains the log feature information includes: Determine whether the source IP address and the destination IP address exist in the second rule bucket; If the source IP address and the destination IP address exist in the second rule bucket, it is confirmed that the log feature information exists in the second rule bucket.

Images