Log detection method, device, equipment and storage medium

Abstract

The application provides a log detection method, device and equipment and a storage medium, the method comprising: obtaining network behavior information, and parsing the network behavior information to obtain log data; filtering the log data by using a first IOC rule in the memory, the first IOC rule being used to filter out log data that will not generate an alarm; matching the filtered log data with a second IOC rule in the memory to obtain a matching result, the second IOC rule being used to filter out log data that can generate an alarm; and the number of fields included in the first IOC rule being less than the number of fields included in the second IOC rule. Since the number of fields included in the first IOC rule is less than that of the second IOC rule, the memory space occupied by the first IOC rule is less than that of the second IOC rule, and most of the log data has been filtered by the first IOC rule, and then the filtered log data and the second IOC rule are matched, thereby saving the memory overhead.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a log detection method, apparatus, device, and storage medium. Background Technology

[0002] Log inspection is crucial in the field of cybersecurity. It can detect threatening network behaviors. Log inspection using Indicators of Compromise (IOC) is a detection method based on intrusion indicators that can discover many threats that traditional security products cannot detect.

[0003] IOC log detection uses Internet Protocol (IP) addresses and domain names to match IOC rules. IOC rules are essentially blacklists of IP addresses and domain names. In existing technologies, IOC detection programs improve efficiency by loading a large number of IOC rules into memory; however, loading too many IOC rules can lead to significant memory overhead. Summary of the Invention

[0004] This invention provides a log detection method, apparatus, device, and storage medium to address the shortcomings of existing technologies where loading too many IOC rules leads to high memory overhead, thereby achieving memory savings.

[0005] In a first aspect, the present invention provides a log detection method, comprising:

[0006] Obtain network behavior information and parse it to obtain log data;

[0007] The log data is filtered using the first IOC rule in memory to obtain the filtered log data. The first IOC rule is used to filter out log data that will not generate alarms.

[0008] The filtered log data is matched with the second IOC rule in memory to obtain the matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule.

[0009] Further, the first IOC rule includes: a first field, the value of which corresponds to the hash verification value of the first attribute information; the first attribute information is attribute information with alarm risk; the first attribute information includes at least one of the following: IP address, domain name, source port, and destination port; the log data is filtered using the first IOC rule in memory to obtain filtered log data, including:

[0010] Hash at least one attribute in the log data to obtain the hash check value corresponding to each attribute.

[0011] Each hash check value is matched with the value of the first field in the first IOC rule. The log data corresponding to the hash check values ​​that do not match the value of the first field are filtered to obtain the filtered log data.

[0012] Further, the step of filtering the log data corresponding to hash checksums that do not match the value of the first field to obtain filtered log data includes:

[0013] The log data corresponding to hash check values ​​that do not match the value of the first field are filtered using the Cuckoo Filter to obtain the filtered log data.

[0014] Furthermore, the second IOC rule includes a second field, and at least one of the following rule fields: a third field, an IOC rule type field, and a network behavior type field; the value of the second field corresponds to the hash check value of the second attribute information; the second attribute information is attribute information that can generate alarms; the second attribute information includes at least one of the following: IP address, domain name, source port, and destination port; the value of the third field is used to indicate the port number corresponding to the destination port included in the log data; the value of the IOC rule type field is used to indicate the type of the second attribute information in the second IOC rule; the value of the network behavior type field is used to indicate the network behavior type corresponding to the domain name.

[0015] Furthermore, the domain name also includes the top-level private domain name. The filtered log data is matched with the second IOC rule in memory to obtain the matching results, including:

[0016] The attribute information contained in the filtered log data is matched with the second field and at least one rule field contained in the second IOC rule to obtain the matching result; the attribute information contained in the filtered log data includes at least one of the following: domain name, IP address, destination port;

[0017] When the preset matching conditions are met, the matching result is determined to be the filtered log data matching the second IOC rule;

[0018] The preset matching criteria include at least one of the following:

[0019] The hash verification value corresponding to the top-level private domain in the filtered log data is the value of the second field, and the value of the IOC rule type field is the top-level private domain type;

[0020] The hash verification value corresponding to the domain name in the filtered log data is the value of the second field, and the value of the IOC rule type field is a non-top-level private domain name type;

[0021] In the filtered log data, the hash check value corresponding to the IP address is the value of the second field, the value of the IOC rule type field is a non-top-level private domain name type, and the port number corresponding to the destination port is the value of the third field.

[0022] In the filtered log data, the hash checksum corresponding to the IP address is the value of the second field, and the value of the IOC rule type field is empty.

[0023] Furthermore, after matching the filtered log data with the second IOC rule in memory and obtaining the matching result, the method further includes:

[0024] Generate alarm information.

[0025] Furthermore, the first IOC rule and / or the second IOC rule are stored in an in-memory IOC rule base, and the method further includes:

[0026] The first IOC rule and / or the second IOC rule in the IOC rule base are cached in the memory of the electronic device, which is then used to execute the log detection method.

[0027] Furthermore, the method also includes:

[0028] When the IOC rules in the IOC rule base are updated, the first IOC rule and / or the second IOC rule cached in memory are updated.

[0029] Secondly, the present invention also provides a log detection device, comprising:

[0030] The acquisition module is used to acquire network behavior information and parse the network behavior information to obtain log data;

[0031] The filtering module is used to filter log data using the first IOC rule in memory to obtain filtered log data. The first IOC rule is used to filter out log data that will not generate alarms.

[0032] The matching module matches the filtered log data with the second IOC rule in memory to obtain the matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule.

[0033] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the log detection methods described above.

[0034] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the log detection methods described above.

[0035] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the log detection methods described above.

[0036] This invention provides a log detection method, apparatus, device, and storage medium. By acquiring network behavior information and parsing the log data within it, the log data is first filtered based on a first IOC rule in memory. This means filtering out log data that cannot match the first IOC rule at all (i.e., log data that will not generate alarms). The filtered log data (logs that may match a second IOC rule) is then further matched against the second IOC rule in memory to obtain a matching result. Because the first IOC rule includes fewer fields than the second IOC rule, the memory space occupied by the first IOC rule is less than that occupied by the second IOC rule. Moreover, since the first IOC rule has already filtered out most of the log data, further matching based on the filtered log data against the second IOC rule achieves memory savings. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts of the log detection method provided by the present invention;

[0039] Figure 2 Another flowchart of the log detection method provided by this invention;

[0040] Figure 3 This is a schematic diagram of the log detection device provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0043] The method of this invention can be applied to network security detection scenarios, realizing fast log detection based on IOC, saving memory overhead, and improving the accuracy of log detection.

[0044] In existing technologies, many IOC detection programs load large-scale IOC rules into memory for fast matching. However, loading such a large number of IOC rules into memory leads to significant memory overhead, exceeding 5GB. Furthermore, practically speaking, not all IOC logs generate alerts; approximately 90% of logs do not. Therefore, a possible solution is to set up a filter to first remove most logs that will not generate alerts (essentially a whitelist), retaining only those that might. Then, based on the filtered log data, precise matching criteria are used to further refine the matching process and obtain the final result.

[0045] The log detection method of this invention obtains network behavior information and parses log data, filters out logs that will not match the first IOC rule, and then matches them using a precise second IOC matching rule to obtain the matching result. This saves memory overhead and improves the accuracy of log detection.

[0046] To facilitate a clearer understanding of the various embodiments of this application, some related technical content will be introduced first.

[0047] Indicators of Compromise (IOC): These are the remote command and control servers used by attackers to control the victim's host. IOC logs are generally associated with network behavior, such as accessing a specific IP address and domain name.

[0048] Filters: For massive data processing tasks, we usually need an indexed data structure to help queries quickly determine if a data record exists. This data structure is often called a filter. Commonly used filters include Bloom filters and Cuckoo filters.

[0049] Alert: The output generated after the IOC detection service matches the logs indicates the IP address or domain name accessed in the IOC rule for that log.

[0050] The following is combined Figures 1-4 The technical solution of the present invention will be described in detail with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0051] Figure 1 This is one of the flowcharts illustrating the log detection method provided in this embodiment of the invention, such as... Figure 1 As shown, this log detection method includes the following steps:

[0052] Step 101: Obtain network behavior information and parse the network behavior information to obtain log data;

[0053] Step 102: Filter the log data using the first IOC rule in memory to obtain filtered log data. The first IOC rule is used to filter out log data that will not generate alarms.

[0054] Step 103: Match the filtered log data with the second IOC rule in memory to obtain the matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule.

[0055] It should be noted that network behavior information includes at least one of the following: Internet Protocol (IP), Domain Name Service (DNS); Transmission Control Protocol (TCP); User Datagram Protocol (UDP); Universal Resource Identifier (URI); Hypertext Transfer Protocol (HTTP); Hypertext Transfer Protocol over Secure Socket Layer (HTTPS). The network behavior information is compressed and stored in, for example, a Kafka message queue cluster on an electronic device.

[0056] Specifically, in step 101, network behavior information is obtained, such as network behavior information consumed from the Kafka cluster. Further, the network behavior information is decompressed and log data is extracted, such as source IP, destination IP, domain name, source port, destination port, etc.

[0057] Further, in step 102, log data that does not match the first IOC rule in memory is filtered out using a filter, such as a cuckoo filter. This filters out log data that will not generate alarms, resulting in filtered log data. In other words, log data that will not match the first IOC rule at all is filtered out, resulting in a set of log data that may match the second IOC rule. From a practical perspective, approximately 80% to 90% of log data will not generate alarm information. Therefore, this step, for example, filters out 80% to 90% of the logs, resulting in filtered log data representing 10% to 20% of all log data.

[0058] Specifically, the attribute information (e.g., IP address, domain name, source port, destination port) included in the log data is hashed to obtain the hash checksum corresponding to each attribute information. Each hash checksum is then matched against a first IOC rule in memory. Log data that fails to match the first IOC rule is filtered out. For example, log data that successfully matches the first IOC rule (including the first field value1) is retained in memory. That is, if at least one hash checksum of the attribute information corresponding to the log data matches the value of the first field value1, the log data is kept in memory. This step only requires loading the first IOC rule, i.e., the first field value1 in the first IOC rule, into memory to achieve preliminary filtering, occupying little memory space and achieving high filtering efficiency.

[0059] Furthermore, in step 103, the filtered log data is precisely matched with the second IOC rule. The second IOC rule is used to filter out logs that can generate alarms. The second IOC rule includes a second field value2, and at least one of the following rule fields: a third field value3, an IOC rule type field ioc_category, and a network behavior type field tag, thereby obtaining a matching result. This matching result is used to indicate whether the filtered log data can match the second IOC rule. If the match is successful, alarm information can be generated subsequently.

[0060] The log detection method provided in this invention obtains network behavior information and parses the log data from the network behavior information. Based on the first IOC rule in memory, the log data is first filtered, that is, log data that cannot be matched with the first IOC rule at all (i.e., log data that will not generate alarms) is filtered out. The filtered log data (logs that may match the second IOC rule) is then matched with the second IOC rule in memory to obtain the matching result. Since the first IOC rule includes fewer fields than the second IOC rule, the memory space occupied by the first IOC rule is less than that occupied by the second IOC rule. Moreover, since most of the log data has already been filtered by the first IOC rule, the further matching based on the filtered log data and the second IOC rule achieves a saving in memory overhead.

[0061] Figure 2 This is a schematic diagram of another embodiment of the log detection method provided by the present invention, as shown below. Figure 2 As shown, it includes:

[0062] It should be noted that information related to network behavior can include IP_access (Internet Protocol, IP) and dns_access (Domain Name Service, DNS). IP_access collects log information of IP access, records source IP address and destination IP address information, and is periodically stored in the search engine server (Elasticsearch, ES), for example, on a daily basis. dns_access collects log information of DNS server access, records the accessed domain name and the IP address resolved based on the domain name, and is periodically stored in ES, for example, on a daily basis.

[0063] First, messages related to network behavior are consumed through the log consumption and parsing module, and log data in the network behavior information is parsed out. The parsed log data includes at least one of the following attribute information: source IP address, destination IP address, source port, destination port, domain name, and top-level private domain name.

[0064] Secondly, the Cuckoo Filter loads the first IOC rule (i.e., the first field) from the IOC rules, and matches the value of the first field with the hash checksums of the log data's various attributes to filter the log data. This filters out log data whose hash checksums cannot match the value of the first field, resulting in filtered log data.

[0065] Then, the precise matching module further queries and matches in the IOC rule base based on the hash check value and network behavior type corresponding to each attribute information of the filtered log data, combined with, for example, the second field in the second IOC rule and at least one rule field: the third field, the IOC rule type field and the network behavior type field.

[0066] Finally, the alarm generation module generates alarm information from the log data that successfully matches the second IOC rule output by the exact matching module, and writes the alarm information to another topic queue, Kafka Topic.

[0067] Furthermore, the IOC incremental update module is used to periodically fetch IOC rules from the IOC rule base and simultaneously synchronize the updated IOC rules to the Cuckoo Filter in real time.

[0068] Furthermore, the system can connect to external components or programs, such as the subscription component `setting-service` and the update component `edr-upgrade`. Specifically,

[0069] The subscription component `setting-service` is used to subscribe to updated configuration information, such as the frequency of crawling the IOC rule base, to achieve regular synchronization of IOC rules.

[0070] For example, five times a day or ten times a day;

[0071] The update component edr-upgrade is used to periodically update the IOC rule base, proactively discover threat intelligence from external or internal sources, and update the IOC rules in the IOC rule base in real time, thereby effectively protecting endpoints.

[0072] The log detection method provided in this invention filters log data in network behavior information based on a first IOC rule, specifically filtering out log data that cannot match the first IOC rule at all. The filtered log data is then further matched using a second IOC rule to obtain the matching result. In this method, log data that does not match the first IOC rule is first filtered out, while the portion of log data that matches the first IOC rule is retained in memory. The filtered log data is then matched against a second IOC rule that includes multiple fields. Because the number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule, this filtering-then-precise-match detection method saves memory overhead.

[0073] Optionally, the first IOC rule includes: a first field, the value of which corresponds to the hash verification value of the first attribute information, wherein the first attribute information is attribute information with alarm risk; the first attribute information includes at least one of the following: IP address, domain name, source port, and destination port; filtering the log data using the first IOC rule in memory to obtain filtered log data may include:

[0074] Step a: Hash at least one attribute in the log data to obtain the hash check value corresponding to each attribute.

[0075] Step b: Match each hash check value with the value of the first field in the first IOC rule, and filter the log data corresponding to the hash check values ​​that do not match the value of the first field to obtain the filtered log data.

[0076] Specifically, a complete IOC rule contains multiple fields, including at least one of the following: a first field (value1), a third field (value3), an IOC rule type field (ioc_category), and a network behavior type tag. The first IOC rule only includes the first field (value1), whose value corresponds to the hash verification value of the first attribute information. This first attribute information represents attributes with potential alarm risks. Therefore, the number of fields in the first IOC rule is significantly less than the number of fields in a complete IOC rule, thus occupying less space. For example, the first IOC rule is as follows:

[0077] {

[0078] "value1":"+++5lvucsz8nG4G6hjy72w=="

[0079] }

[0080] Furthermore, regarding hash checksums, it's essential to first understand hash functions. A hash function transforms an input of arbitrary length (also called a pre-image) into a fixed-length output using a hash algorithm. This output is the hash checksum. This transformation can be understood as a compression mapping; that is, the space of hash checksums is usually much smaller than the space of inputs. Different inputs may hash to the same output, and it's impossible to uniquely determine the input value from the hash value. Simply put, it's a function that compresses a message of arbitrary length into a message digest of a fixed length. Hash functions are primarily used in encryption algorithms in the field of information security, converting information of varying lengths into a scrambled, for example, 128-bit encoding. This encoding is called a hash value, or hash checksum. Commonly used hash algorithms include MD5 and SHA1. Furthermore, the attribute information in the log data is first processed using a hash function. This attribute information may include: IP address, domain name, top-level private domain (tpd), source port, and destination port. After hash processing, a hash verification value (i.e., hash value) corresponding to each attribute information is obtained. For example, the hash verification value corresponding to the IP address is hash(IP), the hash verification value corresponding to the domain name is hash(domain), and the hash verification value corresponding to the top-level private domain is hash(tpd). It is understood that the hash algorithm used here can be either MD5 or SHA1; this embodiment does not limit this. However, for ease of data processing, the hash algorithm used here needs to be consistent with the hash algorithm used to decrypt the first field, value1.

[0081] Furthermore, the hash verification values ​​of each attribute information calculated above can be matched with the value of the first field value1 in the first IOC rule. The value of the first field corresponds to the hash verification value of the first attribute information, which is attribute information with alarm risk. The log data corresponding to each hash verification value that does not match the value of the first field value1 is filtered. Here, matching means that as long as at least one hash verification value matches the value of the first field value1, the log data is defined as matching the first IOC rule, and then the log data is kept in memory to obtain the filtered log data.

[0082] In this embodiment, the hash checksums of the log data corresponding to each attribute are calculated, and each hash checksum is matched with the value of the first field in the first IOC rule. When at least one hash checksum matches the value of the first field, the log data is retained in memory, and all log data that does not match the first IOC rule is filtered out, resulting in filtered log data. Since the first IOC rule only includes the first field, filtering the log data by matching each hash checksum with the value of the first field is highly efficient. Based on the filtering of the first field, log data that will not match the first IOC rule at all can be filtered out, greatly saving memory space.

[0083] Optionally, the log data corresponding to hash checksums that do not match the value of the first field are filtered to obtain filtered log data, which may include:

[0084] The Cuckoo Filter is used to filter the log data corresponding to hash check values ​​that do not match the value of the first field, resulting in filtered log data.

[0085] Specifically, this method uses a cuckoo filter for filtering. The hash checksums calculated from the attributes of the log data are matched against the value of the first field, `value1`. Log data with hash checksums that do not match the value of the first field are filtered out, resulting in filtered log data. Furthermore, the cuckoo filter supports dynamic insertion and deletion of data, and also allows dynamic insertion and deletion of IOC rules within the filter.

[0086] In the method provided by this invention, log data is filtered based on the value of the first field in the first IOC rule by using a cuckoo filter. The value of the first field corresponds to the hash verification value of the first attribute information, which is attribute information with alarm risk. Therefore, the method fully considers the frequent updates of the first IOC rule and supports flexible deletion or modification of the IOC rule, making the log data obtained after filtering by the cuckoo filter more consistent with the actual situation and the filtering accuracy higher.

[0087] Optionally, the second IOC rule includes a second field, and at least one of the following rule fields: a third field, an IOC rule type field `ioc_category`, and a network behavior type field `tag`; the value of the second field corresponds to the hash checksum of the second attribute information, and the second attribute information is attribute information that can generate alarms; the second attribute information includes at least one of the following: IP address, domain name, source port, and destination port; the value of the third field is used to indicate the port number corresponding to the destination port included in the log data; the value of the IOC rule type field is used to indicate the type of the second attribute information in the second IOC rule; the value of the network behavior type field is used to indicate the network behavior type corresponding to the domain name.

[0088] Specifically, the second IOC rule includes multiple fields. Besides the second field `value2`, it may also include at least one of the following rule fields: a third field `value3`, an IOC rule type field `ioc_category`, and a network behavior type field `tag`. The value of the second field corresponds to the hash verification value of the second attribute information, which may include: IP address, domain name, source port, and destination port. It can be understood that the first field in the first IOC rule is used to filter out log data that will not generate alarms, while the second field in the second IOC rule is used to filter out log data that will generate alarms. The values ​​of the first and second fields are different and can be set according to actual needs. The value of the third field is the value used during the second IOC rule matching process to compare with the port number of the destination port in the filtered log data. The value of the IOC rule type field is the type of the second attribute information in the second IOC rule. The value of the network behavior type field is the value used during the second IOC rule matching process to compare with the network behavior type corresponding to the domain name in the filtered log data.

[0089] For example, a second IOC rule is shown below:

[0090] {

[0091] "_id":"0d416bb9e6925c2fc19493110f755f80",

[0092] "action":1,

[0093] "value2":"+++5lvucsz8nG4G6hjy73w==",

[0094] "value3":0,

[0095] "ioc_category":"TPD",

[0096] "tag":[]

[0097] }

[0098] In this embodiment of the invention, the second IOC rule includes multiple fields. In addition to the second field value2, it also includes at least one of the following rule fields: the third field value3, the IOC rule type field ioc_category, and the network behavior type field tag. Each field corresponds to a different dimension during the matching process. Log detection and rule matching are then performed based on multi-dimensional judgment conditions, making the matching results obtained based on the second IOC rule more accurate.

[0099] Optionally, the domain name also includes a top-level private domain name. In step 102, the filtered log data is matched with the second IOC rule in memory to obtain the matching result, which may include:

[0100] The attribute information contained in the filtered log data is matched with the second field and at least one rule field contained in the second IOC rule to obtain the matching result; the attribute information includes at least one of the following: domain name, IP address, destination port;

[0101] When the preset matching conditions are met, the matching result is determined to be the filtered log data matching the second IOC rule;

[0102] The preset matching criteria include at least one of the following:

[0103] The hash verification value corresponding to the top-level private domain in the filtered log data is the value of the second field, and the value of the IOC rule type field is the top-level private domain type;

[0104] The hash verification value corresponding to the domain name in the filtered log data is the value of the second field, and the value of the IOC rule type field is a non-top-level private domain name type;

[0105] In the filtered log data, the hash check value corresponding to the IP address is the value of the second field, the value of the IOC rule type field is a non-top-level private domain name type, and the port number corresponding to the destination port is the value of the third field.

[0106] In the filtered log data, the hash checksum corresponding to the IP address is the value of the second field, and the value of the IOC rule type field is empty.

[0107] Specifically, the second IOC rule contains multiple fields, including the second field value2, and at least one of the following rule fields: the third field value3, the IOC rule type field ioc_category, and the network behavior type field tag. The value of the IOC rule type field ioc_category can be understood as the internal matching logic of the IOC rule, that is, matching the attribute information in the filtered log data with the second IOC rule, that is, the type of the second attribute information in the second IOC rule, to determine whether a match with the second IOC rule is successful.

[0108] For example, the value of the IOC rule type field `ioc_category` can be "Top-level private domain type TPD", "DOMAIN_PORT", or "IP_PORT". These values ​​represent the following meanings: the type of the second attribute information in the second IOC rule is the top-level private domain `tpd`, meaning that the top-level private domain `tpd` in the filtered log data is matched with the second IOC rule; the type of the second attribute information in the second IOC rule is domain name and port number, meaning that the (original) domain name and port number in the filtered log data is matched with the second IOC rule; and the type of the second attribute information in the second IOC rule is IP address and port number, meaning that the IP address and port number in the filtered log data is matched with the second IOC rule.

[0109] The log data filtered in step 102 consists of log data whose hash checksum is the value of the first field (IP address or domain name). Further, the filtered log data is matched against the second IOC rule in memory, as follows:

[0110] First, the attribute information contained in the filtered log data is matched against the second field and at least one rule field contained in the second IOC rule to obtain the matching result. The attribute information contained in the filtered log data includes at least one of the following: domain name (including top-level private domains), IP address, and destination port. The second IOC rule may include the second field value2, and at least one of the following rule fields: the third field value3, the IOC rule type field ioc_category, and the network behavior type field tag. Specifically, the matching process involves matching the hash checksum of each attribute information in the filtered log data with the value of the second field value2 in the second IOC rule, and matching the port number (access port number) of the destination port included in the log data with the value of the third field value3 included in the second IOC rule. When the preset matching conditions are met, the matching result is determined to be a match between the filtered log data and the second IOC rule.

[0111] Furthermore, matching conditions that need to be met can be preset in advance. When the preset matching conditions are met, the matching result is determined as the filtered log data matching the second IOC rule. The specific matching conditions can be any of the following:

[0112]

Matching Condition 1

[0113]

Matching Condition 2

[0114]

Matching Condition 3

[0115]

Matching Condition 4

[0116] For example, suppose the second IOC rule is as follows:

[0117] {

[0118] "_id":"0d416bb9e6925c2fc19493110f755f80",

[0119] "action":1,

[0120] "value2":"+++5lvucsz8nG4G6hjy73w==",

[0121] "value3":0,

[0122] "ioc_category":"TPD",

[0123] "tag":[]

[0124] }

[0125] In the filtered log data, hash(IP) is the value of the second field value2, the IOC rule type field ioc_category is "IP_PORT" (i.e., a non-top-level private domain type), and the port number corresponding to the destination port is the value of the third field value3. Therefore, the filtered log data meets the above [Matching Condition 4], and the matching result is output as follows:

[0126] The filtered log data matches the second IOC rule.

[0127] In the method provided by this embodiment of the invention, based on multiple fields included in the second IOC rule, the attribute information included in the filtered log data is matched with the values ​​of each field in the second IOC rule. For example, the hash checksum values ​​corresponding to the domain name, top-level private domain name, IP address, and other attribute information in the filtered log data are matched with the values ​​of the second field in the second IOC rule; the port number corresponding to the destination port is matched with the value of the third field. Then, when a preset matching condition is met, the matching result is determined to be a match between the filtered log data and the second IOC rule. Because the second IOC rule contains multiple fields, matching based on the values ​​of multiple fields, that is, comprehensive judgment through multiple indicators, can reduce the probability of false positives and false negatives in the log data, obtain more accurate matching results, and further improve the accuracy of log detection.

[0128] Optionally, after step 103 matches the filtered log data with the second IOC rule in memory and obtains the matching result, the method may further include:

[0129] Generate alarm information.

[0130] Specifically, after obtaining the above matching results, the method can also generate alarm information based on the matching results, that is, identify the IP address or domain name in the access IOC rule in the successfully matched log data, and write the alarm information into a specific topic.

[0131] The log detection method provided in this embodiment of the invention provides an alert for matched log data, which facilitates subsequent monitoring and maintenance of network security behavior based on the alert information.

[0132] Optionally, the first IOC rule and / or the second IOC rule are stored in a preset IOC rule base. The method may further include:

[0133] The first IOC rule and / or the second IOC rule in the IOC rule base are cached in the memory of an electronic device, which is used to execute the log detection method.

[0134] Specifically, in the method provided in this embodiment, an IOC rule base can be preset in advance and stored on, for example, an electronic device. The IOC rule base stores multiple first IOC rules and / or second IOC rules. When the log detection method is executed, the first IOC rules and / or second IOC rules that need to be used in the rule base are cached in memory in real time.

[0135] The log detection method provided in this embodiment of the invention can cache the first IOC rule and / or the second IOC rule needed in real time when performing log detection by pre-setting an IOC rule library. The log detection method based on the first IOC rule and the second IOC rule is highly efficient.

[0136] Optionally, the method may further include:

[0137] When the IOC rules in the IOC rule base are updated, the first IOC rule and / or the second IOC rule cached in memory are updated.

[0138] Specifically, the preset IOC rule base can be updated according to actual needs, and the update frequency can also be set. For example, system monitoring personnel can set the update configuration information, such as the preset IOC rule update frequency. In addition, system monitoring personnel can also actively discover threatening intelligence information IOCs by setting up monitoring programs, and update the detected threatening intelligence information IOCs into the IOC rule base.

[0139] Furthermore, if the IOC rules in the IOC rule base are updated, the first IOC rule and / or the second IOC rule cached in memory can be updated.

[0140] The log detection method provided in this embodiment of the invention synchronizes the updated rules of the IOC rule base to the first IOC rule and / or the second IOC rule loaded in the cache in real time, making the accuracy of log detection based on the first IOC rule and / or the second IOC rule more in line with the actual situation.

[0141] The log detection device provided by the present invention is described below. The log detection device described below and the log detection method described above can be referred to in correspondence.

[0142] Figure 3 This is a schematic diagram of the log detection device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the log detection device includes: an acquisition module 310, a filtering module 320, and a matching module 330. Wherein:

[0143] The acquisition module 310 is used to acquire network behavior information and parse the network behavior information to obtain log data;

[0144] The filtering module 320 uses the first IOC rule in memory to filter the log data to obtain filtered log data. The first IOC rule is used to filter out log data that will not generate alarms.

[0145] The matching module 330 is used to match the filtered log data with the second IOC rule in memory to obtain a matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule.

[0146] The log detection device provided by this invention acquires network behavior information through the acquisition module 310 and parses the log data in the network behavior information. The filtering module 320 filters the log data based on the first IOC rule in memory, that is, filters out log data that cannot match the first IOC rule at all (i.e., filters out log data that will not generate alarms). The matching module 330 further matches the filtered log data (logs that may match the second IOC rule) with the second IOC rule in memory to obtain the matching result. Since the first IOC rule includes fewer fields than the second IOC rule, the memory space occupied by the first IOC rule is less than that occupied by the second IOC rule. Moreover, most of the log data has been filtered by the first IOC rule. Further matching based on the filtered log data and the second IOC rule achieves memory saving.

[0147] Optionally, the first IOC rule includes: a first field, the value of which corresponds to the hash verification value of the first attribute information, wherein the first attribute information is attribute information with alarm risk; the first attribute information includes at least one of the following: IP address, domain name, source port, and destination port;

[0148] The filtering module 320 is specifically used for:

[0149] Hash at least one attribute information in the log data to obtain a hash check value corresponding to each attribute information;

[0150] Each hash check value is matched with the value of the first field in the first IOC rule, and the log data corresponding to the hash check values ​​that do not match the value of the first field are filtered to obtain the filtered log data.

[0151] Optionally, the filtering module 320 is specifically used for:

[0152] The log data corresponding to hash check values ​​that do not match the value of the first field are filtered using the Cuckoo Filter to obtain the filtered log data.

[0153] Optionally, the second IOC rule includes a second field, and at least one of the following rule fields: a third field, an IOC rule type field, and a network behavior type field; the value of the second field corresponds to the hash check value of the second attribute information, which is attribute information capable of generating alarms; the second attribute information includes at least one of the following: IP address, domain name, source port, and destination port; the value of the third field is used to indicate the port number corresponding to the destination port included in the log data; the value of the IOC rule type field is used to indicate the type of the second attribute information in the second IOC rule; the value of the network behavior type field is used to indicate the network behavior type corresponding to the domain name.

[0154] Optionally, the domain name may also include a top-level private domain name;

[0155] Matching module 330 is specifically used for:

[0156] The attribute information contained in the filtered log data is matched with the second field and at least one rule field contained in the second IOC rule to obtain a matching result; the attribute information contained in the filtered log data includes at least one of the following: the domain name, the IP address, and the destination port;

[0157] When the preset matching conditions are met, the matching result is determined to be that the filtered log data matches the second IOC rule;

[0158] The preset matching conditions include at least one of the following:

[0159] The hash verification value corresponding to the top-level private domain in the filtered log data is the value of the second field, and the value of the IOC rule type field is the top-level private domain type;

[0160] The hash verification value corresponding to the domain name in the filtered log data is the value of the second field, and the value of the IOC rule type field is a non-top-level private domain name type;

[0161] The hash verification value corresponding to the IP address in the filtered log data is the value of the second field, the value of the IOC rule type field is a non-top-level private domain name type, and the port number corresponding to the destination port is the value of the third field.

[0162] The hash verification value corresponding to the IP address in the filtered log data is the value of the second field, and the value of the IOC rule type field is empty.

[0163] Optionally, the log detection device further includes:

[0164] The alarm generation module is used to generate alarm information.

[0165] Optionally, the log detection device further includes:

[0166] A caching module is used to cache the first IOC rule and / or the second IOC rule in the IOC rule base into the memory of an electronic device, which is used to execute the log detection method.

[0167] Optionally, the log detection device further includes:

[0168] An update module is used to update the first IOC rule and / or the second IOC rule cached in memory when the IOC rules in the IOC rule base are updated.

[0169] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a log detection method, which includes:

[0170] Obtain network behavior information and parse the network behavior information to obtain log data;

[0171] The log data is filtered using the first IOC rule in memory to obtain filtered log data. The first IOC rule is used to filter out log data that will not generate alarms.

[0172] The filtered log data is matched with the second IOC rule in memory to obtain a matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule.

[0173] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the log detection method provided by the above methods, the method including:

[0175] Obtain network behavior information and parse the network behavior information to obtain log data;

[0176] The log data is filtered using the first IOC rule in memory to obtain filtered log data. The first IOC rule is used to filter out log data that will not generate alarms.

[0177] The filtered log data is matched with the second IOC rule in memory to obtain a matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule.

[0178] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the log detection method provided by the methods described above, the method comprising:

[0179] Obtain network behavior information and parse the network behavior information to obtain log data;

[0180] The log data is filtered using the first IOC rule in memory to obtain filtered log data. The first IOC rule is used to filter out log data that will not generate alarms.

[0181] The filtered log data is matched with the second IOC rule in memory to obtain a matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule.

[0182] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log detection method, characterized in that, include: Obtain network behavior information and parse the network behavior information to obtain log data; The log data is filtered using a first IOC rule in memory to obtain filtered log data. The first IOC rule is used to filter out log data that will not generate alarms. The filtered log data accounts for 10% to 20% of the total log data. The filtered log data is matched with the second IOC rule in memory to obtain the matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule. The first IOC rule includes: a first field, the value of which corresponds to the hash verification value of the first attribute information, wherein the first attribute information is attribute information that has an alarm risk; the first attribute information includes at least one of the following: IP address, domain name, source port, and destination port; The process of filtering the log data using the first IOC rule in memory to obtain filtered log data includes: Hash at least one attribute information in the log data to obtain a hash check value corresponding to each attribute information; Each hash check value is matched with the value of the first field in the first IOC rule, and the log data corresponding to the hash check values ​​that do not match the value of the first field are filtered to obtain the filtered log data.

2. The log detection method according to claim 1, characterized in that, The step of filtering the log data corresponding to hash verification values ​​that do not match the value of the first field to obtain filtered log data includes: The log data corresponding to hash check values ​​that do not match the value of the first field are filtered using the Cuckoo Filter to obtain the filtered log data.

3. The log detection method according to claim 1, characterized in that, The second IOC rule includes a second field, and at least one of the following rule fields: a third field, an IOC rule type field, and a network behavior type field; the value of the second field corresponds to the hash check value of the second attribute information, which is attribute information capable of generating alarms; the second attribute information includes at least one of the following: IP address, domain name, source port, and destination port; the value of the third field is used to indicate the port number corresponding to the destination port included in the log data; the value of the IOC rule type field is used to indicate the type of the second attribute information in the second IOC rule; the value of the network behavior type field is used to indicate the network behavior type corresponding to the domain name.

4. The log detection method according to claim 3, characterized in that, The domain name also includes a top-level private domain name. The step of matching the filtered log data with the second IOC rule in memory to obtain the matching result includes: The attribute information contained in the filtered log data is matched with the second field and at least one rule field contained in the second IOC rule to obtain a matching result; the attribute information contained in the filtered log data includes at least one of the following: the domain name, the IP address, and the destination port; When the preset matching conditions are met, the matching result is determined to be that the filtered log data matches the second IOC rule; The preset matching conditions include at least one of the following: The hash verification value corresponding to the top-level private domain in the filtered log data is the value of the second field, and the value of the IOC rule type field is the top-level private domain type; The hash verification value corresponding to the domain name in the filtered log data is the value of the second field, and the value of the IOC rule type field is a non-top-level private domain name type; The hash verification value corresponding to the IP address in the filtered log data is the value of the second field, the value of the IOC rule type field is a non-top-level private domain name type, and the port number corresponding to the destination port is the value of the third field. The hash verification value corresponding to the IP address in the filtered log data is the value of the second field, and the value of the IOC rule type field is empty.

5. The log detection method according to claim 1, characterized in that, After matching the filtered log data with the second IOC rule in memory to obtain the matching result, the process further includes: Generate alarm information.

6. The log detection method according to claim 1, characterized in that, The first IOC rule and / or the second IOC rule are stored in a preset IOC rule base, and the method further includes: The first IOC rule and / or the second IOC rule in the IOC rule base are cached in the memory of the electronic device, which is used to execute the log detection method.

7. The log detection method according to claim 6, characterized in that, The method further includes: If the IOC rules in the IOC rule base are updated, the first IOC rule and / or the second IOC rule cached in memory are updated.

8. A log detection device, characterized in that, include: The acquisition module is used to acquire network behavior information and parse the network behavior information to obtain log data; The filtering module is used to filter the log data using the first IOC rule in memory to obtain filtered log data. The first IOC rule is used to filter out log data that will not generate alarms. The filtered log data accounts for 10% to 20% of the total log data. The first IOC rule includes: a first field, the value of which corresponds to a hash verification value of first attribute information, wherein the first attribute information is attribute information with alarm risk; the first attribute information includes at least one of the following: IP address, domain name, source port, and destination port; the filtering module is specifically used to perform hash processing on at least one attribute information in the log data to obtain a hash verification value corresponding to each attribute information; match each hash verification value with the value of the first field in the first IOC rule, and filter the log data corresponding to hash verification values ​​that do not match the value of the first field to obtain filtered log data; The matching module is used to match the filtered log data with the second IOC rule in memory to obtain the matching result. The second IOC rule is used to filter out log data that can generate alarms. The number of fields included in the first IOC rule is less than the number of fields included in the second IOC rule.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the log detection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the log detection method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the log detection method as described in any one of claims 1 to 7.

Images