A log data compression method based on storage structure
Through the log data compression method based on storage structure, keywords, high-frequency words and session templates are used for digitization, and log entries with the same adjacent semantics are merged, the problem of log data storage and calculation overhead in APT attack is solved, and the problem of log data storage and calculation overhead is realized. Efficient log data compression and analysis.
Patent Information
- Application Number
- CN202210507055.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-10
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-05-10
AI Technical Summary
During the process of defending against APT attacks, the collection, analysis and storage of log data lead to huge storage and computing overhead, especially when the high concealment and persistence of APT attacks make it difficult to detect in the early stage, a large amount of log data is required to store for the purpose of performing the Follow-up security analysis.
Using a log data compression method based on storage structure, log data is converted into digital code through keyword templates, high-frequency word templates and session templates, and stored in formats of monotonic values and timestamps, and log entries with the same semantics are performed. Merge to achieve lossless compression.
It effectively reduces the storage amount and calculation overhead of log data, maintains data integrity and recoverability, and improves the compression efficiency and compatibility of log data.
Smart Images

Figure CN114860678B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of information security, and in particular relates to a log data compression method based on a storage structure. Background Art
[0002] Logs can be used to assist in system debugging, performance measurement, troubleshooting, security testing, etc. In security testing, the value of logs is particularly important. Many security applications, including but not limited to intrusion detection, system monitoring, attack investigation, and attack source tracing analysis, all require the collection and analysis of logs to assist in implementation.
[0003] The current mainstream detection method, Endpoint Detection and Response (EDR), deploys log collection tools on the target host to collect system activity information in order to monitor terminal host activities in restricted networks. Activity information includes file, process, network and other related events, as well as events with high security relevance (such as login attempts, privilege escalation). In order to complete the subsequent analysis and tracing of the source of the attack, EDR has to store a large amount of collected log data.
[0004] For the current APT (Advanced Persistent Threat) attacks, which are extremely harmful and difficult to detect, the collection, analysis and storage of logs are particularly important. For the defenders of APT attacks, when security analysts investigate the APT attacks performed by attackers in the system, logs are the only source that security analysts can use to understand the consequences of the attacks and trace the root causes of the attacks. In APT detection and defense, due to the high concealment and persistence of APT attacks, it is often difficult to detect them in the early stages of the attack. After the initial intrusion is successful, APT attackers often choose to lurk in order to gain greater benefits, and their lurking time can be as long as months or even years. In order to meet the needs of subsequent security analysts' investigation and tracing, defenders often need to store a large amount of log data.
[0005] Whether it is to support the operation of EDR systems or to support APT detection and defense, collecting, analyzing, and storing logs are very important. Storing large amounts of log data is a basic challenge for these log-based systems. Large amounts of log data not only bring huge storage overhead, but also directly increase the computing overhead of system operation and analysis. Summary of the invention
[0006] The purpose of the present invention is to provide a log data compression method based on a storage structure to solve the huge storage overhead and additional computing overhead problems caused by the large amount of log data required for detection and forensic analysis in the process of defending against APT attacks.
[0007] To achieve the above object, the technical solution adopted by the present invention is:
[0008] A log data compression method based on a storage structure, the log data compression method based on a storage structure comprising:
[0009] Step 1: Collect log data;
[0010] Step 2: compressing the log data based on the template to convert the log data into initial compressed data, including:
[0011] Step 2.1, according to the event type of each log entry in the log data, using a keyword template to convert the keyword in the log entry into a preset keyword digital code to obtain first compressed data;
[0012] Step 2.2: based on the high-frequency word template, convert the high-frequency words in the first compressed data into preset high-frequency word digital codes to obtain second compressed data;
[0013] Step 2.3, converting the repeated values of all log entries belonging to the same application session in the second compressed data into a preset session digital code based on the session template to obtain the initial compressed data;
[0014] Step 3: Perform log data compression based on monotonic values on the initial compressed data, store the monotonic values in a format of initial value plus deviation, and obtain middle compressed data, wherein the monotonic values include the timestamp in the log data;
[0015] Step 4: merge the adjacent log entries with the same semantics in the middle compressed data to obtain the final compressed data, thus completing the log data compression.
[0016] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.
[0017] Preferably, the method of converting a keyword in a log entry into a preset keyword digital code using a keyword template according to the event type of each log entry in the log data to obtain the first compressed data includes:
[0018] extracting a keyword numeric code from a keyword template according to an event type of the log entry;
[0019] Extract the values of the key-value pairs in the original format of the log entry in the order of the original format;
[0020] The extracted keyword numeric code is combined with the value as the transformed log entry.
[0021] Preferably, the high-frequency words include file paths.
[0022] Preferably, merging adjacent log entries with the same semantics in the middle compressed data includes:
[0023] Take adjacent log entries with the same semantics as equivalent entries;
[0024] The equivalent entry with the smallest timestamp among the equivalent entries is retained, and the rest of the equivalent entries are deleted;
[0025] A number indicating the number of deleted equivalent entries is added to the value of the key-value pair of the timestamp of the retained equivalent entry to complete the merge.
[0026] The log data compression method based on the storage structure provided by the present invention has the following beneficial effects compared with the prior art:
[0027] 1) This application is specially optimized from multiple angles based on the format of log data, including keywords, high-frequency words, and repeated values, which is more targeted and thus has a better compression effect.
[0028] 2) Since this application is optimized based on the format of log data, it can be optimized, adjusted, and expanded according to actual conditions, and has good compatibility and scalability.
[0029] 3) In the compression process of this application, the log entries are not completely deleted, and all compressed logs can be completely restored, so this compression method is lossless compression. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a flow chart of the log data compression method based on storage structure of the present invention;
[0031] Figure 2 The present invention is a flow chart of performing data compression on log data collected by the SPADE tool. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0034] like Figure 1 As shown, a log data compression method based on a storage structure includes:
[0035] Step 1: Collect log data.
[0036] There are many log collection tools currently available, and you can select one according to your needs during actual operation. This embodiment takes the SPADE tool as an example to illustrate how to use the SPADE tool to collect Linux Audit log data.
[0037] Deploy the log collection tool SPADE on the target Linux host. The SPADE tool is a log collection tool developed based on Audit. The collected log information is optimized based on the Audit log. The collected log data is transferred to the host where the compression system is located for subsequent compression.
[0038] It is easy to understand that if the compression system and the log collection tool are deployed on the same host, the activities generated by the compression system will also be collected by the log collection tool, which is of no value for analyzing the activities of the target host. Therefore, in this embodiment, the compression system is deployed on another host.
[0039] The log format collected by the SPADE tool is as follows. For example, the log entry corresponding to a Read event contains the following main information: operation object information, operation event type, operation object information, and operation timestamp: {"datum":{"uuid":[-68,-56,-9,40,66,66,-75,114,56,62,37,-86,-126,-73,104,-77],"sequence":64,"type":"EVENT_READ ","threadId":1027,"subject":[13,-7,-70,41,-29,124,-81,88,-11,23,-74,-53,52,-95,127,7],"predi cateObject":[27,-97,72,-90,90,74,-99,-116,-101,-17,76,55,122,119,12,22],"predicateObjectPath" : / etc / sysconfig / network,"predicateObject2":null,"predicateObject2Path":null,"timestampNanos":1587004392617000000,"names":null,"parameters":null,"location":null,"size":36,"programPoint": null,"properties":{}},"CDMVersion":"19","type":"RECORD_EVENT","hostId":[9,-93,-13,-34,-49,-28 ,11,90,82,-113,64,-2,94,-52,-68,82],"sessionNumber":9,"source":"SOURCE_LINUX_SYSCALL_TRACE"}.
[0040] Step 2: Perform template-based log data compression on the log data to convert the log data into initial compressed data.
[0041] This embodiment compresses the collected log data based on the template, converts the original log data into a simpler format for storage. The template mainly involves: keywords, repeated high-frequency words, and repeated values in the same session. The specific operation is as follows.
[0042] Step 2.1: According to the event type of each log entry in the log data, a keyword in the log entry is converted into a preset keyword digital code using a keyword template to obtain first compressed data.
[0043] Log entries in log data are composed of keywords and values, that is, key-value pairs. Take "timestampNanos":1587004392617000000 as an example, where timestampNanos is the keyword (key) and 1587004392617000000 is the value (value).
[0044] In the log, the keywords are limited, while the values are infinitely changing. It is the combination of these limited keywords and infinitely changing values that can generate countless different log entries. In the Linux Audit log, although SPADE can collect millions of log data, there are only a limited number of different types of logs. Therefore, this type of log entry can be converted and the keywords in it can be replaced with more concise digital codes. The specific implementation method is as follows.
[0045] Extract keyword numeric codes from keyword templates according to event types of log entries; extract values of key-value pairs in the original format of log entries in the order of the original format; combine the extracted keyword numeric codes and values as converted log entries.
[0046] For the convenience of explanation, a simplified log entry is used as an example, for example, a log entry of the Read event type: {uuid:93,time:1672,suject:21,object:22,type:read}. The above log entry is converted to a keyword, and the keyword numeric code 01 is used to represent the log entry of the Read event type, and the original keyword is removed. The converted log entry is: (01:{93,1672,21,22,01}), and the value of the keyword type read is also replaced by the keyword array code of the corresponding event type.
[0047] Considering the convenience of actual storage, it is usually stored in the format of 01,93,1672,21,22,01. At the same time, a reference code r is added, and the conversion rule corresponding to the reference code r is marked as It means that the decompression program can restore the original keyword order in the log entry through 01 and the reference code r. Here, "01" is the digital code at the front of the converted log entry, that is, the digital code that replaces all the keywords in the log entry is placed at the front of the converted log entry. The "01" corresponding to the value of the keyword type read in the log entry is only for simplification. During the decompression process, the program will query the reference table to find the corresponding translation rules, and restore the compressed log data to its original appearance according to the rules.
[0048] This embodiment designs an extensible keyword template as shown in Table 1 for the event types of common Linux system logs that SPADE can collect:
[0049] Table 1 Keyword template
[0050]
[0051]
[0052] In this embodiment, the events collected by the SPADE tool are converted into a 2-digit code, which is added to the front end of the converted log. At the same time, the table can be expanded as the log collection tool is optimized. Although there are 257 types of kernel system calls in total, there are usually only dozens of commonly used types, such as those used in SPADE, so it is completely sufficient to replace them with a 2-digit code.
[0053] It is easy to understand that Table 1 is only one implementation of this embodiment. In other embodiments, different event types can be obtained according to different log collection tools and different systems, and their corresponding keyword array codes can be freely set, and the format of the corresponding event type will also change accordingly.
[0054] Step 2.2: based on the high-frequency word template, convert the high-frequency words in the first compressed data into preset high-frequency word digital codes to obtain second compressed data.
[0055] Log data often contains file paths. For example, during file reading and writing operations, the log records information about the file path, etc. During process creation, execution, fork / clone, etc., the log records information about the process path, etc. After analyzing the log data, we found that many logs with different entries record the same data.
[0056] For example, when process A and process B read file F successively, the description of file F information will appear twice in the two corresponding log entries generated. Therefore, the information of the path where file F is located will be recorded twice. For another example, when process A reads and writes files E and F in the same folder, the description of file E information and file F information will appear in the two corresponding log entries generated. File E and file F belong to the same folder (assuming that this folder is / etc / sysconfig / network), then the file path / etc / sysconfig / network will be recorded twice in two different log entries. This situation occurs very frequently on Linux servers. When web services such as tomcat7 are deployed on Linux servers, all web page files that need to be accessed will be placed in the directory / var / lib / tomcat7 / webapps by default. Therefore, when someone accesses a web page located on the server, access records of files in this directory will be continuously generated.
[0057] In summary, this embodiment converts the high-frequency character strings (mainly file paths, and also some system parameters) that may appear in the target host, and also uses more concise digital codes to represent them. For example, in the above log entry example, the path / etc / sysconfig / network will be converted to 01. A high-frequency word digital code and high-frequency character string translation reference table is generated, as shown in Table 2. The table can be modified by the administrator according to the actual situation of the host. For example, the character strings that appear more than 10 times in the log are set as high-frequency words and the corresponding information is added to Table 2. If there are hundreds or more high-frequency words, the corresponding value can be increased from 2 digits to 3 digits.
[0058] Table 2 High frequency word templates
[0059] High frequency string High frequency word numeric codes / etc / sysconfig / network 01 / var / lib / tomcat7 / webapps 02 / etc / hosts 03 / etc / bashrc 04 / boot / grub / grub.conf 05 / etc / sysconfig / network-scripts / ifcfg-ethx 06 ~ / .bash_profile 07 / usr / share / mysql / 08 / var / lib / mysql / 09 / etc / my.cnf 10 … …
[0060] Step 2.3: convert the repeated values of all log entries belonging to the same application session in the second compressed data into a preset session digital code based on the session template to obtain the initial compressed data.
[0061] By analyzing the application behaviors in the logs and the log entries belonging to the same application session, we can see that applications and systems are often distinguished by application sessions when they are operating. The logs generated by the behaviors belonging to the same application session usually share a lot of information. For example, in Linux system logs, the values of pid, ppid (parent process id), hostname, etc. in the logs generated by the same process are the same, and such repeated values will bring additional storage overhead.
[0062] Therefore, this embodiment analyzes common session data, extracts the repeated parts, and uses the same conversion idea as step 2.1 to replace the repeated values with numbers. The session-related fields are summarized into tuples stored in the mapping table. The optimization rule defines the conversion rules between actual values and compressed values. Similar to step 2.1, after the log is compressed, a reference code will be added to the log. For example, for (pid:76, ppid:70, hostname:ubuntu), the reference code t is used to map it to (pid:0, ppid:0, hostname:0). In the session template, i.e., the mapping table (Table 3 is an example of the mapping table), a tuple will be added, and its content is: In this way, all log entries belonging to the same session whose pid, ppid, and hostname values are consistent with the above (76,70,ubuntu) can be replaced with (0,0,0). Unlike step 2.1, which optimizes keywords, step 2.2 focuses more on optimizing repeated values.
[0063] Table 3 Session template
[0064] Raw value (pid, ppid, hostname) Conversion value (pid, ppid, hostname) (76,70,ubuntu) (0,0,0) (69,41,ubuntu) (1,1,0) (57,29,ubuntu) (2,2,0) … …
[0065] Step 3: Perform log data compression based on monotonic values on the initial compressed data, store the monotonic values in the format of initial value plus deviation, and obtain the middle compressed data.
[0066] The log compressed in step 2 is compressed based on the monotonic value, the monotonic values in the log data are converted, and the monotonic values are stored in a more concise manner. The monotonic value referred to in this embodiment includes: the timestamp in the log data.
[0067] In continuous log data, the values corresponding to the keyword timestamp all show a monotonically increasing trend, so these monotonous values can be represented by the initial value + deviation method. Most of the timestamps used in log data are Unix timestamps, whose values are the number of seconds (excluding leap seconds) that have passed since January 1, 1970 (midnight in UTC / GMT). Referring to the log entry example described above, its timestamp is represented as: "timestampNanos":1587004392617000000. This string is too long, but it will be much simpler if it is represented by the initial value + deviation method. Set the initial value to 1587004392616 (the 6 digits of 0 following timestampNanos in the log collected by SPADE are the time under high precision, and after removing the last 6 digits, it is still the millisecond-level accuracy, and in the log collected by SPADE, these 6 digits are always 0, so they have no practical meaning). For this log, its deviation value is 1, so you only need to record "timestampNanos":1. In actual use, the initial value can be set to the timestamp corresponding to the first log entry collected by the system.
[0068] Step 4: merge the adjacent log entries with the same semantics in the middle compressed data to obtain the final compressed data, thus completing the log data compression.
[0069] The log compressed in step 3 is further optimized by merging the log information with the same semantics in adjacent logs to leave an integrated log entry, and the timestamp of the merged log entry is added to the timestamp of the integrated log entry.
[0070] In the collected logs, there are often consecutive equivalent log entries. For example, process A writes to file E at time point 5 and time point 6. For the two corresponding log entries generated by its operation, the operation object information, operation event type, and operation object information are the same. In this embodiment, two log entries with similar situations are called semantically identical, or equivalent, that is, adjacent log entries with the same semantics are taken as equivalent entries.
[0071] For these log entries with exactly the same semantics, this embodiment adopts a merge operation, retaining only the first recorded log entry with the same semantics (the one with the smallest timestamp), deleting all equivalent entries after it, and adding an extra bit after the timestamp of the retained log to indicate the deleted equivalent log data.
[0072] That is, the equivalent entry with the smallest timestamp among the equivalent entries is retained, and the remaining equivalent entries are deleted; a number representing the number of deleted equivalent entries is added to the value of the key-value pair of the timestamp of the retained equivalent entry to complete the merge. For example, three consecutive equivalent logs {uuid:93,time:1672,suject:21,object:22,type:read}; {uuid:94,time:1673,suject:21,object:22,type:read}; {uuid:95,time:1674,suject:21,object:22,type:read} can be merged into {uuid:93,time:[1672,2],suject:21,object:22,type:read}. In the subsequent decompression process, the two deleted equivalent logs can be restored through the value 2 corresponding to the time keyword.
[0073] The uuid indicates the identification number of the log, which is similar to a serial number. Its value is irrelevant to the log information, so only the value in the equivalent entry is retained after merging. After the above optimization, the collected log data has been compressed to a great extent, solving the huge storage overhead and additional computing overhead caused by a large amount of log data.
[0074] In another embodiment, Figure 2 As shown, the logs collected by the SPADE tool can be further optimized by deleting some meaningless data, such as SPADE version information, Linux system information, etc., and setting the data with null value to -1. This capture can be customized according to different collection tools.
[0075] Taking the log entries collected by SPADE as an example, the log entries after compression and conversion in the above steps are:
[0076] Original data: {"datum":{"uuid":[-68,-56,-9,40,66,66,-75,114,56,62,37,-86,-126,-73,104,-77]," sequence":64,"type":"EVENT_READ","threadId":1027,"subject":[13,-7,-70,41,-29,124,-81 ,88,-11,23,-74,-53,52,-95,127,7],"predicateObject":[27,-97,72,-90,90,74,-99,-116,-1 01,-17,76,55,122,119,12,22],"predicateObjectPath": / etc / sysconfig / network,"predicateO bject2":null,"predicateObject2Path":null,"timestampNanos":1587004392617000000,"names":null,"parameters":null,"location":null,"size":36,"programPoint":null,"properties" :{}},"CDMVersion":"19","type":"RECORD_EVENT","hostId":[9,-93,-13,-34,-49,-28,11,90,8 2,-113,64,-2,94,-52,-68,82],"sessionNumber":9,"source":"SOURCE_LINUX_SYSCALL_TRACE"}
[0077] The compressed log data is as follows: {{01,[-68,-56,-9,40,66,66,-75,114,56,62,37,-86,-126,-73,104,-77],64,01,1027,[13,-7,-70,41,-29,124,-81,88,-11,23,-74,-53,52,-95,127,7],[27,- 97,72,-90,90,74,-99,-116,-101,-17,76,55,122,119,12,22],01,-1,-1,[1,0],-1,-1,-1,36,-1,},01,[9,-93,-13,-34,-49,-28,11,90,82,-113,64,-2,94,-52,-68,82],9}
[0078] Among them, "datum", "CDMVersion", "type", "hostId", "sessionNumber", "source", "CDMVersion":"19", and "source":"SOURCE_LINUX_SYSCALL_TRACE" are the data added when the SPADE tool is collected and have nothing to do with the log data, so they are directly removed during compression and automatically added when the data is compressed. The first "01" in the compressed data represents the keyword digital code corresponding to the log entry with event type Read. The third "01" in the compressed data is the high-frequency word digital code corresponding to the high-frequency word / etc / sysconfig / network. The "timestampNanos":1587004392617000000 in the original data is converted to [1,0], where 1 is the deviation relative to the initial value, and 0 means that the log entry is not a merger of multiple log entries, but an original single log entry. In other embodiments, the converted value of the timestamp in the original single log entry may also only show the deviation relative to the initial value instead of 0.
[0079] From the comparison of the log entries before and after the above conversion, it can be seen that the letters in the log entries after conversion by the compression method of this embodiment are greatly reduced or there are no letters, and the storage structure is more standardized, without unnecessary colons, quotation marks, etc., and all are separated by commas. Relatively speaking, the amount of data is greatly reduced, effectively reducing the pressure of storing a large amount of log data.
[0080] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0081] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.
Claims
1. A log data compression method based on storage structure, It is characterized in that The log data compression method based on the storage structure includes: Step 1: Collect log data; Step 2: compressing the log data based on the template to convert the log data into initial compressed data, including: Step 2.1, according to the event type of each log entry in the log data, using a keyword template to convert the keyword in the log entry into a preset keyword digital code to obtain first compressed data; Step 2.2: based on the high-frequency word template, convert the high-frequency words in the first compressed data into preset high-frequency word digital codes to obtain second compressed data; Step 2.3, converting the repeated values of all log entries belonging to the same application session in the second compressed data into a preset session digital code based on the session template to obtain the initial compressed data; Step 3: Perform log data compression based on monotonic values on the initial compressed data, store the monotonic values in a format of initial value plus deviation, and obtain middle compressed data, wherein the monotonic values include the timestamp in the log data; Step 4: merge adjacent log entries with the same semantics in the middle compressed data to obtain the final compressed data, and complete the log data compression, including: Take adjacent log entries with the same semantics as equivalent entries; The equivalent entry with the smallest timestamp among the equivalent entries is retained, and the rest of the equivalent entries are deleted; A number indicating the number of deleted equivalent entries is added to the value of the key-value pair of the timestamp of the retained equivalent entry to complete the merge.
2. The log data compression method based on the storage structure as claimed in claim 1, It is characterized in that The method of converting the keywords in the log entry into a preset keyword digital code using a keyword template according to the event type of each log entry in the log data to obtain the first compressed data includes: extracting a keyword numeric code from a keyword template according to an event type of the log entry; Extract the values of the key-value pairs in the original format of the log entry in the order of the original format; The extracted keyword numeric code is combined with the value as the transformed log entry.
3. The log data compression method based on the storage structure as claimed in claim 1, It is characterized in that The high frequency words include file paths.
