Automatic labeling method combined with multi-source log data

Abstract

The invention discloses an automatic labeling method combined with multi-source log data. The method comprises the following steps: 1) preprocessing an application and a tool: generating a corresponding application program log by using related information in a target application program; meanwhile, modifying an attack plan and an attack tool of an attacker; 2) attack execution: executing the attack by using the preprocessed attack tool and the designed attacked application program, and generating an attack log at the same time; 3) data marking: constructing a provenance graph by using an audit log; positioning attack injection points of different types of logs; all system calls in the thread are divided into different processing programs, processing program IDs are recorded, and a tree structure of a processing program derivation relation is constructed; and 4) marking malicious sub-graphs. According to the method, the auditing log, the traffic log and the application program log are integrated, so that the full view of the network attack can be captured from different dimensions, and the accuracy of attack behavior recognition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an automatic labeling method combining multi-source log data. Background Art

[0002] As advanced persistent threat (APT) attacks become increasingly prominent in the field of network security, the need for effective detection, blocking and tracking technologies is becoming increasingly urgent. APT attacks are usually launched by well-funded and well-organized groups, and are known for their long-term and hidden nature. Attackers often lurk in the victim's network for months, using intermediate hosts to escalate attacks or steal sensitive data. Therefore, log analysis technology plays a vital role in the detection and tracing of APT attacks. Currently, existing automated log tagging methods usually cannot meet the following limitations:

[0003] 1) Time window-based methods: These methods mark all logs within a specific time frame as attack-related, but due to the covert nature of APTs, this granularity is too coarse, resulting in poor accuracy.

[0004] 2) Behavior profile-based methods: These methods generate labeled network traffic by simulating normal and malicious behaviors. However, they are not suitable for real traffic and cannot label audit and application logs.

[0005] 3) Detection tool-based methods: Use tools such as sniffers and honeypots to identify and mark attack behaviors. However, their effectiveness is heavily dependent on the performance of the detection tools and may not accurately capture all attack-related logs.

[0006] 4) Rule-based matching methods: Traditional methods use rule-based keyword matching, and some have been improved by combining event context. Despite the improvements, a lot of manual work is still required to create rules for each trace of APT attacks, which leads to a heavy reliance on domain expertise and makes it difficult to ensure the accuracy and completeness of the rules.

[0007] Therefore, in view of the limitations of the prior art, the present invention proposes a new automated log tagging method to address the above challenges. Summary of the invention

[0008] In view of the deficiencies of the prior art, the present invention provides an automatic annotation method combining multi-source log data.

[0009] The technical solution of the present invention is as follows:

[0010] A method for automatically labeling multi-source log data includes the following steps:

[0011] Step 1) Application and tool preprocessing steps, including: using relevant information in the target application to generate corresponding application logs and using a delimiter recorder to automatically generate delimiters; at the same time, modifying the attacker's attack plan and attack tools so that the attack mark can be injected into the attack process, so that the attack leaves traces; in the collection environment, adding an attack mark scanner;

[0012] Step 2) The steps of attack execution include: using the pre-processed attack tools and the designed attacked application to execute the attack and generate the attack log;

[0013] Step 3) Data labeling step, including: first, the provenance graph is constructed: the audit log is used to build the provenance graph, ensuring that the edges are associated with the audit log, traffic log and application log; then, the attack injection points of different types of logs are located; finally, the execution unit is divided, all system calls in the thread are divided into different handlers, the handler ID is recorded, and the tree structure of the handler derivation relationship is constructed. Then, dynamic and static analysis is used, combining the abstract syntax tree (AST) and the dynamic behavior of the code to accurately locate the activities of the handler.

[0014] Step 4) Malicious subgraph marking step, including: according to the analysis in step (3), all logs involved in the attack behavior are organized into an origin graph structure, processed in reverse topological order, and marked as malicious subgraphs.

[0015] In the further step 1), the detailed steps of application and tool preprocessing are as follows:

[0016] (1-1) Modify the application: disable the buffer mechanism and generate alignment logs; integrate the delimiter logger;

[0017] (1-2) Modify the attack tool and plan: embed a flag generation module in the attack tool to inject a recognizable attack flag at the attack operation point (the point where the attacker uses the attack tool to perform the attack steps). At the same time, the flag will also be used to generate the attack log.

[0018] (1-3) Modify the attack environment: Implement the attack flag scanner as a kernel module, that is, add a module to detect attack flags in the code.

[0019] In the further step 2), the detailed steps of attack execution are as follows:

[0020] (2-1) Launching an attack: Based on the modified attack tools and applications in step (1), the attacker launches an attack;

[0021] (2-2) Log collection: Collect the following logs: unlabeled logs; alignment information logs; separator logs: attack flags; attack logs;

[0022] In the further step 3), the detailed steps of data labeling are as follows:

[0023] (3-1) Provenance graph construction: Based on the audit log, a graph structure is constructed, where nodes represent system entities (such as processes, files, etc.) and edges represent interactions between entities.

[0024] (3-2) Log association process: This process mainly maps traffic logs and application logs to audit logs;

[0025] (3-3) Attack location process: By injecting and identifying attack signs in traffic and audit logs, combined with rule matching strategies, the attack-related anchor points are located and marked.

[0026] (3-4) Unit partitioning process: Divide the system calls of the application into different execution units and build a handler derivation relationship tree to solve the dependency explosion problem.

[0027] (3-5) Attack subgraph extraction process: Attack subgraph extraction algorithm (O(kn) complexity, k is the number of anchor points):

[0028] In the further step 4), the detailed steps of marking malicious sub-graphs are as follows:

[0029] (4-1) Subgraph alarm: Based on the attack subgraph extracted in step (3), mark the subgraph as malicious and report the graph output.

[0030] The beneficial effects of the present invention are mainly manifested in:

[0031] 1) High accuracy: By integrating audit logs, traffic logs, and application logs, it can capture the full picture of network attacks from different dimensions, improving the accuracy of identifying attack behaviors; using attack markers and label rules, combined with real-time monitoring of kernel modules, it can effectively identify attack activities.

[0032] 2) High efficiency. Through automated log association and anchor point positioning, the workload of manual analysis is reduced, greatly improving the efficiency of the traceability process. Through unit partitioning and execution unit division, this method effectively solves the dependency explosion problem in large-scale data sets and avoids the efficiency reduction caused by complex dependency relationships.

[0033] 3) Strong interpretability. Through clear log association steps, a clear source and destination are provided for each log entry, making the traceability results easy to understand and explain. By constructing origin graphs and attack subgraphs, complex network attack behaviors become intuitive and easy to understand, making it easier for analysts to quickly grasp the attack process and key nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings.

[0036] Reference Figure 1 , a flowchart of a method for automatically labeling multi-source log data, comprising the following steps:

[0037] Step (1) application and tool preprocessing step, including: using relevant information in the target application to generate a corresponding application log and using a delimiter recorder to automatically generate delimiters; at the same time, modifying the attacker's attack plan and attack tool so that the attack mark can be injected into the attack process, so that the attack leaves an attack trace; in the collection environment, adding an attack mark scanner;

[0038] (1-1) Modify the application:

[0039] 1-1-1) Disable the buffering mechanism:

[0040] a) For basic languages ​​such as C / C++: Use the dynamic link library hook technology to call the setbuf function to disable buffering after obtaining the file handle (such as after the fdopen function).

[0041] b) For high-level languages ​​such as Python: Set the buffer of the Logging module to 0.

[0042] c) For languages ​​that use a logging framework, such as Java: Modify the configuration file (such as log4j.properties) to disable buffering.

[0043] 1-1-2) Generate alignment logs: Use a kernel module, such as (using ktime_get_real_ts64), to inject the following information into each log entry:

[0044] - Accurate timestamp

[0045] -Thread ID

[0046] 1-1-3) Integrated separator recorder: divide all system calls in a thread into different handlers; record its ID at the beginning of each handler so as to know when to switch and which handler starts running; separate different execution units according to different thread IDs.

[0047] (1-2) Modify the attack tool and plan: embed a flag generation module in the attack tool to inject a recognizable attack flag at the attack operation point; at the same time, the flag will also be used to generate the attack log.

[0048] (1-3) Modify the attack environment: Implement the attack signature scanner as a kernel module. During the attack execution phase, the module scans and extracts attack signatures from traffic and audit logs, providing information that can accurately locate key events in preparation for the labeling phase. At the same time, to prevent the attack from failing or leaving irreversible traces in the log, it also removes the attack signature from the attack payload and restores the payload to its state before modification.

[0049] Step (2) of attack execution includes: using a pre-processed attack tool and a designed attacked application to execute the attack and generate an attack log;

[0050] (2-1) Launching an attack: Based on the modified attack tools and applications in step (1), the attacker launches an attack;

[0051] (2-2) Log collection: Collect the following logs:

[0052] Unlabeled logs: Original logs that need to be labeled.

[0053] Alignment information logs: used to correlate application logs and audit logs; used in the log correlation step of data tagging.

[0054] Delimiter log: used to perform cell partitioning; used in the cell partitioning step of data labeling.

[0055] Attack Flags: Key attack indicators; used in the attack location step of data labeling.

[0056] Attack log: records the attack process; used in the attack location step of data tagging; contains the timestamp range of each attack step; contains log records that can be used for rule matching.

[0057] Step (3) data labeling step, including:

[0058] The first is provenance graph construction: Use audit logs to build a provenance graph and ensure that edges are associated with audit logs, traffic logs, and application logs;

[0059] Next, locate the attack injection points of different types of logs;

[0060] Finally, execution unit partitioning divides all system calls in the thread into different handlers, records the handler IDs, and builds a tree structure of handler derivation relationships. Subsequently, dynamic and static analysis is used to combine the abstract syntax tree (AST) and the dynamic behavior of the code to accurately locate the activities of the handler.

[0061] (3-1) Provenance graph construction: Based on the audit log, a graph structure is constructed, where nodes represent system entities (such as processes, files, etc.) and edges represent interactions between entities.

[0062] (3-2) Log association process: This process mainly maps traffic logs and application logs to audit logs;

[0063] (3-2-1) For application logs:

[0064] a. Use the injected alignment information log (including timestamp and thread ID);

[0065] b. Sort the audit logs by thread ID;

[0066] c. Use the timestamp to match the application log to the closest system call.

[0067] (3-2-2) For traffic logs:

[0068] a. Goal: Match each TCP connection and UDP session with the corresponding network-related system calls in the audit log;

[0069] b. Using the four-tuple information in the audit log (source IP address, source port, destination IP address, destination port)

[0070] Perform one-to-one matching;

[0071] c. Matching process:

[0072] i. Match the closest system call: Identify the network system call with the closest timestamp and quad-tuple;

[0073] ii. Locate the life cycle of traffic file descriptors: Track the creation and destruction of file descriptors used by system calls in the process audit log and establish matching boundaries;

[0074] iii. Batch matching within the interval: All system calls with the same quadruple within the boundary are considered to match the traffic log entry.

[0075] (3-2-3) Implement a two-step scanning algorithm with O(n) time complexity:

[0076] a. Step 1: Scan the audit log

[0077] i. Track the life cycle of each network file descriptor (fd) related to the network system call;

[0078] ii. Record the creation and destruction time of each fd.

[0079] b. Step 2: Scan traffic logs

[0080] i. For each traffic log entry, find the entry e in the audit log that has the closest timestamp and a matching quadruple;

[0081] ii. Locate all relevant audit log entries with the same four-tuple within the life cycle of the fd corresponding to e;

[0082] iii. Associate these audit log entries with the current traffic log entries.

[0083] (3-3) Attack positioning process:

[0084] (3-3-1) Define anchor point:

[0085] Anchors are defined as log entries representing attack activities, corresponding to edges in the provenance graph. Anchors can be divided into traffic logs and audit logs according to their types;

[0086] (3-3-2) Identify anchor points in traffic logs: First, the scanner records all IP addresses and their reverse addresses in the scenario; once the scanner detects a reverse IP address, it leaves an attack mark and then sets the reserved bit of the IP header to 1. Connections with the reserved bit of the IP header set to 1 are marked as anchor points;

[0087] (3-3-3) Identify anchor points in the audit log: First, identify the injection point: file name, file content (file read and write operations), and command parameters (command execution). Integrate the kernel module into the system call through step (1) and inject the attack flag into the attack payload. Intercept the system call during the log collection in step 2 and restore it to the pre-attack state before the payload is executed. Finally, by capturing the system call parameters with the attack flag, it is easy to locate the attack-related logs.

[0088] (3-3-4) Rule-based log location strategy: If the key anchor point cannot be identified by the above method, the rule matching strategy proposed by Kyoushi is adopted; the attacker can selectively record actions in the attack log, including:

[0089] a. Timestamps of when the attack started and ended

[0090] b. Attack parameters

[0091] c. Specific matching rules

[0092] Integrate attack logs to locate anchor points and make up for the shortcomings of previous methods.

[0093] (3-4) Unit partitioning process:

[0094] (3-4-1) Execution unit definition: An execution unit contains all the operations performed by the application when processing a single request.

[0095] (3-4-2) Partitioning step: a. Divide all system calls in the thread into different handlers. Record the ID of each handler at the beginning. b. Build a tree structure (handler tree) of handler derivation relationships. Record the ID of the new handler when a new handler is created.

[0096] (3-5) Attack subgraph extraction process: Attack subgraph extraction algorithm (O(kn) complexity, k is the number of anchor points):

[0097] a. Define the "starting point" node of the edges corresponding to all anchor points.

[0098] b. Start from the node with no outgoing edges and process in reverse topological order.

[0099] c. Maintain a set Si for each node i, containing all starting points reachable from i via a unique path.

[0100] d. For each new node t, ensure that the path from t remains unique by taking the union of the sets of all direct successor nodes and subtracting any intersections.

[0101] e. If i is the starting point, mark all paths starting from i and adjust Si to only include i.

[0102] Step (4) malicious subgraph marking step, including: according to the analysis in step (3), all logs involved in the attack behavior are organized into an origin graph structure, processed in reverse topological order, and marked as malicious subgraphs.

[0103] (4-1) Subgraph alarm: Based on the attack subgraph extracted in step (3), mark the subgraph as malicious and report the graph output.

Claims

1. A method for automatically labeling multi-source log data, characterized in that: The steps include: 1) Application and tool preprocessing: Use relevant information in the target application to generate the corresponding application log, and then use the delimiter recorder to automatically generate delimiters; at the same time, modify the attacker's attack plan and attack tools to inject attack markers into the attack process, leaving attack traces; And add an attack sign scanner in the collection environment; 2) Attack execution: Use the pre-processed attack tools and the designed attacked applications to execute the attack and generate attack logs; 3) Data labeling: Use audit logs to build a provenance graph and ensure that edges are associated with audit logs, traffic logs, and application logs; locate attack injection points for different types of logs; divide all system calls in a thread into different handlers, record handler IDs, build a tree structure of handler derivation relationships, and use dynamic and static analysis, combined with the abstract syntax tree AST and code dynamic behavior, to locate handler activities; 4) Malicious subgraph marking: All logs involved in attack behaviors are organized into a provenance graph structure, processed in reverse topological order, and marked as malicious subgraphs.

2. The method for automatically labeling multi-source log data according to claim 1, characterized in that: The process of step 1) is as follows: 1-1) Modify the application: disable the buffering mechanism and generate aligned logs; also integrate the delimiter logger; 1-2) Modify the attack tool and plan: embed a flag generation module in the attack tool to inject identifiable attack flags at the attack operation point and generate attack logs; 1-3) Modify the attack environment: Implement the attack signature scanner as a kernel module.

3. The method for automatically labeling multi-source log data according to claim 2, characterized in that: The process of step 2) is as follows: 2-1) Launching an attack: Based on the modified attack tools and applications in step 1), the attacker launches an attack; 2-2) Log collection: Collect the following logs including unmarked logs, alignment information logs, separator logs, attack flags, and attack logs.

4. The method for automatically labeling multi-source log data according to claim 1, characterized in that: The process of step 3) is as follows: 3-1) Provenance graph construction: Based on the audit log, a graph structure is constructed, where nodes represent system entities and edges represent interactions between entities; 3-2) Log association process: Map traffic logs and application logs to audit logs; 3-3) Attack location process: By injecting and identifying attack signs in traffic logs and audit logs, combined with rule matching strategies, the attack-related anchor points are located and marked; 3-4) Unit partitioning process: divide the system calls of the application into different execution units and build a handler derivation relationship tree; 3-5) Attack subgraph extraction process: The attack subgraph extraction algorithm extracts the attack subgraph.

5. The method for automatically labeling multi-source log data according to claim 1, characterized in that: The process of step 3-2) is as follows: 3-2-1) For application logs: a. Use injected alignment information log, including timestamp and thread ID; b. Sort the audit logs by thread ID; c. Use timestamps to match application logs to the closest system call; 3-2-2) For traffic logs: a. Match each Transmission Control Protocol (TCP) connection and User Datagram Protocol (UDP) session with the corresponding network-related system call in the audit log; b. Use the four-tuple information {source IP address, source port, destination IP address, destination port} in the audit log for one-to-one matching; c. Matching process: i. Match the closest system call: Identify the network system call with the closest timestamp and quad-tuple; ii. Locate the life cycle of traffic file descriptors: Track the creation and destruction of file descriptors used by system calls in the process audit log and establish matching boundaries; iii. Batch matching within the interval: All system calls with the same quadruple within the boundary are considered to match the traffic log entry; 3-2-3) Two-step scanning algorithm with time complexity O(n): a. Step 1: Scan the audit log: i. Track the life cycle of each network file descriptor fd related to the network system call; ii. Record the creation and destruction time of each fd; b. Step 2: Scan traffic logs: i. For each traffic log entry, find the entry e in the audit log that has the closest timestamp and a matching quadruple; ii. Locate all relevant audit log entries with the same four-tuple within the life cycle of the fd corresponding to e; iii. Associate these audit log entries with the current traffic log entries.

6. The method for automatically labeling multi-source log data according to claim 1, characterized in that: The process of step 3-3) is as follows: 3-3-1) Define anchor points: Anchor points are defined as log entries representing attack activities, corresponding to the edges in the provenance graph. Anchor points are divided into traffic logs and audit logs according to their types. 3-3-2) Identify anchor points in traffic logs: The scanner records all IP addresses and their reverse addresses in the scenario; sets the reserved bit of the IP header to 1, and the connection with the reserved bit of the IP header set to 1 is marked as an anchor point; 3-3-3) Identify anchor points in audit logs: identify injection points including file name, file content, and command parameters; integrate kernel modules into system calls through step 1) and inject attack flags into attack payloads; intercept system calls during log collection in step 2) and restore them to the pre-attack state before payload execution; locate attack-related logs by capturing system call parameters with attack flags.

7. The method for automatically labeling multi-source log data according to claim 1, characterized in that: The process of steps 3-4) is as follows: 3-4-1) Execution unit definition: An execution unit contains all the operations performed by the application when processing a single request; 3-4-2) Partition: a. Divide all system calls in a thread into different handlers and record their IDs at the start of each handler; b. Build a tree structure of handler derivation relationships and record the ID of a new handler when it is created.

8. The method for automatically labeling multi-source log data according to claim 1, characterized in that: The process of steps 3-5) is as follows: a. Define the starting point nodes of the edges corresponding to all anchor points; b. Start from the node without outgoing edges and process in reverse topological order; c. Maintain a set Si for each node i, containing all the starting points reachable from i through a unique path; d. For each new node t, ensure that the path from t remains unique by taking the union of the sets of all direct successor nodes and subtracting any intersections; e. If i is the starting point, mark all paths starting from i and adjust Si to only contain i.

Images