Multi-source log fusion method and device for advanced persistent attack scene
Through the multi-source log fusion method, application logs, traffic logs and system audit logs are collected and integrated, which solves the problem that the existing technology is difficult to detect and track APT attacks, and effectively detect and trace APT attacks are realized.
Patent Information
- Application Number
- CN202411410911.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-02-25
AI Technical Summary
It is difficult for the prior art to effectively detect and detect advanced persistent threat (APT) attacks, especially when the attack steps are small, long time and large log volume, it is difficult to restore the attacker's attack path.
A multi-source log fusion method is proposed, which can detect and track APT attacks by collecting application logs, traffic logs and system audit logs, and fuse them based on thread identifiers and log acquisition moments.
It realizes the detection of attack events through multi-faceted log data, and can detect APT attacks and restore the attack path, which helps users handle APT attacks and achieves a comprehensive source tracing attack.
Smart Images
Figure CN119513861A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of network security technology, and in particular, relates to a multi-source log fusion method and device for advanced persistent attack scenarios. Background Art
[0002] In recent years, network attacks have become more and more prevalent, and the means of attack have become more and more complex and professional. For example, advanced persistent threats (APT) have posed a huge threat to network security. In the relevant technology, there are intrusion detection methods based on log analysis, but because each step of APT attacks is small, the attack time is long, and the amount of logs generated by the system is large, it is impossible to detect through multiple log data, making it difficult to detect APT attacks and restore the attacker's attack path. Summary of the invention
[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a multi-source log fusion method and device for advanced persistent attack scenarios, which realizes the detection of attack events through multi-faceted log data, especially APT attacks, and can restore the attacker's attack path based on malicious logs, thereby helping users to handle APT attacks according to the attack path and realize comprehensive attack tracing.
[0004] In a first aspect, the present application provides a multi-source log fusion method for advanced persistent attack scenarios, the method comprising:
[0005] Collecting application logs corresponding to target applications in the electronic device, traffic logs corresponding to target interfaces in the electronic device, and system audit logs corresponding to the electronic device;
[0006] Based on the thread identifier corresponding to the application log and the collection time corresponding to the first log data of each line in the application log, the application log is merged into the audit log, and the traffic log is merged into the audit log to obtain a merged audit log;
[0007] Based on the fused audit log, a malicious log corresponding to the target application is obtained; the malicious log is used to process events corresponding to the advanced persistent attack.
[0008] According to the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiments of the present application, by fusing the collected application logs, traffic logs and audit logs to obtain a fused audit log, and then obtaining the malicious log corresponding to the target application, a more comprehensive semantic context can be established, and attack events can be detected through various log data, especially APT attacks can be detected, and the attacker's attack path can be restored based on the malicious log, thereby helping users to handle APT attacks according to the attack path and achieving comprehensive attack tracing.
[0009] A multi-source log fusion method for an advanced persistent attack scenario in one embodiment of the present application obtains a thread identifier corresponding to the application log and a collection time corresponding to the first log data of each line in the application log, including:
[0010] Inserting a kernel module input by a user into the target application;
[0011] Based on the kernel module, inserting a thread identifier corresponding to a thread for executing the target application into a corresponding position of the application log;
[0012] Based on the kernel module, a collection time corresponding to each line of the first log data is obtained.
[0013] In one embodiment of the present application, a multi-source log fusion method for an advanced persistent attack scenario is provided, wherein based on a thread identifier corresponding to the application log and a collection time corresponding to the first log data of each row in the application log, the application log is fused into the audit log to obtain a fused audit log, including:
[0014] Acquire, from the audit log, second log data having the same thread identifier as that corresponding to the application log and having a time difference between the collection time corresponding to the first log data and the collection time corresponding to the first log data being less than a target threshold;
[0015] The matched groups of first log data and second log data are merged to obtain the merged audit log.
[0016] In one embodiment of the present application, a multi-source log fusion method for an advanced persistent attack scenario is provided, wherein the flow log is fused into the audit log to obtain the fused audit log, including:
[0017] Acquire fourth log data having the same IP address and target port as the third log data included in the traffic log from the audit log;
[0018] The matched groups of third log data and fourth log data are merged to obtain the merged audit log.
[0019] In one embodiment of the present application, a multi-source log fusion method for an advanced persistent attack scenario is provided, wherein the malicious log corresponding to the target application is obtained based on the fused audit log, including:
[0020] Based on the fused audit log, updating the directed traceability graph corresponding to the audit log, each edge of the updated directed traceability graph corresponds to a plurality of fused log entries;
[0021] Based on the updated directed tracing graph, the malicious log is obtained.
[0022] In one embodiment of the present application, a multi-source log fusion method for an advanced persistent attack scenario is provided, wherein the malicious log is obtained based on the updated directed tracing graph, including:
[0023] Obtain all attack entries in the updated directed traceability graph;
[0024] Connecting a target attack entry among all the attack entries and other attack entries among all the attack entries except the target attack entry to obtain multiple attack paths;
[0025] The malicious log is obtained based on the fused log entries corresponding to the edges included in the attack path.
[0026] In an embodiment of the present application, a multi-source log fusion method for an advanced persistent attack scenario, before collecting application logs corresponding to a target application in an electronic device, traffic logs corresponding to a target interface in the electronic device, and system audit logs corresponding to the electronic device, the method further includes:
[0027] Control the buffer corresponding to the target application to close.
[0028] In a second aspect, the present application provides a multi-source log fusion device for advanced persistent attack scenarios, including:
[0029] A first processing module, used to collect application logs corresponding to target applications in the electronic device, traffic logs corresponding to target interfaces in the electronic device, and system audit logs corresponding to the electronic device;
[0030] A second processing module is used to merge the application log into the audit log based on the thread identifier corresponding to the application log and the collection time corresponding to each line of the first log data in the application log, and merge the traffic log into the audit log to obtain a merged audit log;
[0031] The third processing module is used to obtain a malicious log corresponding to the target application based on the fused audit log; the malicious log is used to process events corresponding to the advanced persistent attack.
[0032] According to the multi-source log fusion device for advanced persistent attack scenarios provided by the embodiment of the present application, by fusing the collected application logs, traffic logs and audit logs to obtain a fused audit log, and then obtaining the malicious log corresponding to the target application, a more comprehensive semantic context can be established, and attack events can be detected through various log data, especially APT attacks can be detected, and the attacker's attack path can be restored based on the malicious log, thereby helping users to handle APT attacks according to the attack path and achieving comprehensive attack tracing.
[0033] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the multi-source log fusion method for advanced persistent attack scenarios as described in the first aspect above is implemented.
[0034] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-source log fusion method for advanced persistent attack scenarios as described in the first aspect above.
[0035] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-source log fusion method for advanced persistent attack scenarios as described in the first aspect above.
[0036] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
[0037] By fusing the collected application logs, traffic logs, and audit logs to obtain the fused audit logs, and then obtaining the malicious logs corresponding to the target application, a more comprehensive semantic context can be established, and attack events can be detected through multi-faceted log data. In particular, APT attacks can be detected, and the attacker's attack path can be restored based on the malicious logs, which helps users handle APT attacks based on the attack path and achieves comprehensive attack tracing.
[0038] Furthermore, by inserting the kernel module, the time of system call execution and the identifier of the thread that executes the system call can be inserted into the log data corresponding to each line of the application log, thereby achieving accurate matching of the application log and the audit log, thereby improving the accuracy and precision of the matching.
[0039] Furthermore, by closing the buffer corresponding to the target application before collecting log data, a line of output in the application log can correspond one-to-one with the underlying file write system call, thereby matching the application log with the audit log.
[0040] Furthermore, by fusing logs from multiple data sources, the overall traceability information of the traceability graph can be enriched, so that the attack subgraph can be automatically obtained based on the traceability graph, and malicious logs related to the attack can be extracted based on the correspondence between the edges in the traceability graph and the log entries, so that attack events can be detected and processed based on the fused logs.
[0041] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0043] Figure 1 This is one of the flow charts of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0044] Figure 2 This is one of the principle schematic diagrams of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0045] Figure 3 This is the second principle schematic diagram of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0046] Figure 4 This is the third principle schematic diagram of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0047] Figure 5 This is the fourth principle schematic diagram of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0048] Figure 6 This is the fifth principle schematic diagram of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0049] Figure 7 This is the sixth principle schematic diagram of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0050] Figure 8This is the seventh principle schematic diagram of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0051] Fig. 9 This is the eighth principle schematic diagram of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0052] Fig.10 This is one of the test result schematic diagrams of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0053] Fig.11 This is the second schematic diagram of the test results of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0054] Fig.12 This is the second flow chart of the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application;
[0055] Fig.13 It is a structural schematic diagram of a multi-source log fusion device for advanced persistent attack scenarios provided by an embodiment of the present application;
[0056] Fig.14 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0058] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0059] In conjunction with the accompanying drawings, the multi-source log fusion method for advanced persistent attack scenarios, the multi-source log fusion device for advanced persistent attack scenarios, the electronic device and the readable storage medium provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios.
[0060] Among them, the multi-source log fusion method for advanced persistent attack scenarios can be applied to terminals, and can be specifically executed by hardware or software in the terminal.
[0061] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).
[0062] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.
[0063] The multi-source log fusion method for advanced persistent attack scenarios provided in the embodiments of the present application may be an electronic device or a functional module or functional entity in the electronic device that can implement the multi-source log fusion method for advanced persistent attack scenarios. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The multi-source log fusion method for advanced persistent attack scenarios provided in the embodiments of the present application is described below using the electronic device as an example of the execution subject.
[0064] like Figure 1 As shown, in some embodiments, a scenario of attacking the system can be simulated, and the fusion process of the data of the three log types can be simulated in the attack scenario.
[0065] In this embodiment, malicious items in the log can be matched by rules, and additional information can be obtained, such as specific attack steps and malicious IP addresses, etc., and then the additional information can be used to assist the traffic log and application log to complete the unified mapping on the audit log.
[0066] Containerization technologies such as Docker can be used on a host to deploy network attacks involving multi-hop hosts into different containers on the same host. Each container encapsulates an application required during the attack.
[0067] First, the application services used in the attack process can be abstracted to form a topology diagram, and the corresponding Docker image can be created to define the required software and configuration.
[0068] You can then use docker-compose or kubernetes to orchestrate multiple containers, set up network configurations to simulate a real environment, start the containers and run the attack script to complete the attack scenario.
[0069] By modifying the experimental environment and applications, auxiliary information required for automatic annotation can be generated, and the user's input can be received to modify the attack tool and inject key features that can be used for matching, thereby obtaining auxiliary information for log fusion.
[0070] like Figure 2 As shown, the user can enter the attack steps written in JSON data format.
[0071] The modified attack tool can automatically record information during the attack process as the attack experiment proceeds, such as the timestamp of each step, the file name accessed, and the command executed.
[0072] Attack scenarios can then be run and multiple logs from multiple sources can be collected.
[0073] Use the alignment log information generated during the environment transformation phase to map traffic logs and application logs to audit logs.
[0074] Locate the attack entry point information and map it to the audit log traceability graph. By dividing the subgraph and unit, the attack information and chain can be obtained.
[0075] like Fig.12 As shown, the multi-source log fusion method for advanced persistent attack scenarios includes: step S1, step S2 and step S3.
[0076] Step S1, collecting application logs corresponding to target applications in electronic devices, traffic logs corresponding to target interfaces in electronic devices, and system audit logs corresponding to electronic devices;
[0077] In this step, electronic devices include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices.
[0078] The electronic device is the attacked device.
[0079] The target application is the application being attacked.
[0080] Application logs are log information generated by the target application during its operation. Application logs include various activities, events, and status of the target application.
[0081] Application logs can be recorded in specific log files, such as files in the " / var / log" directory.
[0082] Log files may include "syslog" or "message", and specific application logs may have their own independent files. Examples of application logs are: Figure 4 shown.
[0083] Traffic logs are used to record the details of all normal network traffic passing through network devices (such as firewalls, routers or switches, etc.), including but not limited to source and destination IP addresses, port numbers, transmission protocols (such as TCP or UDP, etc.) and traffic size.
[0084] The Tcpdump tool can be used to complete the collection of traffic logs. Tcpdump is an open source tool for network packet capture, which is used to capture and display data packets passing through the network interface; or Wireshark, Logstash or Filebeat can be used to collect traffic logs. The choice can be based on user needs and is not limited in this application.
[0085] Tcpdump can capture and display network packets in real time in the command line, allowing users to select a specific network interface (network card) for packet capture. By specifying specific network interface parameters in the command line, users can ensure that Tcpdump only captures network traffic on the specified interface (target interface) rather than all traffic on the entire system.
[0086] Regarding the reserved bits of the traffic pcap packet, Tcpdump also allows users to manipulate the flag bits of the data packet during the capture process to meet subsequent specific requirements for annotation.
[0087] System audit logs are used to record key activities and events of computer systems. These records contain detailed information on various operations such as user login, file access, process access, and system calls.
[0088] Audit log collection can be completed using Sysdig, Graylog, Splunk or other tools.
[0089] Sysdig is an open source system and container tracing tool that provides deep, real-time analysis of operating system and application layer activity.
[0090] Sysdig can not only capture and display system calls and events, but also support powerful filtering and query functions, allowing users to gain in-depth understanding of the system's operating status, monitor performance indicators, and perform fault diagnosis. Examples of audit logs captured by Sysdig are as follows: Figure 3 shown.
[0091] Step S2: Based on the thread identifier corresponding to the application log and the collection time corresponding to the first log data in each line of the application log, the application log is merged into the audit log, and the traffic log is merged into the audit log to obtain a merged audit log;
[0092] In this step, when content needs to be written to the target application, a file writing-related system call needs to be intercepted, and the TID (Thread Identifier) of the thread executing the system call is the thread identifier corresponding to the application log.
[0093] The application log includes multiple lines of first log data.
[0094] The collection time is the time when the system call corresponding to the first log data is executed (that is, the time when the content of the line is written).
[0095] Based on the thread identifier corresponding to the application log and the collection time corresponding to the first log data of each line in the application log, the application log and the audit log can be matched to merge the application log into the audit log.
[0096] Both the traffic log and the audit log contain timestamps. Based on the timestamps corresponding to the data in the traffic log and the audit log, the traffic log can be integrated into the audit log.
[0097] The merged audit log contains information of application log and traffic log.
[0098] In some embodiments, step S2 may include:
[0099] The application log and the audit log are parsed and processed respectively to obtain the first field information corresponding to the application log and the second field information corresponding to the audit log;
[0100] Based on the first field information and the second field information, the application log is integrated into the audit log;
[0101] Parse the traffic log to obtain the third field information corresponding to the traffic log;
[0102] Based on the third field information and the second field information, the traffic log is merged into the audit log to obtain a merged audit log.
[0103] In this embodiment, each line of the audit log has the same format and contains multiple useful fields. When parsing the audit log, it is necessary to parse the format of each line of text, divide the various fields by spaces or other symbols to obtain the second field information, thereby converting all audit logs into a typical array similar to a character.
[0104] For example, this line of audit log "167095 2023-09-05 22:03:18.228532182 1apt(11899)>write res=26data=(Reading database…10%" contains field information such as log sequence number 167095, timestamp 2023-09-0522:03:18.228532182, process name apt, TID=11899, system call name write, and system call content (i.e., write content).
[0105] Parsing audit logs requires extracting useful fields to facilitate subsequent matching with other types of logs.
[0106] The pcap packet traffic log file obtained using Tcpdump can also be parsed by a tool or script to extract the first field information.
[0107] Wireshark is an open source network packet analysis software that can obtain and analyze data packets on the network.
[0108] In Wireshark, users can view detailed information of a data packet, including various fields such as source address, destination address, protocol and port.
[0109] Tshark is the command-line version of Wireshark, which can be used to extract information from pcap files in the terminal and can also assist users in writing scripts for automated parsing.
[0110] For the analysis of traffic logs, important information fields in the pcap package can be extracted, such as source IP, source port, destination IP and destination port, and timestamp. Based on the third field information corresponding to the traffic log, a traffic log can be matched with the information recorded in the audit log through data.
[0111] Step S3: Based on the fused audit log, obtain the malicious log corresponding to the target application.
[0112] In this step, malicious logs are used to process events corresponding to advanced persistent attacks.
[0113] For example, you can trace the source of an attack based on malicious logs to find all behaviors related to the attack, thereby detecting and handling the attack.
[0114] The fused audit log integrates data from multiple sources and can provide more comprehensive attack chain information.
[0115] In this application, application logs, traffic logs and audit logs are fused and processed to achieve the total annotation of logs from multiple data sources.
[0116] According to the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiments of the present application, by fusing the collected application logs, traffic logs and audit logs to obtain a fused audit log, and then obtaining the malicious log corresponding to the target application, a more comprehensive semantic context can be established, and attack events can be detected through various log data, especially APT attacks can be detected, and the attacker's attack path can be restored based on the malicious log, thereby helping users to handle APT attacks according to the attack path and achieving comprehensive attack tracing.
[0117] In some embodiments, obtaining a thread identifier corresponding to an application log and a collection time corresponding to each line of first log data in the application log may include:
[0118] Insert user-entered kernel modules into target applications;
[0119] Based on the kernel module, inserting a thread identifier corresponding to a thread for executing the target application into a corresponding position of the application log;
[0120] The collection time corresponding to the first log data in each line is obtained based on the kernel module.
[0121] In this embodiment, during the experiment, the experimental environment and the applications in the environment may be modified to generate various auxiliary information required for automatic fusion based on the experimental environment.
[0122] For a single thread, the system calls it executes will occur serially. Based on the execution time of the system call corresponding to each line of the application log and the identifier of the thread that executes the system call, the application log and the audit log can be matched.
[0123] You can insert kernel modules to instrument file writing related system calls.
[0124] By pre-configuring the directory where the application log is located, it is possible to detect in the kernel module whether the written file is located in the directory, and thus determine that this is a file write operation related to the application log.
[0125] During the attack execution phase, for each write operation, the kernel module inserts the TID of the current thread in the header of the written content, as well as a precise timestamp with nanosecond accuracy obtained using a kernel function (e.g., ktime_get_real_ts64 function).
[0126] In some embodiments, based on the thread identifier corresponding to the application log and the collection time corresponding to the first log data of each line in the application log, the application log is merged into the audit log to obtain the merged audit log, which may include:
[0127] Acquire, from the audit log, second log data having the same thread identifier as that corresponding to the application log and having a time difference between the collection time corresponding to the first log data and the collection time corresponding to the first log data less than a target threshold;
[0128] The matched groups of first log data and second log data are fused to obtain a fused audit log.
[0129] In this embodiment, the thread identifier corresponding to each piece of log data in the audit log may be different, and the collection time corresponding to each piece of log data is also different.
[0130] Second log data having the same thread identifier as that corresponding to the application log and having the same or close collection time as that corresponding to the first log data may be obtained.
[0131] For each piece of first log data, second log data matching therewith can be obtained.
[0132] The matching groups of first log data and second log data may be fused to obtain a fused audit log.
[0133] For example, in the fusion matching step, Figure 8 As shown, all audit logs can be decomposed according to TID, and the application logs can be traversed line by line. At the same time, for each TID, a pointer can be maintained to point to the system call of the application log that is closest to the current traversed in the audit log related to the TID.
[0134] The sum of the movement amounts of all pointers is less than or equal to the total audit log amount, so the total time complexity of Log Grafting is O(n).
[0135] According to the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application, by inserting a kernel module, the time of system call execution and the identifier of the thread that executes the system call can be inserted into the log data corresponding to each line of the application log, thereby achieving accurate matching of the application log and the audit log, thereby improving the accuracy and precision of the matching.
[0136] In some embodiments, before step S1, the multi-source log fusion method for advanced persistent attack scenarios may further include:
[0137] Controls the closing of the buffer corresponding to the target application.
[0138] In this embodiment, in order to ensure a one-to-one correspondence between the API call for writing application logs (generating a line of application logs) and the underlying file writing system call, the buffer of the application needs to be closed.
[0139] The inventor discovered during the research and development process that when a program prints a log, it will write a code statement for printing the log, and pass the log string as a parameter to the log printing function provided by the language or framework, such as the printf statement in the C language, the logging.info statement in the logging library provided by Python, or the logger.info in the Java Log4j library.
[0140] When implementing log printing functions, most of them have a Buffer mechanism, that is, the output data will be accumulated in the Buffer. When the Buffer length reaches the threshold or encounters a line break, the output in the Buffer will be output to the device in a centralized manner.
[0141] Under the buffer mechanism, such as Figure 7 As shown in the figure, by calling the Write-related system calls, the data is output to the file, a new line of content is added to the application log, and a file write audit log is generated.
[0142] The implementation of the buffer mechanism varies depending on the language and framework. For example, for programs developed in basic languages such as C / C++, output functions such as printf are implemented by the standard library; for log libraries provided by high-level languages such as Python's Logging library or Java's Log4j library, they are often implemented internally by the high-level language.
[0143] like Figure 6 As shown in the figure, when the Buffer mechanism exists, a line of application log output cannot correspond one-to-one with the underlying file write system call. The buffer can be closed based on the following steps:
[0144] For programs developed in low-level languages such as C / C++, the buffer can be automatically closed by hooking the dynamic link library. Taking the glibc standard library implementation in C language as an example, functions such as fdopen, fopen, and fopen64 that obtain file handles can be hooked. Before the function returns, the setbuf function provided by the standard library is actively called to close the buffer of the file handle.
[0145] For high-level language logging libraries, you can perform global buffer configuration to turn off the buffer. For example, for Python's Logging module, you can use Monkey-Patch to ensure that the buffering of FileHandler is set to 0. For Java's Log4j logging method, you can disable the buffer by changing the configuration items in the log4j.properties file.
[0146] According to the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application, by closing the buffer corresponding to the target application before collecting log data, a one-to-one correspondence between a line of application log output and the underlying file write system call can be achieved, thereby achieving matching of application logs and audit logs.
[0147] In some embodiments, the traffic log is integrated into the audit log to obtain the integrated audit log, which may include:
[0148] Acquire, from the audit log, fourth log data having the same IP address and target port as the third log data included in the traffic log;
[0149] The matched groups of third log data and fourth log data are merged to obtain a merged audit log.
[0150] In this embodiment, when the port numbers are not repeated, the four-tuple consisting of the IP addresses and port numbers of both parties of the network traffic can correspond one-to-one to a TCP connection / UDP session.
[0151] A plurality of sets of matching third log data and fourth log data may be obtained, and then each set of the third log data and the fourth log data may be fused to obtain a fused audit log.
[0152] In the actual execution process, the audit log includes information about network system calls, and the audit log and traffic log can be matched in the Labeling Phase of Autolabel.
[0153] like Fig. 9 As shown, for each object that needs to be matched in the traffic log, the traffic-related system call with the closest timestamp and the same quadruple can be obtained.
[0154] For the network file descriptor used by the system call, by searching forward and backward in the audit log generated by the process, the location where the file descriptor was created and destroyed can be found, and the boundaries that match the front and back can be obtained.
[0155] Within the boundary range, all system calls with the same quadruple are matched with the traffic log.
[0156] In the process of traffic log grafting, the system calls involved are shown in Table 1;
[0157] Table 1
[0158]
[0159] In the fusion matching step, the audit log can be scanned first, and for each network system call, the creation and destruction life cycle of the fd can be maintained.
[0160] Then scan the traffic logs and maintain the audit log with the same four-tuple as the current one. For this audit log, find the audit log with the same four-tuple within its FD life cycle. These audit logs and traffic logs can be integrated.
[0161] In this application, the grafting of traffic logs to audit logs can be achieved within the time complexity of O(n).
[0162] In some embodiments, obtaining malicious logs corresponding to the target application based on the fused audit logs may include:
[0163] Based on the fused audit logs, update the directed traceability graph corresponding to the audit logs;
[0164] Based on the updated directed tracing graph, malicious logs are obtained.
[0165] In this embodiment, a traceability graph can be constructed based on the audit log.
[0166] The traceability graph is a directed graph used to represent the relationship between subjects (such as processes or threads) and objects (such as files or network sockets) in the system. Figure 5 As shown, the direction of the edge in the traceability graph represents the direction of data flow, and each edge in the traceability graph corresponds to several specific audit log entries.
[0167] Each edge of the updated directed traceability graph corresponds to multiple fused log entries, that is, after log fusion is completed, the traceability graph will contain the underlying low-level semantics (the underlying fine-grained audit information) and the upper-level high-level behavioral semantics (information such as traffic and application).
[0168] In some embodiments, obtaining malicious logs based on the updated directed tracing graph may include:
[0169] Obtain all attack entries in the updated directed traceability graph;
[0170] Connecting a target attack entry among all attack entries with other attack entries except the target attack entry among all attack entries to obtain multiple attack paths;
[0171] Based on the fused log entries corresponding to the edges included in the attack path, malicious logs are obtained.
[0172] In this embodiment, multiple malicious behavior entries can be located from traffic logs, application logs or attack scripts as entry points for tracing.
[0173] The algorithm executed by the attack subgraph can be used to mark all log entries related to the attack on the traceability graph.
[0174] For example, the paths between multiple tracing entry points can be connected to form an attack subgraph containing all relevant logs.
[0175] When there is a unique path between two entry points, all edges of the path can be marked.
[0176] In the actual execution process, the starting nodes of the edges corresponding to all malicious entry points can be defined as "starting points".
[0177] You can start from the node with no outgoing edges and process each node one by one in reverse topological order.
[0178] For each node i, a set S[i] can be maintained, which contains all the starting points that can be reached from i through a unique path.
[0179] When a new node i is processed, S[i] can be updated to the union of all its direct successor node sets, and then the intersection of these sets can be eliminated.
[0180] When node i is the starting point, all paths starting from i can be marked and the S[i] set can be updated, keeping only i itself.
[0181] After the above path marking is completed, if there are still disconnected tracing entry points, the locations of the disconnected entry points can be displayed, and the annotator can be prompted to add more anchor points. For example, rules can be written to improve the construction of the attack subgraph.
[0182] According to the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application, by fusing logs from multiple data sources, the overall traceability information of the traceability graph can be enriched, so that the attack subgraph can be automatically obtained according to the traceability graph, and malicious logs related to the attack can be extracted based on the correspondence between the edges in the traceability graph and the log entries, so that the attack events can be detected and processed based on the fused logs.
[0183] The inventor has conducted multiple tests and verifications on the multi-source log fusion method for advanced persistent attack scenarios provided by the embodiment of the present application in combination with actual application scenarios. The test steps and test results are as follows:
[0184] As shown in Table 2, the accuracy of log fusion in the above method can be tested using 6 different server applications covering common categories:
[0185] Table 2
[0186]
[0187] These applications are accessed extensively to generate traffic logs, application logs, and audit logs.
[0188] For Nginx, Apache, FastAPI App, and Express.js App, use Siege for stress testing. For Redis and MySQL Server, native benchmarking tools are provided to test the applications.
[0189] Test the integration effect of application logs and audit logs based on the following steps:
[0190] First, verify that the above method can correctly map application logs to audit logs.
[0191] Confirm whether there is a one-to-one correspondence: Confirm whether there is an Alignment InfoLog for each line in the application log to ensure that multiple log items are not mixed together through the same buffer.
[0192] Confirm the correctness of the match: The write system call of sysdig will display the prefixes of the written content. Check whether these prefixes match the application log items.
[0193] The log buffering mechanisms of these six applications are as follows:
[0194] Nginx and Python Logging library implement custom log buffering mechanisms; they both provide ways to configure the buffer, and the buffer is turned off by default.
[0195] Apache, Morgan (Node.js), Redis, and MySQL Server do not implement a log buffering mechanism (excluding the fact that MySQL Server has a binlog buffer), and use the buffering mechanism provided by the C standard library.
[0196] The experimental results are shown in Tables 3 and 4. For all applications, the closing of the buffer is in line with expectations, and for Nginx, FastAPI App, Express.js App, Redis, and Mysql Server, one-to-one corresponding audit log entries can be found.
[0197] For Apache, when the concurrency does not exceed 50, one-to-one corresponding audit logs can be found; when using the audit log recording framework [eAudit] without data loss, the log loss of sysdig audit logs can be effectively avoided, so that one-to-one corresponding audit logs can be found.
[0198] Table 3
[0199]
[0200] Table 4
[0201]
[0202] Test the integration effect of traffic logs and audit logs based on the following steps:
[0203] As shown in Table 5, all traffic log items have found corresponding quadruple pairs in the audit log, and log grafting has been successfully implemented.
[0204] Table 5
[0205]
[0206] With a concurrency of 255, various applications were accessed concurrently to test the time and space load.
[0207] During the attack execution phase, the time load mainly comes from inserting kernel modules to print auxiliary fusion information for log fusion and inserting dynamic link libraries to close buffers. Fig.10 The black part of the chart records the time changes of request access before and after the transformation of each application. Fig.10 It can be seen that Redis has the largest runtime overhead, which can reach 8.5%.
[0208] In the Attack Execution Phase, the space load mainly comes from the Alignment Info Logs injected into the application log. Since each line of the application log corresponds to an Alignment Info Log Entry, this part will generate a large space load. Fig.11 The black part records the change in the application logs of each application, and the maximum application log increment can reach 53.7%.
[0209] The test shows that the time complexity of each step of fusion matching is linearly related to the number of log lines, which is equivalent to scanning the logs multiple times. When the number of scanned logs is large, the time consumption is shorter. The speed of traversing the three logs using Python is about 10^5 logs / s, which is fast.
[0210] The multi-source log fusion method for advanced persistent attack scenarios provided by this application can be used to solve problems in the field of network security, and can provide a more comprehensive and in-depth perspective for security analysis. By solving the problem of incomplete forensic information, the speed of attack identification is improved, and the attack interpretation and detection process is accelerated. In addition, this application has completed tests in a variety of attack scenarios, and the deterministic matching method of the algorithm can complete the work of each stage with 100% accuracy, while the runtime overhead does not exceed 8.5% and the space overhead does not exceed 53.7%.
[0211] The following is a description of the multi-source log fusion device for advanced persistent attack scenarios provided by the present application. The multi-source log fusion device for advanced persistent attack scenarios described below and the multi-source log fusion method for advanced persistent attack scenarios described above can refer to each other.
[0212] The multi-source log fusion method for advanced persistent attack scenarios provided in the embodiments of the present application can be executed by a multi-source log fusion device for advanced persistent attack scenarios. In the embodiments of the present application, a multi-source log fusion device for advanced persistent attack scenarios performs a multi-source log fusion method for advanced persistent attack scenarios as an example to illustrate the multi-source log fusion device for advanced persistent attack scenarios provided in the embodiments of the present application.
[0213] The embodiment of the present application also provides a multi-source log fusion device for advanced persistent attack scenarios.
[0214] like Fig.13 As shown, the multi-source log fusion device for advanced persistent attack scenarios includes: a first processing module 1310 , a second processing module 1320 and a third processing module 1330 .
[0215] The first processing module 1310 is used to collect application logs corresponding to target applications in the electronic device, traffic logs corresponding to target interfaces in the electronic device, and system audit logs corresponding to the electronic device;
[0216] The second processing module 1320 is used to merge the application log into the audit log based on the thread identifier corresponding to the application log and the collection time corresponding to each line of the first log data in the application log, and merge the traffic log into the audit log to obtain a merged audit log;
[0217] The third processing module 1330 is used to obtain the malicious log corresponding to the target application based on the fused audit log; the malicious log is used to process events corresponding to advanced persistent attacks.
[0218] According to the multi-source log fusion device for advanced persistent attack scenarios provided by the embodiment of the present application, by fusing the collected application logs, traffic logs and audit logs to obtain a fused audit log, and then obtaining the malicious log corresponding to the target application, a more comprehensive semantic context can be established, and attack events can be detected through various log data, especially APT attacks can be detected, and the attacker's attack path can be restored based on the malicious log, thereby helping users to handle APT attacks according to the attack path and achieving comprehensive attack tracing.
[0219] In some embodiments, the second processing module 1320 may also be used to:
[0220] Insert user-entered kernel modules into target applications;
[0221] Based on the kernel module, inserting a thread identifier corresponding to a thread for executing the target application into a corresponding position of the application log;
[0222] The collection time corresponding to the first log data in each line is obtained based on the kernel module.
[0223] In some embodiments, the second processing module 1320 may also be used to:
[0224] Acquire, from the audit log, second log data having the same thread identifier as that corresponding to the application log and having a time difference between the collection time corresponding to the first log data and the collection time corresponding to the first log data less than a target threshold;
[0225] The matched groups of first log data and second log data are fused to obtain a fused audit log.
[0226] In some embodiments, the second processing module 1320 may also be used to:
[0227] Acquire, from the audit log, fourth log data having the same IP address and target port as the third log data included in the traffic log;
[0228] The matched groups of third log data and fourth log data are merged to obtain a merged audit log.
[0229] In some embodiments, the third processing module 1330 may also be used to:
[0230] Based on the fused audit log, the directed traceability graph corresponding to the audit log is updated, and each edge of the updated directed traceability graph corresponds to multiple fused log entries;
[0231] Based on the updated directed tracing graph, malicious logs are obtained.
[0232] In some embodiments, the third processing module 1330 may also be used to:
[0233] Obtain all attack entries in the updated directed traceability graph;
[0234] Connecting a target attack entry among all attack entries with other attack entries except the target attack entry among all attack entries to obtain multiple attack paths;
[0235] Based on the fused log entries corresponding to the edges included in the attack path, malicious logs are obtained.
[0236] In some embodiments, the multi-source log fusion device for advanced persistent attack scenarios may also include a fourth processing module for controlling the closure of the buffer corresponding to the target application before collecting the application log corresponding to the target application in the electronic device, the traffic log corresponding to the target interface in the electronic device, and the system audit log corresponding to the electronic device.
[0237] The multi-source log fusion device for advanced persistent attack scenarios in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a car-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personaldigital assistant, PDA), etc., and can also be a server, a network attached storage (Network AttachedStorage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.
[0238] The multi-source log fusion device for advanced persistent attack scenarios in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0239] The multi-source log fusion device for advanced persistent attack scenarios provided by the embodiment of the present application can achieve Figures 1 to 12 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0240] In some embodiments, Fig.14 As shown, an embodiment of the present application further provides an electronic device 1400, including a processor 1401, a memory 1402, and a computer program stored in the memory 1402 and executable on the processor 1401. When the program is executed by the processor 1401, each process of the multi-source log fusion method embodiment for the advanced persistent attack scenario is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0241] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0242] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the various processes of the above-mentioned multi-source log fusion method embodiment for advanced persistent attack scenarios, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0243] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the various processes of the above-mentioned multi-source log fusion method embodiment for advanced persistent attack scenarios, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0244] On the other hand, an embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned multi-source log fusion method embodiment for advanced persistent attack scenarios, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0245] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0246] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0247] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A multi-source log fusion method for advanced persistent attack scenarios, characterized in that: include: Collecting application logs corresponding to target applications in the electronic device, traffic logs corresponding to target interfaces in the electronic device, and system audit logs corresponding to the electronic device; Based on the thread identifier corresponding to the application log and the collection time corresponding to the first log data of each line in the application log, the application log is merged into the audit log, and the traffic log is merged into the audit log to obtain a merged audit log; Based on the fused audit log, a malicious log corresponding to the target application is obtained; the malicious log is used to process events corresponding to the advanced persistent attack.
2. According to claim 1, the multi-source log fusion method for advanced persistent attack scenarios is characterized in that: Obtaining a thread identifier corresponding to the application log and a collection time corresponding to each line of first log data in the application log, including: Inserting a kernel module input by a user into the target application; Based on the kernel module, inserting a thread identifier corresponding to a thread for executing the target application into a corresponding position of the application log; Based on the kernel module, a collection time corresponding to each line of the first log data is obtained.
3. The multi-source log fusion method for advanced persistent attack scenarios according to claim 1 is characterized in that: The step of fusing the application log into the audit log based on the thread identifier corresponding to the application log and the collection time corresponding to the first log data in each row of the application log to obtain the fused audit log includes: Acquire, from the audit log, second log data having the same thread identifier as that corresponding to the application log and having a time difference between the collection time corresponding to the first log data and the collection time corresponding to the first log data being less than a target threshold; The matched groups of first log data and second log data are merged to obtain the merged audit log.
4. The multi-source log fusion method for advanced persistent attack scenarios according to any one of claims 1 to 3, characterized in that: The merging of the traffic log into the audit log to obtain the fused audit log includes: Acquire fourth log data having the same IP address and target port as the third log data included in the traffic log from the audit log; The matched groups of third log data and fourth log data are merged to obtain the merged audit log.
5. The multi-source log fusion method for advanced persistent attack scenarios according to any one of claims 1 to 3, characterized in that: The acquiring of the malicious log corresponding to the target application based on the fused audit log includes: Based on the fused audit log, updating the directed traceability graph corresponding to the audit log, each edge of the updated directed traceability graph corresponds to a plurality of fused log entries; Based on the updated directed tracing graph, the malicious log is obtained.
6. The multi-source log fusion method for advanced persistent attack scenarios according to claim 5 is characterized in that: The obtaining the malicious log based on the updated directed tracing graph includes: Obtain all attack entries in the updated directed traceability graph; Connecting a target attack entry among all the attack entries and other attack entries among all the attack entries except the target attack entry to obtain multiple attack paths; The malicious log is obtained based on the fused log entries corresponding to the edges included in the attack path.
7. The multi-source log fusion method for advanced persistent attack scenarios according to any one of claims 1 to 3, characterized in that: Before collecting the application log corresponding to the target application in the electronic device, the traffic log corresponding to the target interface in the electronic device, and the system audit log corresponding to the electronic device, the method further includes: Control the buffer corresponding to the target application to close.
8. A multi-source log fusion device for advanced persistent attack scenarios, characterized in that: include: A first processing module, configured to collect application logs corresponding to a target application in an electronic device, flow logs corresponding to a target interface in the electronic device, and system audit logs corresponding to the electronic device; A second processing module is used to merge the application log into the audit log based on the thread identifier corresponding to the application log and the collection time corresponding to each line of the first log data in the application log, and merge the traffic log into the audit log to obtain a merged audit log; The third processing module is used to obtain a malicious log corresponding to the target application based on the fused audit log; the malicious log is used to process events corresponding to the advanced persistent attack.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multi-source log fusion method for advanced persistent attack scenarios is implemented as described in any one of claims 1-7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-source log fusion method for advanced persistent attack scenarios as described in any one of claims 1 to 7 is implemented.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-source log fusion method for advanced persistent attack scenarios as described in any one of claims 1 to 7 is implemented.