An APT detection method integrating cloud network logs and threat knowledge

Abstract

The present invention discloses an APT detection method that integrates cloud-network-end logs and threat knowledge. The cloud-network-end fusion logs of multiple platforms are collected and processed into a four-tuple sequence in process units and time order to construct an origin graph of the cloud-network-end. Attack entities and entity relationships are extracted from network threat intelligence to form an attack causal graph, which is used in cloud-network-end APT detection. The present invention collects cloud-network-end fusion logs of multiple platforms and extracts attack causal graphs from network threat intelligence. It does not rely on manual work, thereby avoiding the formulation of a large number of rules, helping security industry personnel to more accurately and efficiently identify cloud-network-end fusion APT attacks, thereby achieving effective detection of cloud-network-end fusion APTs.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and specifically relates to an APT detection method integrating cloud network terminal logs and threat knowledge. Background Art

[0002] With the rapid development of the information industry, the cyberspace environment with deep integration of cloud, network and terminal presents a complex trend of diverse and heterogeneous devices, multi-scenario behaviors, polymorphic and blurred boundaries, and multi-source reconstruction of threats. The network attack surface has been infinitely magnified and has become the biggest hidden danger threatening national cyberspace sovereignty and the security of new digital infrastructure. In response to the network attacks of cloud, network and terminal integration, countries around the world have raised cyberspace security governance to the national strategic level and issued relevant policies to promote theoretical research and technical development of malware governance.

[0003] In the face of the endless cyber attacks in recent years, new methods are needed to support and face more and more difficult tasks. Although the academic and industrial research on APT detection in cloud-network-end integration has achieved certain results, it is still difficult to cope with its diversity, concealment, persistence, dynamics and other characteristics, resulting in many challenges in its application and implementation, mainly manifested in:

[0004] The existing cloud-network-end fusion APT detection methods face problems such as single function, difficulty in linkage, low efficiency, and insufficient intelligence. In addition, threat intelligence is usually stored in the form of unstructured text, and the efficiency of manual reading and understanding is too low. It is very different from ordinary text, and it is very difficult to mine structured knowledge from it. Therefore, how to uniformly represent, extract and fuse knowledge and establish a cloud-network-end fusion attack knowledge mining algorithm that adapts to the characteristics of threat intelligence is a key issue in realizing automated extraction and detection of APT. Summary of the invention

[0005] The purpose of the present invention is to provide an APT detection method that integrates cloud-network-end logs and threat knowledge, so as to achieve efficient and accurate detection of APTs integrated with cloud-network-end.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] An APT detection method integrating cloud network terminal logs and threat knowledge, the APT detection method integrating cloud network terminal logs and threat knowledge comprising:

[0008] Step 1: Obtain an application data set, where the application data set includes network threat intelligence, and cloud logs, network end logs, and terminal logs obtained by launching a simulated attack based on the network threat intelligence;

[0009] Step 2: Preprocess the cloud logs, network logs, and terminal logs in the application data set to generate a four-tuple sequence in process units and time order, and build an attack origin graph based on the four-tuple sequence. Each four-tuple in the four-tuple sequence is (P, E, O, T), where P is the process, E is the event type, O is the object, and T is the timestamp;

[0010] Step 3: Process the network threat intelligence in the application data set and build an attack causal graph library, including:

[0011] Step 3.1: For a text of cyber threat intelligence, split long sentences into short sentences according to the symbols contained in the text, and convert the sentences into a standard form;

[0012] Step 3.2, disambiguate the text obtained in step 3.1;

[0013] Step 3.3, delete the redundant text obtained in step 3.2, and keep all entities and subject, predicate and object parts in the sentence;

[0014] Step 3.4: Extract entities and entity relationships from the text obtained in step 3.3;

[0015] Step 3.5: Generate an attack causal graph based on the entities and entity relationships extracted in step 3.4;

[0016] Step 3.6: Repeat steps 3.1 to 3.5 to store the generated single attack causal graph into the attack causal graph library;

[0017] Step 4: Train the Auto-encoder network based on the attack origin graph in step 2 and the attack causal graph library obtained in step 3. During the training, the attack origin graph is used as input and the attack causal graph is used as the true label.

[0018] Step 5: Generate an origin graph to be detected from the cloud logs, network logs and terminal logs to be detected, input the origin graph to be detected into the trained Auto-encoder network, and output the causal graph to be detected. The causal graph is compared with the existing attack causal graph library. If the preset similarity is reached, it is determined that there is an APT attack on the cloud network side and an alarm is issued; otherwise, there is no APT attack on the cloud network side and the detection continues.

[0019] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0020] Preferably, the network threat intelligence is a preset number of unstructured threat intelligence taken from Microsoft security intelligence reports.

[0021] Preferably, converting the sentence into a standard form comprises:

[0022] Detect passive voice in sentences using part-of-speech tags and dependency trees and convert passive voice to active voice using dependency trees.

[0023] Preferably, the disambiguation processing of the text obtained in step 3.1 comprises:

[0024] To address the problem of omitted subjects in texts, we combine grammatical parsing, entity recognition and entity dictionary to select candidate entities from before sentences without subjects, and give priority to the closest entities to restore the omitted subjects; to address the problem of using pronouns in texts, we use the reference resolution algorithm to restore pronouns to entities; to address the problem of synonymous but different words in texts, we use a synonym library compiled by experts to perform disambiguation.

[0025] Preferably, the step of deleting redundancy from the text obtained in step 3.2 comprises:

[0026] To address the problem of text redundancy, the BERT model is used as a classifier to classify sentences, and sentences irrelevant to the attack description are deleted based on the classification results;

[0027] To address the problem of word redundancy, grammatical parsing and entity recognition algorithms are performed successively to retain all entities and subject, predicate and object parts in the sentence.

[0028] Preferably, according to the entities and entity relationships extracted in step 3.4, an attack causal graph is generated, including:

[0029] Semantic role labeling: Extract the two roles of implementer and recipient from the sentences containing the entities and entity relations extracted in step 3.4, and associate each role in the sentence with a semantic label;

[0030] System entity extractor: extracts concise nodes representing system entities from roles generated from semantic role tagging, and prunes the description parts that cannot constitute system entities;

[0031] Causal inference: Distinguish the subject and object of each role based on semantic role labels, and use the direction of the edge from subject to object to represent the causal relationship and information flow between nodes;

[0032] Graph Generator: Generates corresponding edge and node pairs for sentences containing at least one verb and two nodes, and determines the direction of the edge based on causality and information flow.

[0033] Preferably, the step of training the Auto-encoder network according to the attack origin graph of step 2 and the attack causal graph library obtained in step 3 includes:

[0034] Step 4.1, taking the attack origin graph and attack causal graph corresponding to the same network threat intelligence as a training pair, inputting the attack origin graph in the training pair into the Auto-encoder network, and obtaining the predicted causal graph output by the Auto-encoder network;

[0035] Step 4.2, modify the network parameters according to the predicted causal graph and the attack causal graph as the true label in the training pair;

[0036] Step 4.3: Return to step 4.1 until the training reaches the preset end condition.

[0037] The APT detection method provided by the present invention integrates cloud-network-end logs and threat knowledge, collects cloud-network-end fusion logs from multiple platforms, processes them into a four-tuple sequence based on processes and in time order, to construct an origin graph of the cloud-network-end. Attack entities and entity relationships are extracted from network threat intelligence to form an attack causal graph, which is used in cloud-network-end APT detection. The present invention collects cloud-network-end fusion logs from multiple platforms, and extracts attack causal graphs from network threat intelligence, without relying on manual work, thereby avoiding the formulation of a large number of rules, helping security industry personnel to more accurately and efficiently identify cloud-network-end fusion APT attacks, thereby achieving effective detection of cloud-network-end fusion APTs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of the APT detection method of the present application that integrates cloud network terminal logs and threat knowledge;

[0039] Figure 2 This is the threat intelligence conversion flow chart for this application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0042] In order to overcome the drawbacks of cloud-network APT attack detection in the prior art, in one embodiment, an APT detection method that integrates cloud-network logs and threat knowledge is provided.

[0043] like Figure 1 As shown, the APT detection method integrating cloud network terminal logs and threat knowledge in this embodiment includes the following steps:

[0044] Step 1: Obtain an application data set, where the application data set includes network threat intelligence, as well as cloud logs, network end logs, and terminal logs obtained by launching a simulated attack based on the network threat intelligence.

[0045] The APT detection method proposed in this embodiment that integrates cloud-network-end logs and threat knowledge has an important basic part, which is data collection. The authenticity and reliability of the data directly determine the effectiveness of the subsequent training model. For this purpose, the data sets to be used are: cloud-network-end integrated logs, DARPA TC project data sets, and Microsoft security intelligence.

[0046] Cloud-network-end integrated logs: Logs for different terminals and systems can be divided into two main categories: the first category is user-layer log data, which defines user behavior and software behavior (such as software behavior data and user operation data) through human-computer interaction data, and uses the collection of API call data to record software operation logs; the second category is kernel-layer data, which records underlying fine-grained related behaviors by collecting system call methods and related parameters (such as ETW event tracking log data in Windows systems, auditd audit data in Linux systems, and Dtrace tracking data in BSD systems).

[0047] DARPA TC Project Dataset: The DARPA TC Project Dataset is a transparent computing project of the Defense Advanced Research Projects Agency of the United States Department of Defense. It is the standard data set in the field of network attack detection to date, including DDoS (Distribution Deny of Service) attacks, internal attacks, internal monitoring data, and Windows NT traffic and attacks.

[0048] Microsoft Security Intelligence: Microsoft Security Intelligence is security research and threat intelligence collected by Microsoft from a global network of security experts. This embodiment intends to use 4,000 pieces of unstructured threat intelligence for processing. Table 1 describes some malware reports.

[0049] Table 1 Malware Report

[0050]

[0051] This embodiment collects and stores cloud logs, network logs, and terminal logs to collect real-time and efficient cross-platform and multi-domain data, reduces data storage overhead without affecting attack tracking and tracing, and implements multi-source heterogeneous data fusion to improve data quality.

[0052] Step 2: Preprocess the cloud logs, network logs, and terminal logs in the application data set to generate a four-tuple sequence in process units and time order, and build an attack origin graph based on the four-tuple sequence. Each four-tuple in the four-tuple sequence is (P, E, O, T), where P is the process, E is the event type, O is the object, and T is the timestamp.

[0053] In this embodiment, CamFlow processes cloud network logs and generates a streaming four-tuple sequence for detection model training in step four, where P is the process, which is used to uniquely identify the sequence node; E is the event type, indicating the operation of the sequence node, such as process reading and writing; O is the object, which usually refers to files, registry entries, network IPs, etc. related to the process; T is the timestamp, which is used to record the occurrence time of the event and mark the event stream.

[0054] This embodiment constructs a provenance graph based on the kernel audit log (quadruple), which can well express the contextual relationship. Its nodes represent system entities in the kernel log, and the edges represent the causal relationship between the nodes.

[0055] Step 3: Process the network threat intelligence in the application dataset and build an attack causal graph library.

[0056] Cyber ​​threat intelligence is stored in the form of unstructured text. Unlike ordinary text, it has many professional expressions, many redundant sentences, complex entity relationships, and long attack behavior paths. It is difficult to effectively mine cyber attack knowledge if the existing knowledge extraction algorithm is used directly. The present invention uses natural language processing technology to extract entities and entity relationships from text; uses the BERT model of a bidirectional language model to establish a fine-grained representation model, deletes text that is not related to the attack description, and reduces text redundancy; uses semantic role labels and a series of rule modules to extract causal relationships and the direction of information flow to construct an attack causal graph.

[0057] like Figure 2 As shown, the specific transformation steps are as follows:

[0058] Step 3.1: Standardize the text of cyber threat intelligence so that each sentence expresses an action and convert it into a standard form.

[0059] During specific processing, long sentences are divided into short sentences with relatively complete subject, predicate and object and relatively concentrated semantics according to punctuation marks and bullet points and line breaks commonly used in threat intelligence. Part-of-speech tags (POS) and dependency trees are used to detect the passive voice in the sentences, and dependency trees are used to convert the passive voice into active voice.

[0060] Step 3.2: Disambiguate the text obtained in step 3.1 to resolve the ambiguity in the sentences, so that the text consists of sentences with clear subjects, objects and verbs.

[0061] To address the problem of omitted subjects in threat intelligence, we combine grammatical parsing, entity recognition and entity dictionary to select candidate entities from sentences without subjects, and give priority to the nearest entity to restore the omitted subject; to address the problem of using pronouns in threat intelligence, we use reference resolution algorithms (such as NeuralCoref) to restore pronouns to specific entities; to address the problem of synonymous but different words in threat intelligence, we use a synonym library compiled by experts to perform disambiguation.

[0062] Step 3.3: Further summarize the text obtained in step 3.2 and only retain the text that is strongly related to the attack behavior.

[0063] To address the problem of text redundancy in threat intelligence, the BERT model that can learn fine-grained context is used as a classifier in the present invention. Manual labeling is used to determine whether the classified sentences are related to the attack description, and "redundant" sentences that are not related to the attack description are deleted based on the labels. To address the problem of word redundancy in threat intelligence, grammatical parsing and entity recognition algorithms are combined to retain all entities and subject, predicate, and object parts in the sentence.

[0064] Step 3.4: Extract entities and entity relationships from the text obtained in step 3.3.

[0065] Entity extraction: Named Entity Recognition (NER) is the mainstream technology for entity extraction. To address the problem of extremely limited threat intelligence data with entity annotations, we first pre-trained a named entity recognition model on a general corpus with entity annotations, and then used a small amount of threat intelligence corpus with entity annotations to adapt the pre-trained named entity recognition model to the threat intelligence field based on a transfer learning algorithm. To reduce the workload of entity annotation, we combined predefined rules and expert knowledge to semi-automatically complete entity annotation.

[0066] Entity relationship extraction: The relationship between the subject and the object in threat intelligence refers to the behavior of the subject on the object (such as a process sending a file or a process accessing a server). It is usually expressed by the verbs in the sentence. The subject, predicate, and object of the sentence are obtained based on grammatical analysis, and the verb of the predicate is mapped to the standard system behavior based on the system behavior disambiguation dictionary. Finally, the direction of the system behavior is determined by the entity types represented by the subject and object, which is from the subject to the object.

[0067] The process of constructing the system behavior disambiguation dictionary in this embodiment is to first obtain all verbs in the system call, and then take the corresponding synonyms according to each verb, and form the system behavior disambiguation dictionary by the system call verbs and the corresponding synonyms. Therefore, the process of mapping the verb of the predicate to the standard system behavior according to the system behavior disambiguation dictionary is to find the corresponding system call verb according to the verb of the predicate, and replace the verb of the predicate with the found system call verb.

[0068] Step 3.5: Generate an attack causal graph based on the entities and entity relationships extracted in step 3.4, including: semantic role labeling, system entity extractor, causal inference, and graph generator.

[0069] Semantic role labeling: Semantic role labeling is a technique for discovering semantic roles in sentences. It can extract two roles from each sentence: the implementer and the recipient, and correctly associate each component in the sentence with a semantic label. Semantic role labeling is an NLP technique and will not be described in detail in this embodiment.

[0070] System Entity Extractor: Extracts concise nodes representing system entities from roles generated from semantic role tagging and prunes the description parts that cannot constitute system entities. The system entity extractor uses regular expressions (e.g., file entity regular expressions, process entity regular expressions) and a database of application names or well-known processes to detect possible system entity names (e.g., file or process names, IPs, registry keys). The system entity extractor module analyzes each noun phrase and removes the parts that do not match the regular expression.

[0071] Causal Inference: Determine the correct direction of edges in the graph to represent causal relationships and information flow between nodes. To infer this direction, it uses a map of system calls to system flow directions, containing an entry associated with each system call, i.e., the direction of the edge between the subject and the object.

[0072] Graph Builder: Merge semantic role tags with the same text into the same node and use the system entity extractor to remove words that do not belong to the system entity. The graph is constructed using the following method:

[0073] 1) Node-edge-node, for each sentence, if it has at least three roles (including a verb and two nodes), the graph generator generates the corresponding edge and node pairs;

[0074] 2) The direction of the edge: the direction of the edge between nodes is determined based on the results of causal inference.

[0075] Step 3.6: Repeat steps 3.1 to 3.5, and store the generated single causal graph into the attack causal graph library.

[0076] To address the problem that node data occupies a large amount of memory, the present invention adopts the widely used in-memory database Redis in the industry as the storage system of the causal graph generation model, and uses the least recently used method as the memory elimination strategy.

[0077] Step 4: Train the Auto-encoder network based on the attack origin graph in step 2 and the attack causal graph library obtained in step 3, including:

[0078] Step 4.1: Take the attack origin graph and attack causal graph corresponding to the same network threat intelligence as a training pair, input the attack origin graph in the training pair into the Auto-encoder network, and obtain the predicted causal graph output by the Auto-encoder network.

[0079] Step 4.2: Modify the network parameters based on the predicted causal graph and the attack causal graph as the true label in the training pair.

[0080] Step 4.3: Return to step 4.1 until the training reaches the preset end condition. The preset end condition can be reaching the preset number of training cycles, or the Auto-encoder network can accurately output the attack origin graph as the corresponding attack causal graph.

[0081] Step 5: Generate an origin graph to be detected from the cloud logs, network logs and terminal logs to be detected, input the origin graph to be detected into the trained Auto-encoder network, and output the causal graph to be detected. The causal graph is compared with the existing attack causal graph library. If the preset similarity is reached, it is determined that there is an APT attack on the cloud network side and an alarm is issued; otherwise, there is no APT attack on the cloud network side and the detection continues.

[0082] The APT detection method provided by this application integrates cloud-network-end logs and threat knowledge, collects cloud-network-end integrated logs from multiple platforms, processes them into a four-tuple sequence based on processes and time, and constructs a cloud-network-end origin graph. Attack entities and entity relationships are extracted from network threat intelligence to form an attack causal graph, which is used in cloud-network-end APT detection, achieving efficient and accurate detection of cloud-network-end integrated APTs.

[0083] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. An APT detection method integrating cloud network terminal logs and threat knowledge, characterized in that: The APT detection method integrating cloud network terminal logs and threat knowledge includes: Step 1: Obtain an application data set, where the application data set includes network threat intelligence, and cloud logs, network end logs, and terminal logs obtained by launching a simulated attack based on the network threat intelligence; Step 2: Preprocess the cloud logs, network logs, and terminal logs in the application data set to generate a four-tuple sequence in process units and time order, and build an attack origin graph based on the four-tuple sequence. Each four-tuple in the four-tuple sequence is (P, E, O, T), where P is the process, E is the event type, O is the object, and T is the timestamp; Step 3: Process the network threat intelligence in the application data set and build an attack causal graph library, including: Step 3.1: For a text of cyber threat intelligence, split long sentences into short sentences according to the symbols contained in the text, and convert the sentences into a standard form; Step 3.2, disambiguate the text obtained in step 3.1; Step 3.3, delete the redundant text obtained in step 3.2, and keep all entities and subject, predicate and object parts in the sentence; Step 3.4: Extract entities and entity relationships from the text obtained in step 3.3; Step 3.5: Generate an attack causal graph based on the entities and entity relationships extracted in step 3.4; Step 3.6: Repeat steps 3.1 to 3.5 to store the generated single attack causal graph into the attack causal graph library; Step 4: Train the Auto-encoder network based on the attack origin graph in step 2 and the attack causal graph library obtained in step 3. During the training, the attack origin graph is used as input and the attack causal graph is used as the true label. Step 5: Generate an origin graph to be detected from the cloud logs, network logs and terminal logs to be detected, input the origin graph to be detected into the trained Auto-encoder network, output the causal graph to be detected, and compare the causal graph to be detected with the existing attack causal graph library. If the preset similarity is reached, it is determined that there is an APT attack on the cloud network side and an alarm is issued; otherwise, there is no APT attack on the cloud network side and the detection continues.

2. The APT detection method integrating cloud network terminal logs and threat knowledge as claimed in claim 1, characterized in that: The network threat intelligence described is a preset number of unstructured threat intelligence taken from Microsoft Security Intelligence Reports.

3. The APT detection method integrating cloud network terminal logs and threat knowledge as claimed in claim 1, characterized in that: The sentence is converted into a standard form, including: Detect passive voice in sentences using part-of-speech tags and dependency trees and convert passive voice to active voice using dependency trees.

4. The APT detection method integrating cloud network terminal logs and threat knowledge as claimed in claim 1, characterized in that: The disambiguation process of the text obtained in step 3.1 includes: To address the problem of omitted subjects in texts, we combine grammatical parsing, entity recognition and entity dictionary to select candidate entities from sentences without subjects, and select the nearest entity to restore the omitted subject; to address the problem of using pronouns in texts, we use the reference resolution algorithm to restore pronouns to entities; to address the problem of synonymous words in texts, we use a synonym library compiled by experts to perform disambiguation.

5. The APT detection method integrating cloud network terminal logs and threat knowledge as claimed in claim 1, characterized in that: Deleting the redundant text obtained in step 3.2 includes: To address the problem of text redundancy, the BERT model is used as a classifier to classify sentences, and sentences irrelevant to the attack description are deleted based on the classification results; To address the problem of word redundancy, grammatical parsing and entity recognition algorithms are performed successively to retain all entities and subject, predicate and object parts in the sentence.

6. The APT detection method integrating cloud network terminal logs and threat knowledge as claimed in claim 1, characterized in that: According to the entities and entity relationships extracted in step 3.4, an attack causal graph is generated, including: Semantic role labeling: Extract the two roles of implementer and recipient from the sentences containing the entities and entity relations extracted in step 3.4, and associate each role in the sentence with a semantic label; System entity extractor: extracts concise nodes representing system entities from roles generated from semantic role tagging, and prunes the description parts that cannot constitute system entities; Causal inference: Distinguish the subject and object of each role based on semantic role labels, and use the direction of the edge from subject to object to represent the causal relationship and information flow between nodes; Graph Generator: Generates corresponding edge and node pairs for sentences containing at least one verb and two nodes, and determines the direction of the edge based on causality and information flow.

7. The APT detection method integrating cloud network terminal logs and threat knowledge as claimed in claim 1, characterized in that: The Auto-encoder network is trained according to the attack origin graph of step 2 and the attack causal graph library obtained in step 3, including: Step 4.1, taking the attack origin graph and attack causal graph corresponding to the same network threat intelligence as a training pair, inputting the attack origin graph in the training pair into the Auto-encoder network, and obtaining the predicted causal graph output by the Auto-encoder network; Step 4.2, modify the network parameters according to the predicted causal graph and the attack causal graph as the true label in the training pair; Step 4.3: Return to step 4.1 until the training reaches the preset end condition.

Images