A clustering method of APT attacks based on threat intelligence reports

Abstract

The present invention discloses an APT attack clustering method based on threat intelligence reports, including: cleaning CTI reports based on NLP technology to obtain standardized texts; extracting candidate threat behaviors in the standardized texts, matching the candidate threat behaviors with a pre-built attack behavior template library to extract techniques or tactics, and generating attack chains of CTI reports from the extracted techniques or tactics according to the text word order of the CTI reports; clustering the attack chains corresponding to all CTI reports according to the transitive closure concept, and obtaining different categories of APT attacks according to the clustering results. The present invention realizes clustering of APT attacks based on CTI reports.

Description

Technical Field

[0001] The present invention belongs to the field of network security and artificial intelligence technology, and specifically relates to an APT attack clustering method based on threat intelligence reports. Background Art

[0002] Cyber ​​Threat Intelligence (CTI) is data collected, processed and analyzed to understand the motivations, goals and attack behaviors of threat actors. It combines internal and external intelligence to provide defense departments with descriptive information related to APT network threats. CTI reports are unstructured texts provided by network security vendors that describe the context of network threat behaviors. CTI reports are divided into different categories according to different attack methods, which can summarize different attack modes, making it easier for analysts to determine the process, technology and tools of a certain type of attack, and formulate defense strategies based on this existing information.

[0003] CTI reports are multi-source, heterogeneous, and have different styles and are huge in number. Manual analysis of them takes a lot of time. In this regard, machine learning technology can automatically cluster CTI reports into different clusters based on text features. Each cluster represents an attack mode to achieve analysis under a specific attack mode. Commonly used machine learning models for text clustering include K-means, GMM (Gaussian mixture model clustering), Agglomerative, DBSCAN, etc. However, the above methods do not start from the perspective of attack and defense knowledge and combine attack techniques or tactics to restore the attack mode represented by the report. Only considering text features to cluster CTI reports will affect the accuracy and interpretability of the results due to lack of domain knowledge.

[0004] Therefore, how to extract attack tactics and techniques as attack sample features, restore the attack process on this basis, and further cluster APT attacks based on CTI reports is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide an APT attack clustering method based on threat intelligence reports, so as to achieve clustering of APT attacks based on CTI reports.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for clustering APT attacks based on threat intelligence reports, the method comprising:

[0008] Step 1: Clean the CTI report based on NLP technology to obtain standardized text;

[0009] Step 1.1, filter and transform the CTI report to obtain standardized sentences related to the attack;

[0010] Step 1.2: Perform IOC entity recognition on each standardized sentence, replace the matched entity with its attribute word, and then perform word digestion on the standardized sentence to obtain a standardized text containing threatening behavior;

[0011] Step 2: extract candidate threat behaviors from the standardized text, match the candidate threat behaviors with a pre-built attack behavior template library to extract techniques or tactics, and generate an attack chain of the CTI report with the extracted techniques or tactics according to the text word order of the CTI report;

[0012] Step 3: Cluster the attack chains corresponding to all CTI reports according to the transitive closure idea, and derive different categories of APT attacks based on the clustering results.

[0013] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0014] Preferably, the filtering and converting of the CTI report to obtain a standardized sentence related to the attack includes:

[0015] Use the HAN model to filter out text in CTI reports that is not related to the attack;

[0016] Use the BiLSTM model to filter redundant words in sentences in CTI reports;

[0017] Add sentence separators to the remaining attack-related texts, break long sentences into short ones, and add antecedent nouns as subjects to sentences that lack subjects;

[0018] All sentences in passive voice are converted into active voice to obtain standardized sentences related to attacks in CTI reports.

[0019] Preferably, the step of performing word resolution on the standardized sentence includes:

[0020] Pronoun resolution: remove pronouns in normalized sentences and replace them with the antecedent nouns they refer to;

[0021] Entity resolution: replace the reference in the normalized sentence and replace the verb phrase with the verb itself;

[0022] Dependency verb resolution: replace synonymous verbs describing entity dependencies with standard verbs.

[0023] Preferably, the process of constructing the attack behavior template library is as follows:

[0024] According to the descriptions of all attack techniques or sub-techniques in the MITRE ATT&CK framework, construct an attack behavior template T = (c, T b , i, p, t e , t a} for each technique or sub-technique, where c is the current state of the technique or sub-technique, T b is the attack behavior of the technique or sub-technique, i is the tool used by the technique or sub-technique, p is the attack intention of the technique or sub-technique, t e is the threat technique of the technique or sub-technique, and t a is the threat tactic of the technique or sub-technique;

[0025] Among them, the attack behavior T b = ((a1, o1),..., (a n , o n )}, where a i is the attack action of the attack behavior, o i is the attack target of the attack behavior, 1 ≤ i ≤ n, and (a i , o i ) is an atomic attack behavior;

[0026] Aggregate the attack behavior templates of all techniques or sub-techniques as the attack behavior library.

[0027] Preferably, the extraction of candidate threat behaviors from the standardized text includes:

[0028] Use the Stanford CoreNLP model to identify the triple SVO (Subject + Verb + Object) related to threat behaviors in the standardized text, where Subject is the name C of the malware or malicious user b , Verb is the threat action C executed by Subject a , and Object is the execution target C of the malicious threat action o , and record the candidate threat behavior C b = (C a , C o ).

[0029] Preferably, the matching of candidate threat behaviors with the pre-constructed attack behavior template library to extract techniques or tactics includes:

[0030] A. Use the bag-of-words model to represent the candidate threat behaviors and the attack behaviors in the attack behavior templates respectively;

[0031] B. Match all candidate threat behaviors with attack behaviors that only contain one atomic attack behavior. Use the word2vec model to obtain the vectorized representation of each word in the bag-of-words model, and perform weighted addition on each vector to obtain the text vector of the bag-of-words model corresponding to the candidate threat behavior and the attack behavior, and calculate the matching score of the two text vectors. When the matching score is the highest and greater than the matching threshold, the candidate threat behavior is successfully matched with the attack behavior. The matching score calculation formula is as follows:

[0032]

[0033] In the formula, is the text vector of the i-th candidate threat behavior, is the text vector of the j-th attack behavior, For text vector and The matching score of

[0034] C. Match the candidate threat behaviors with the attack behaviors containing multiple atomic attack behaviors, calculate the average m of the number of atomic attack behaviors contained in all attack behaviors, and use the average m as the combination window. According to the sliding combination of the combination window, m candidate threat behaviors are extracted and recorded as the candidate set, and matched with the attack behaviors containing multiple atomic attack behaviors. When formula 2 is satisfied, the match is considered successful;

[0035]

[0036] In the formula, is a candidate set, where represents the mth candidate threat behavior in the candidate set, Included in the combination Inside, Indicates the candidate threat behaviors and attack behaviors T b The best match score, C th is the combined matching threshold, Represents the candidate set with attack behavior T b The best match score;

[0037] D. Delete the candidate threat behaviors that are successfully matched in both step B and step C from the matching results of step B;

[0038] E. Label the techniques or tactics recorded in the template where the attack behavior is located for the candidate threat behavior that has been successfully matched.

[0039] Preferably, clustering the attack chains corresponding to all CTI reports according to the transitive closure concept includes:

[0040] Initialize the adjacency matrix E and the transitive closure matrix E * ;

[0041] Match the attack chains corresponding to all CTI reports in pairs. If the transitive closure threshold is met, the element value of the corresponding position in the adjacency matrix E is set to 1.

[0042] If the element r in the adjacency matrix E ik =1, and element r kj =1, then we get element r ij * =1Add the transitive closure matrix E * , traverse all i, j, k, i, j, k∈[1, n] to get the final transitive closure matrix E * =(r ij * ) n×n , n is the number of CTI reports;

[0043] Traverse the transitive closure matrix E * , if r ij * =1, all attack chains from transfer chain i to j are added to the same class to obtain the final clustering result.

[0044] The beneficial effects of the present invention are mainly manifested in: 1. By analyzing network threat intelligence, a mapping table from IOC entities to regular expressions is constructed, which is convenient for identifying different types of IOCs. 2. Based on the attack behavior template of ATT&CK construction technology and sub-technology, most threat behaviors can be identified. 3. Using the TTP extracted from the CTI report as the feature of clustering, it can combine attack and defense knowledge to improve the accuracy and interpretability of clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the APT attack clustering method based on threat intelligence report of the present invention;

[0046] Figure 2 Schematic diagram of the attack behavior template of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0049] like Figure 1 As shown, this embodiment provides an APT attack clustering method based on threat intelligence reports, including the following steps:

[0050] Step 1: CTI report cleaning based on NLP technology: Use NLP technology to filter and clean the CTI report, identify and replace IOC entities, and resolve words to obtain standardized text containing threat behaviors.

[0051] Step 1.1: Text filtering, cleaning and normalization: Filter and transform the CTI report to obtain normalized sentences related to the attack.

[0052] First, the HAN model is used to filter text in the CTI report that is not related to the attack or cannot be matched to the underlying system log. Then, the BiLSTM model is used to filter redundant words in the sentence. Then, sentence separators are added to the remaining attack-related text to decompose complex long sentences into concise short sentences. For sentences that lack a subject, an antecedent noun is added as the subject. Finally, all sentences in the passive voice are converted to the active voice. At this point, a standardized sentence describing the attack is obtained.

[0053] Step 1.2, IOC replacement and word resolution: Perform IOC entity recognition on each standardized sentence, replace the matched entity with its attribute word, and then perform word resolution on the standardized sentence to obtain a standardized text containing threatening behavior.

[0054] First, rule matching is used to identify IOC entities. After analyzing a large amount of threat intelligence, this article summarizes common IOC types and designs regular expressions for matching six common IOC types: IP, Email, CVE, URL, Domain, and Registry. Successfully matched IOC entities will be replaced with their attribute words (i.e., the corresponding IOC types). Table 1 lists regular expressions for several common IOC types:

[0055] Table 1 Common IOC types and their regular expressions

[0056]

[0057] Then, word resolution is performed, including pronoun resolution, entity resolution, and dependent verb resolution: pronoun resolution refers to deleting the pronouns it / that in the text and replacing them with the antecedent nouns they refer to; entity resolution refers to eliminating co-reference phenomena (such as using attacker in the following text to indicate that the attack organization APT28 was mentioned in the previous text) and deleting verb phrases and replacing them with the verbs themselves; dependent verb resolution refers to replacing synonymous verbs that describe entity dependencies (such as echo, inject, and add all mean write) with pre-defined standard verbs according to Table 2. At this point, a standardized text description that can be used to extract attack behaviors has been obtained.

[0058] Table 2 Dependent verb resolution mapping table

[0059]

[0060] Step 2: TTP extraction based on threat behavior matching: Extract threat behaviors from the processed text and match them with the existing attack behavior template library to extract tactics / techniques. According to the word order of the CTI report text, combine the matched technologies and generate an attack chain for this CTI report on this basis.

[0061] Step 2.1: Build an attack behavior template library: Based on the description of all attack technologies or sub-techniques in MITRE ATT&CK, build an attack behavior template T = {c, T b ,i,p,t e , t a}. Reference Figure 2 The template includes the current state of technology or sub-technology c, attack behavior T b , tools used i, attack intent p, threat technology t e and threat tactics a Aggregate the attack behavior templates of all technologies or sub-technologies as an attack behavior template library.

[0062] Among them, the attack behavior T b ={(a1, o1), ..., (a n , o n )}, where a i The attack action of the attack behavior, o i is the target of the attack behavior, 1≤i≤n. Because the attack behavior contains the attack action a i With the attack target o i There may be more than one. In order to make the attack action correspond to the attack target one by one, the attack behavior template library merges them into atomic attack behaviors (a i , o i ), and store them together in the attack behavior set T b .

[0063] Step 2.2, candidate threat behavior extraction: Use the Stanford CoreNLP model to identify the triples SVO (Subject+Verb+Object) related to threat behaviors in the standardized text, where Subject is the name of the malware or malicious user C b Verb is the threat action performed by Subject C a 、Object is the execution target of the malicious threat action C o , record the candidate threat behavior C b =(C a , C o ).

[0064] Step 2.3, TTP extraction based on template library: Extract candidate threat behavior C b The attack behavior T in the attack behavior template T b Matching. It includes the following steps:

[0065] A. Use the bag-of-words model to represent C b The candidate threat behaviors represented by T b Representative aggressive behavior, such as Figure 2 The example in can be described as {execute, Get-Keystrokes.ps1, write, key.log}.

[0066] Since some attack behavior templates may contain multiple atomic attack behaviors, in order to improve the matching accuracy, this embodiment divides the candidate threat behavior matching into two steps:

[0067] B. First, all candidate threat behaviors are compared with only one atomic attack behavior, namely T b ={(a1, o1)}, use the word2vec model to get the vector representation of each word in the bag of words, and perform weighted addition on each vector to obtain the text vectors of the two bag of words models. Calculate the matching score S according to Formula 1. Take the highest matching score among the candidate threat behaviors that match the same attack behavior and record it as When the highest matching score is greater than the matching threshold S th ,Right now , the match is successful.

[0068]

[0069] In the formula, is the text vector of the i-th candidate threat behavior, is the text vector of the j-th attack behavior, For text vector and The matching score.

[0070] C. Secondly, for attack behavior matching containing multiple atomic attack behaviors, the average m of the number of atomic attack behaviors contained in all attack behaviors (i.e., the number of attack actions and targets contained) is calculated, and the average m is used as the atomic attack behavior combination window. The candidate threat behaviors extracted by the sliding combination are recorded as candidate sets and matched with attack behaviors containing multiple atomic attack behaviors.

[0071] T for aggressive behavior bj , first calculate the matching degree with a single candidate threat behavior according to Formula 1, and get the highest matching score as Then, according to the candidate threat behavior with the highest matching score Determine the candidate set obtained for each slide The candidate set Contains the candidate threat behaviors with the highest matching score The subscripts 1 to m in the candidate set are from 1 to m candidate threat behaviors in the candidate set. Then the attack behavior T is calculated according to Formula 3. bj The matching degree with each candidate set is obtained, and the highest matching score is recorded as When formula 2 is satisfied, the match is considered successful, and the candidate threat behaviors in the candidate set are taken as a whole and the corresponding tactics / techniques are marked.

[0072]

[0073]

[0074] In the formula, is a candidate set, where represents the mth candidate threat behavior in the candidate set, Included in the combination Inside, Indicates the candidate threat behaviors and attack behaviors T b The best match score, C th is the combined matching threshold, Represents the candidate set with attack behavior T b The best match score;

[0075] D. Delete the candidate threat behaviors that are successfully matched in both step B and step C from the matching results in step B.

[0076] E. When all candidate threat behaviors are matched, connect the tactics / techniques into an attack technology chain in the order of the text description in the CTI report.

[0077] Step 3: Clustering of attack samples based on transitive closure: Cluster the attack chains corresponding to all CTI reports according to the transitive closure idea, and summarize different categories of APT attacks from the clustering results.

[0078] Suppose there are n CTI reports, the set of all CTI reports is U, and each CTI report has an attack technology set describing its attack behavior, namely attack chain u i ={t e1 , t e2 , ..., t em}, t em is the mth threat technology in the attack chain. Define the relationship R(u a ,u b ) indicates that CTI report a and CTI report b have a transitive closure threshold value t th The same timing technology, namely |u a ∩u b |≥t th .

[0079] According to the transitive closure idea, There is a relation R(u a ,u b ) and the relation R(u b ,u c ), then the relationship R(u a ,u c ) also exists, and the adjacency matrix E = (r ij ) n×n Represents the relationship between two CTI reports i and j. The specific clustering process is as follows:

[0080] A. Match the attack chains corresponding to all CTI reports in pairs. If the number of common techniques between the i-th CTI report and the j-th CTI report is greater than t th , that is, it is considered that the two meet the transitive closure threshold, and that there is a relationship R between them. Then the element value of the corresponding position in the adjacency matrix E is set to 1, that is, r ij =1.

[0081] B. If the element r in the adjacency matrix E ik =1, and element r kj =1, then we get r ij * =1Add the transitive closure matrix E * , traverse all i, j, k, i, j, k∈[1, n] to get the final transitive closure matrix E * =(r ij * ) n×n , find the transitive closure matrix E corresponding to the adjacency matrix E * =(rij * ) n×n The pseudo code is as follows:

[0082]

[0083] C. Traverse the transitive closure matrix E * , if r ij * =1, if r ij * = 1, then add all attack chains from transfer chain i to j to the same class and traverse the transfer closure matrix E * If there are remaining attack chains that have not been added to the class, each remaining attack chain is added to a new class to obtain the final clustering result, and it is considered that each type of CTI report describes an APT attack process that is similar at the technical and tactical level.

[0084] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A method for clustering APT attacks based on threat intelligence reports, characterized in that: The APT attack clustering method based on threat intelligence reports includes: Step 1: Clean the CTI report based on NLP technology to obtain standardized text; Step 1.1, filter and transform the CTI report to obtain standardized sentences related to the attack; Step 1.2: Perform IOC entity recognition on each standardized sentence, replace the matched entity with its attribute word, and then perform word digestion on the standardized sentence to obtain a standardized text containing threatening behavior; Step 2: extract candidate threat behaviors from the standardized text, match the candidate threat behaviors with a pre-built attack behavior template library to extract techniques or tactics, and generate an attack chain of the CTI report with the extracted techniques or tactics according to the text word order of the CTI report; Step 3: Cluster the attack chains corresponding to all CTI reports according to the transitive closure idea, and derive different categories of APT attacks based on the clustering results.

2. The APT attack clustering method based on threat intelligence report according to claim 1, characterized in that: The filtering and conversion of the CTI report to obtain standardized sentences related to the attack includes: Use the HAN model to filter out text in CTI reports that is not related to the attack; Use the BiLSTM model to filter redundant words in sentences in CTI reports; Add sentence separators to the remaining attack-related texts, break long sentences into short ones, and add antecedent nouns as subjects to sentences that lack subjects; All sentences in passive voice are converted into active voice to obtain standardized sentences related to attacks in CTI reports.

3. The APT attack clustering method based on threat intelligence report according to claim 1, characterized in that: The word resolution of the standardized sentence includes: Pronoun resolution: remove pronouns in normalized sentences and replace them with the antecedent nouns they refer to; Entity resolution: replace the reference in the normalized sentence and replace the verb phrase with the verb itself; Dependency verb resolution: replace synonymous verbs describing entity dependencies with standard verbs.

4. The APT attack clustering method based on threat intelligence report according to claim 1, characterized in that: The attack behavior template library construction process is as follows: According to the description of all attack technologies or sub-techniques in the MITRE ATT&CK framework, an attack behavior template T = {c, T b ,i,p,t e ,t a }, where c is the current state of the technology or sub-technology, T b is the attack behavior of a technology or sub-technology, i is the tool used by the technology or sub-technology, p is the attack intention of the technology or sub-technology, t e Threat technology for a technology or sub-technology, t a Threat tactics for technologies or sub-technologies; Among them, the attack behavior T b ={(a1,o1),…,(a n ,o n )}, where a i The attack action of the attack behavior, o i is the target of the attack behavior, 1≤i≤n, (a i ,o i ) is an act of atomic attack; Aggregate the attack behavior templates of all technologies or sub-technologies as an attack behavior template library.

5. The APT attack clustering method based on threat intelligence report according to claim 1, characterized in that: The step of extracting candidate threat behaviors from the standardized text includes: The Stanford CoreNLP model is used to identify triples SVO (Subject+Verb+Object) related to threat behaviors in standardized texts, where Subject is the name of the malware or malicious user C b Verb is the threat action performed by Subject C a 、Object is the execution target of the malicious threat action C o , record the candidate threat behavior C b =(C a ,C o ).

6. The APT attack clustering method based on threat intelligence report according to claim 1, characterized in that: The extraction of techniques or tactics by matching candidate threat behaviors with a pre-built attack behavior template library includes: A. Use the bag-of-words model to represent candidate threat behaviors and attack behaviors in attack behavior templates; B. Match all candidate threat behaviors with attack behaviors that only contain one atomic attack behavior. Use the word2vec model to obtain the vectorized representation of each word in the bag-of-words model, and perform weighted addition on each vector to obtain the text vector of the bag-of-words model corresponding to the candidate threat behavior and the attack behavior, and calculate the matching score of the two text vectors. When the matching score is the highest and greater than the matching threshold, the candidate threat behavior is successfully matched with the attack behavior. The matching score calculation formula is as follows: In the formula, is the text vector of the i-th candidate threat behavior, is the text vector of the jth attack behavior, For text vector and The matching score of C. Match the candidate threat behaviors with the attack behaviors containing multiple atomic attack behaviors, calculate the average m of the number of atomic attack behaviors contained in all attack behaviors, and use the average m as the combination window. According to the sliding combination of the combination window, m candidate threat behaviors are extracted and recorded as the candidate set, and matched with the attack behaviors containing multiple atomic attack behaviors. When formula 2 is satisfied, the match is considered successful; In the formula, is a candidate set, where represents the mth candidate threat behavior in the candidate set, Included in the combination Inside, Indicates the candidate threat behaviors and attack behaviors T b The best match score, C th is the combined matching threshold, Represents the candidate set with attack behavior T b The best match score; D. Delete the candidate threat behaviors that are successfully matched in both step B and step C from the matching results of step B; E. Label the techniques or tactics recorded in the template where the attack behavior is located for the candidate threat behavior that has been successfully matched.

7. The APT attack clustering method based on threat intelligence report according to claim 1, characterized in that: The attack chains corresponding to all CTI reports are clustered according to the transitive closure concept, including: Initialize the adjacency matrix E and the transitive closure matrix E * ; Match the attack chains corresponding to all CTI reports in pairs. If the transitive closure threshold is met, the element value of the corresponding position in the adjacency matrix E is set to 1. If the element r in the adjacency matrix E ik =1, and element r kj =1, then we get element r ij * =1Add the transitive closure matrix E * , traverse all i,j,k, i,j,k∈[1,n] to obtain the final transitive closure matrix E * =(r ij * ) n×n , n is the number of CTI reports; Traverse the transitive closure matrix E * , if r ij * =1, all attack chains from transfer chain i to j are added to the same class to obtain the final clustering result.

Images