APT detection analysis method and system based on network threat intelligence knowledge

Abstract

The invention discloses an APT detection analysis method and system based on network threat intelligence knowledge, and belongs to the technical field of information security. The method comprises the following steps: acquiring an application data set, wherein the application data set comprises multi-source heterogeneous network threat intelligence data and related system traceability logs; constructing an attack traceability graph based on a related system traceability log; according to multi-source heterogeneous network threat intelligence data, constructing a threat intelligence automatic utilization metadatabase composed of coding results of key information tetrads; coding the traceability log edge, and obtaining an attack-related traceability log by matching a coding result of the key information tetrad; constructing an attack traceability sub-graph according to the attack-related traceability log; and for the attack traceability sub-graph, performing attack attribution based on technical and tactical label semantics by using the technical and tactical labels of the traceability log edge, and generating an alarm APT life cycle chain and a standardized attack report. According to the method, efficient APT detection and attribution analysis as well as high-interpretability APT detection analysis can be realized.

Description

Technical Field

[0001] This invention relates to the field of information security technology, specifically to an APT detection and analysis method and system based on network threat intelligence knowledge. Background Technology

[0002] Advanced Persistent Threats (APTs), due to their stealthy, complex, persistent, and highly destructive characteristics, have become a key tool in great power competition. my country has consistently been one of the main victims of APT attacks, and in recent years, both the frequency and severity of these attacks have significantly increased. Consequently, APT attack detection technology has received considerable attention from countries worldwide.

[0003] To achieve timely detection and defense against APT attacks, researchers have proposed using attribution analysis technology to detect and investigate APT attacks recorded in system attribution logs. The advantage of attribution analysis technology lies in its ability not only to detect APT attacks but also to generate attack attribution graphs that describe the data and control flow dependencies between attack steps, revealing the attacker's attack strategy and enabling accurate attack investigations.

[0004] However, existing APT detection and analysis technologies based on attribution analysis lack accuracy and interpretability of detection results, and have insufficient generalization ability to defend against heterogeneous new attacks. Furthermore, since network threat intelligence is mostly stored in unstructured text format, it is difficult to match the structured attributes of commonly analyzed objects—system attribution logs—making it difficult to directly map threat intelligence to system attribution logs and achieve threat intelligence-based APT detection. This results in existing solutions failing to provide practical assistance to users in real-world production applications and significantly hindering the improvement of system security defense capabilities. Therefore, how to fully and efficiently utilize unstructured network threat intelligence text to achieve accurate detection of APT attacks contained in structured system attribution logs, thereby achieving interpretable analysis of APT attacks and defending against heterogeneous new attacks, is the key issue in realizing high-accuracy, highly interpretable, and generalizable attribution analysis-based APT detection. Summary of the Invention

[0005] The technical solution adopted by this invention to solve its technical problem is: an APT detection and analysis method and system based on network threat intelligence knowledge, which can achieve efficient APT detection and attribution analysis, as well as highly interpretable APT detection and analysis.

[0006] To achieve the above-mentioned objectives, the technical solution of the present invention includes the following:

[0007] An APT detection and analysis method based on network threat intelligence knowledge, the method comprising:

[0008] Obtain the application dataset, which includes: multi-source heterogeneous network threat intelligence data and related system tracing logs;

[0009] An attack source graph is constructed based on the relevant system source logs. The source log nodes in the attack source graph represent system entities in the system source logs, and the source log edges represent the causal relationships between the source log nodes.

[0010] Based on the multi-source heterogeneous network threat intelligence data, a threat intelligence automated utilization meta-database is constructed, consisting of the encoded results of key information quadruples; wherein, the elements in the key information quadruples include: a detailed description of the entity performing the operation, the operation relationship, a detailed description of the entity being operated on, and compromise indicators;

[0011] The source tracing log edges are encoded, and the attack-related source tracing logs are obtained by matching the encoding results of the key information quadruples.

[0012] Based on the attack-related attribution logs, construct an attack attribution subgraph;

[0013] For the attack attribution subgraph, the technical and tactical tags of the attribution log edges are used to perform attack attribution based on the semantics of the technical and tactical tags, and an alert APT lifecycle chain is generated.

[0014] Furthermore, the construction of the attack tracing graph based on the relevant system tracing logs includes:

[0015] The relevant system traceability logs are preprocessed to generate a quintuple sequence in chronological order, with process name and number as the unit. Each quintuple in the sequence is (S, P, O, C, T), where S is the type of system call, P is the process name and number that issued the system call, O is the target of the system call, C is the command line to which the system call belongs, and T is the time when the system call occurred.

[0016] Construct an attack origin graph based on the quintuple sequence.

[0017] Furthermore, based on the aforementioned multi-source heterogeneous network threat intelligence data, a threat intelligence automated utilization metadata database is constructed, consisting of the encoded results of key information quadruples, including:

[0018] Step 3.1: Identify whether the network threat intelligence data has technical and tactical labels. If the network threat intelligence data has technical and tactical labels, proceed to step 3.2; if the network threat intelligence data does not have technical and tactical labels, proceed to step 3.5.

[0019] Step 3.2: Based on the attack description tags and tactical tags in the network threat intelligence data, construct a triplet sequence containing tactical tags in text sentences. Each tuple in the triplet sequence is (Sen, Tac, Tech), where Sen is the text sentence, Tac is the tactical tag corresponding to the text sentence, and Tech is the technical tag corresponding to the text sentence.

[0020] Step 3.3: Extract key information from the text sentences to obtain key information quadruples;

[0021] Step 3.4: Encode the key information quadruple using a fast encoding model to obtain the encoding result of the key information quadruple; wherein, the fast encoding model is pre-trained using network security text;

[0022] Step 3.5: Identify attack description tags in network threat intelligence texts, and segment the attack description text into text sentences according to the symbols contained therein;

[0023] Step 3.6: Extract key information from the text sentences to obtain key information quadruples;

[0024] Step 3.7: Use the fast encoding model to encode the key information quadruple to obtain the encoding result of the key information quadruple.

[0025] Furthermore, the extraction of key information from the text sentence yields a key information quadruple, including:

[0026] Using syntactic analysis and lexical analysis techniques, the prefix and suffix modifiers of the subject and object entities in the text sentence are expanded to obtain entities with complete modifiers;

[0027] And / or,

[0028] Using Stanford Natural Language Processing (NLP) technology, we can identify supplementary clauses in text sentences and extract key information from these supplementary clauses.

[0029] And / or,

[0030] For text sentences with incomplete information, key information is extracted by constructing incomplete tuples.

[0031] And / or,

[0032] The IoCParser tool is extended to detect and identify IoCs of path, file, registry, and command line types.

[0033] And / or,

[0034] Passive sentences are identified using lexical and syntactic analysis techniques, and key information is extracted after the passive sentences are converted into active sentences.

[0035] Further, the process of encoding the source tracing log edges and obtaining attack-related source tracing logs by matching the encoding results of the key information quadruples includes:

[0036] Obtain a specific portion from the source log edge, the specific portion including: the type of system call, the name and number of the process that issued the system call, the target of the system call, the command line to which the system call belongs, and the time when the system call occurred;

[0037] After encoding specific parts of the source log edge and specific parts of the quadruple information, the similarity is calculated one by one. The sum of the overall similarity between the source log edge and each element of the quadruple information is used as the matching score between the log edge and the current quadruple. The sum of the matching scores between all quadruples in each technology / tactic and the source log edge is used as the threat score between the current technology / tactic and the log.

[0038] A specific number of technical tags, tactical tags, and corresponding threat scores with the highest threat scores are retained to generate attack-related attribution logs;

[0039] On the benign source analysis log dataset, threat intelligence is used to automatically match entries in the metadata database and obtain the minimum detection threshold based on the Gibbs test.

[0040] When the matching score between the application dataset and the entries in the threat intelligence automated utilization metadata database is higher than the minimum detection threshold, the corresponding information is retained in the attack-related tracing log.

[0041] Furthermore, the step of calculating the similarity between specific parts of the source log edge and specific parts of the four-tuple information after encoding includes:

[0042] The specific descriptions of the entities performing the operation, the operation relationships, and the specific descriptions of the entities being operated on in each key information quadruple in the threat intelligence automation utilization metadata database are clustered using a bi-band optimized clustering algorithm, and the cluster center vector is calculated.

[0043] The cluster center vector is matched with the information of the source log edge using encoded vector matching; wherein, during the encoded vector matching calculation, the vector matching calculation acceleration library in the NumPy library and the graphics processor are used to accelerate the encoded vector matching calculation.

[0044] Furthermore, the step of constructing an attack attribution subgraph based on the attack-related attribution logs includes:

[0045] The attack-related source logs are used to generate a set of Steiner trees in the candidate source graph using a source graph construction technique based on the Steiner tree optimization algorithm.

[0046] Based on the Steiner tree, obtain the attack tracing subgraph.

[0047] Furthermore, the attack attribution subgraph utilizes the technical and tactical tags of the attribution log edges to perform attack attribution based on the semantics of the technical and tactical tags, generating an alert APT lifecycle chain, including:

[0048] Based on expert knowledge, the technical and tactical labels of the source tracing log edges are mapped and the APT lifecycle stages of each node in the attack source tracing subgraph are summarized.

[0049] The APT lifecycle phases of the nodes are merged to obtain an attack tracing subgraph with reduced node and edge size;

[0050] For the attack tracing subgraph with reduced node and edge size, a legality check is performed based on the temporal logical relationship of the APT lifecycle stages to obtain an alarm APT lifecycle chain that conforms to the APT lifecycle theory. The threat level of the alarm APT lifecycle chain is then calculated based on the node threat score.

[0051] Furthermore, after generating an alert APT lifecycle chain by performing attack attribution based on the semantics of technical and tactical tags using the technical and tactical tags of the attack attribution log edges on the attack attribution subgraph, the process also includes:

[0052] Based on the attack tactics and lifecycle stage tags of the alerted APT lifecycle chain, generative artificial intelligence tools are used to generate attack description text, and further, standardized APT attack reports are generated. The process of generating attack description text and standardized APT attack reports based on the attack tactics and lifecycle stage tags of the alerted APT lifecycle chain, using generative artificial intelligence tools, includes:

[0053] Generative AI tools are trained and fine-tuned in the field of security threat intelligence using open-source threat intelligence datasets, including GPT, LLaMa, Claude, or Qwen.

[0054] Further fine-tuning of the generative AI tool using prompt words and standardized APT attack reports included in open-source threat intelligence datasets;

[0055] For each node in each stage of the APT attack lifecycle chain, a script is used to automatically generate a prompt word containing information about the node itself and its context. This allows a fine-tuned generative AI tool to generate a behavioral semantic explanation of the node in that stage based on the prompt word. Based on the behavioral semantic explanation of the node and the node's tag, an attack description text for that stage is obtained.

[0056] Based on the attack description text of each node in each stage, and using prompt words, the generative artificial intelligence tool summarizes and generates an overall APT attack report.

[0057] An APT detection and analysis system based on network threat intelligence knowledge, the system comprising:

[0058] The dataset acquisition module is used to acquire application datasets, which include: multi-source heterogeneous network threat intelligence data and related system tracing logs;

[0059] An attack source graph construction module is used to construct an attack source graph based on the relevant system source logs. The source log nodes in the attack source graph represent system entities in the system source logs, and the source log edges represent the causal relationships between the source log nodes.

[0060] The meta-database construction module is used to construct a threat intelligence automated utilization meta-database composed of the encoded results of key information quadruples based on the multi-source heterogeneous network threat intelligence data; wherein, the elements in the key information quadruples include: a specific description of the entity performing the operation, the operation relationship, a specific description of the entity being operated on, and a compromise index.

[0061] An attack-related tracing log generation module is used to encode the edges of the tracing log and obtain the attack-related tracing log by matching the encoding results of the key information quadruple;

[0062] The attack attribution subgraph construction module is used to construct an attack attribution subgraph based on the attack-related attribution logs.

[0063] The detection and analysis result generation module is used to perform attack attribution based on the semantics of technical and tactical tags using the technical and tactical tags of the attack tracing subgraph and generate an alert APT lifecycle chain.

[0064] The attack report generation module is used to generate attack description text based on the attack tactics and lifecycle stage tags of the alerted APT attack lifecycle chain, and generate standardized APT attack reports using generative artificial intelligence tools.

[0065] Compared with the prior art, the present invention has at least the following beneficial effects.

[0066] This invention provides an automated APT detection and analysis method utilizing open-source network threat intelligence. It collects source tracing log datasets and processes them into a sequence of five-tuples ordered by process name and ID, and time, to construct a system-level source tracing graph. Key information four-tuples are extracted from multi-source heterogeneous open-source network threat intelligence and applied to APT detection and semantic-based attribution in system source tracing logs, making the APT detection results readable. Through matching optimization, attack source tracing subgraph construction optimization, and rapid attack attribution, efficient APT detection and attribution analysis are achieved. This invention collects system-level source tracing logs and extracts key information four-tuples from multi-source heterogeneous open-source network threat intelligence, completely eliminating manual intervention and avoiding extensive manual rule formulation. It also enriches the interpretability of APT detection results by presenting APT attack lifecycle chains and corresponding standardized attack reports, helping security professionals analyze and understand APT attack alerts more intuitively and efficiently, thus achieving highly interpretable automated APT detection and analysis using open-source network threat intelligence. Attached Figure Description

[0067] Figure 1 This is a flowchart of the advanced persistent threat detection and analysis system based on network threat intelligence knowledge, which is based on the present invention.

[0068] Figure 2 This is a flowchart of the automated threat intelligence utilization metadata database construction process of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments that can be carried out by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0071] In order to overcome the shortcomings of existing APT attack detection and analysis technologies, one embodiment provides an automated APT detection and analysis method that utilizes open-source network threat intelligence knowledge.

[0072] like Figure 1As shown, the automated APT detection and analysis method utilizing open-source network threat intelligence knowledge in this embodiment includes the following steps:

[0073] Step 1: Obtain the application dataset. The open-source threat intelligence includes multi-source heterogeneous open-source threat intelligence data and related system traceability logs.

[0074] The automated APT detection and analysis method proposed in this embodiment, which utilizes open-source network threat intelligence, relies heavily on data collection. The authenticity and reliability of the data directly determine the effectiveness of the subsequent model training. The relevant system attribution logs include: system attribution logs obtained from simulated APT attacks based on the latest security vulnerabilities, and open-source APT attribution analysis log datasets. The datasets to be used include: the DARPA TC project dataset, the In-lab Arena dataset, the NewlySim dataset, and open-source heterogeneous threat intelligence from companies and organizations such as Google Cloud, Kaspersky, and MITRE.

[0075] The DARPA TC project dataset is a transparent computing project of the U.S. Defense Advanced Research Projects Agency (DARPA). It is the standard dataset in the field of network attack detection to date, and includes DDoS (Distribution Deny of Service) attacks, insider attacks, insider eavesdropping data, and Windows NT traffic and attacks.

[0076] In-lab Arena Dataset: The In-lab Arena dataset is a standard dataset for source tracing analysis obtained by NodLink, the latest work on APT attack detection, based on real APT attack instances that occur in industrial production environments and simulated in a laboratory environment. It includes various types of APT attacks such as APT29 and SideWinder, and is currently the mainstream test dataset.

[0077] The NewlySim dataset contains new APT attack instances based on the latest cybersecurity vulnerabilities (CVEs) that occurred in 2023 and 2024. It is an important test dataset used to verify the generalization ability of APT detection and analysis work.

[0078] Open-source heterogeneous threat intelligence from companies like Google Cloud, Kaspersky, and MITRE: This example utilizes approximately 600 pieces of open-source heterogeneous threat intelligence collected and compiled by global cybersecurity experts through analysis and research of cyber threat incidents.

[0079] Step 2: Preprocess the relevant system source logs in the application dataset to generate a quintuple sequence in chronological order, with process name and number as the unit. Construct an attack source graph based on the quintuple sequence. Each quintuple in the sequence is (S, P, O, C, T), where S is the system call type, P is the process name and number that issued the system call, O is the target of the system call, C is the command line to which the system call belongs, and T is the time when the system call occurred.

[0080] This example uses stream processing technology and coded scripts to preprocess system tracing logs in the application dataset and generate a streaming 5-tuple sequence for verification in step 3. Here, S is the system call type, indicating the operation of the sequence node, such as process spawning; P is the process and its number that issued the system call, used to uniquely identify the sequence node; O is the target of the system call, such as the file, registry key, network address, or other processes operated on by the process; C is the command line to which the system call belongs, including the command line command and specific parameter information; and T is the time when the system call occurred, used to record the time of the event and mark the event stream.

[0081] This embodiment constructs an origin graph based on the system source log (five-tuple), which can well express the contextual relationships and contains sufficient semantic information about system behavior. Its nodes represent system entities in the source log, and edges represent the causal relationships between nodes.

[0082] Step 3: Process the multi-source heterogeneous open-source threat intelligence data in the application dataset to establish a metadata database for automated threat intelligence utilization.

[0083] Cyber ​​threat intelligence is text information recorded in unstructured, professional text format. Unlike ordinary text information, it is characterized by numerous professional expressions, non-natural language terms, incomplete sentences, complex relationships, and rich modifiers. Directly using existing information extraction algorithms makes it difficult to fully and accurately extract cyber threat knowledge from it. This invention employs natural language processing technology, based on lexical analysis, syntactic analysis, and regular expression extraction, to completely identify entities with specific descriptions, operational relationships between entities, and compromise indicators from cyber threat intelligence text. It uses a FastText encoding model for efficient encoding of key cyber threat intelligence information and pre-trains the FastText model using cybersecurity text to overcome the problem of incomplete corpus coverage in the model.

[0084] like Figure 2 As shown, the specific construction steps are as follows:

[0085] Step 3.1: For a network threat intelligence text, first identify whether it has a technical and tactical (ATT&CK) tag description. If it does, proceed to step 3.2; otherwise, proceed to step 3.5.

[0086] In specific processing, based on the tags and technical and tactical characteristics (such as technical and tactical names and numbers) contained in the network threat intelligence text itself, we can identify and determine whether the current network threat intelligence has technical and tactical tags.

[0087] Step 3.2: Identify attack description tags and tactical analysis tags in network threat intelligence texts, obtain attack description text and tactical tag information of network threat intelligence, segment the attack description text into text sentences according to the symbols contained therein, and extract the corresponding tactical tags, construct a triplet sequence containing tactical tags with text sentences as units, each tuple in the triplet sequence is (Sen, Tac, Tech), where Sen is the text sentence, Tac is the tactical tag corresponding to the text sentence, and Tech is the technical tag corresponding to the text sentence.

[0088] In specific processing, based on the tag information described in 3.1 and 3.2, the attack description text and technical and tactical tag information of network threat intelligence contained in network threat intelligence are identified and segmented. According to punctuation marks and commonly used bullet points and line breaks in threat intelligence, long text segments are divided into short text sentences with complete information and concentrated semantics.

[0089] Step 3.3: Extract the key information from the text sentences in each triplet obtained in Step 3.2, including entities with specific descriptions, operational relationships between entities, and the Inversion of Control (IoC). This key information can be denoted as a quadruple, (S, V, O, I), where S is the specific description of the entity performing the operation, V is the operational relationship, O is the specific description of the entity being operated on, and I is the Inversion of Control.

[0090] To address the issue of incomplete descriptive modifiers for entities, syntactic and lexical analysis techniques are used to expand the prefix and suffix modifiers of traditional subject and object entities, obtaining entities with complete modifiers. To address the issue of insufficient clause information extraction, Stanford CoreNLP is used to identify supplementary clauses in text sentences and extract key information from them. To address the issue of incomplete text sentence information, incomplete tuples are used to extract partial information, forming partial tuples as supplementary information. To address the issue of incomplete IoC extraction, based on regular expressions, the existing best-practice tool (IoCParser) is extended to support a wide range of IoC detection and recognition types, including paths, files, registry entries, and command lines. To address the issue of inconsistent subject and passive voice in text sentences, lexical and syntactic analysis techniques are used to identify passive voice, convert passive sentences into active sentences, and extract key information.

[0091] Specifically, based on regular expressions, this technique extends the existing optimal tool (IoCParser). Considering that the existing optimal tool IoCParser only supports generalized IoC retrieval and mining, this technique uses regular expressions to refine the IoC retrieval and mining rules of IoCParser, and modifies and corrects some of the erroneous rules:

[0092] 1) To address the issue that IoCParser only supports recognizing Windows paths and files, this technology, based on the characteristics of Linux, Huawei HarmonyOS, and other system paths and files, uses regular expressions to write rules for recognizing and mining Linux, Huawei HarmonyOS, and other system paths and files, thus extending the system support of IoCParser.

[0093] 2) To address the issues of false positives and overly broad IP address identification by IoCPaser, this technology develops corresponding identification and mining rules based on the differences in characteristics between internal and external IP addresses. Additionally, it uses the nslookup command to identify the domain name of an external IP address and adds it to the identified external IP address, thereby achieving refined identification and information expansion for both internal and external IP addresses.

[0094] 3) To address the issue of IoCParser's lack of accurate support for identifying Windows registry keys, this technology develops comprehensive identification and mining rules based on the identification characteristics of Windows registry keys, thereby extending IoCParser's full support for identifying Windows registry keys.

[0095] 4) To address the lack of command-line recognition support in IoCParser, this technique crawls commonly used Linux and Windows command lines from Linux and Windows technical documentation and supplements these command lines using GPT. Corresponding recognition and mining rules are then developed using these command lines to enable IoCParser to recognize command lines.

[0096] Step 3.4: Repeat step 3.3, encoding each quadruple obtained in step 3.3 and storing them in the corresponding technical and tactical tags in the threat intelligence automated utilization metadata database. After completion, proceed to step 4.

[0097] To address the issue of insufficient encoding speed, a FastText encoding model is used for information encoding. To address the issue of out-of-vocabulary (OOV) encoding, the FastText encoding model is pre-trained using cybersecurity texts to enable it to identify cybersecurity-specific entities and overcome the OOV problem.

[0098] Step 3.5: Identify attack description tags in the network threat intelligence text, and segment the attack description text into text sentences according to the symbols contained therein.

[0099] In specific processing, based on the tag information described in 3.5, the attack description text of network threat intelligence contained in the network threat intelligence is identified and segmented, and long text segments are divided into short text sentences with complete information and concentrated semantics according to punctuation marks and commonly used bullet points and line breaks in threat intelligence.

[0100] Step 3.6: Perform the same operation as in Step 3.3 on each text sentence obtained in Step 3.5 to extract the key information from the text sentence and obtain the corresponding key information quadruple.

[0101] To address the issue of incomplete descriptive modifiers for entities, syntactic and lexical analysis techniques are used to expand the prefix and suffix modifiers of traditional subject and object entities, obtaining entities with complete modifiers. To address the issue of insufficient clause information extraction, Stanford CoreNLP is used to identify supplementary clauses in text sentences and extract key information from them. To address the issue of incomplete text sentence information, incomplete tuples are used to extract partial information, forming partial tuples as supplementary information. To address the issue of incomplete IoC extraction, based on regular expressions, the existing best-practice tool (IoCParser) is extended to support a wide range of IoC detection and recognition types, including paths, files, registry entries, and command lines. To address the issue of inconsistent subject and passive voice in text sentences, lexical and syntactic analysis techniques are used to identify passive voice, convert passive sentences into active sentences, and extract key information.

[0102] Step 3.7: Repeat step 3.6, encoding each quadruple obtained in step 3.6, and then performing correlation detection with the existing quadruple codes in the threat intelligence automated metadata database. Add each quadruple code to the corresponding highly relevant database entry. After completion, proceed to step 4.

[0103] To address the issue of insufficient encoding speed, a FastText encoding model is used for information encoding. To address the issue of out-of-vocabulary (OOV) encoding, the FastText encoding model is pre-trained using cybersecurity texts to enable it to identify cybersecurity-specific entities and overcome the OOV problem.

[0104] Step 4: Encode each source log edge in the attack source graph generated in Step 2, and match it with each quadruple code in the threat intelligence automation metadata database to detect attack-related source logs and assign them corresponding technical tags, tactical tags, and threat scores.

[0105] Matching Method: Following expert-knowledge-based rules, specific parts of the source log edge (i.e., the information in the quintuple in step 2) are encoded one by one with specific parts of the quadruple information, and their similarity is calculated. The final similarity score between the log edge and each element of the quadruple is used as the matching score between the log edge and the current quadruple. The sum of the matching scores between all quadruples in each technique / tactic and the log edge is used as the threat score between the current technique / tactic and the log. A specific number of technique tags, tactical tags, and their corresponding threat scores with the highest threat scores are retained. For example, if the top three technique / tactical tags are retained, they would be: Tactic 1 - Technique 1 - Corresponding Score 1 (highest score), Tactic 2 - Technique 2 - Corresponding Score 2 (second highest score), and Tactic 3 - Technique 3 - Corresponding Score 3 (third highest score).

[0106] Matching Acceleration Optimization: To address the issue of slow matching speed due to the large number of entries in the matching database, the threat intelligence automation utilizes the S, V, and O three-dimensional information from each key information quadruple in the metadata database. A bi-band optimized clustering method (Bi-Kmeans++) is used for clustering, and representative elements of each cluster (i.e., cluster center vectors) are matched with log edge information to improve matching speed. To address the issue of slow matching speed due to low single-match computation rates, a vector matching computation acceleration library from the NumPy library and a graphics processing unit (GPU) are used to accelerate the encoded vector matching computation.

[0107] Matching performance optimization: On the benign source analysis log dataset, the entries in the threat intelligence automated utilization metadata database are matched. The minimum detection threshold is obtained based on the Gibbs test. Only when the matching score between the detection dataset and the entries in the threat intelligence automated utilization metadata database is higher than the minimum detection threshold is the matching score and threat score included, thereby achieving the elimination of false alarms for benign behavior.

[0108] Step 5: Based on the Steiner tree optimization algorithm, construct an accurate and low-redundancy attack tracing subgraph from the attack-related source logs detected in Step 4.

[0109] The attack-related source logs detected in step 4 are used to construct a source graph based on the Steiner tree (STP) optimization algorithm. A set of Steiner trees are generated in the candidate source graph. These Steiner trees obtain the minimum source graph that most approximates all detected attack-related logs in real time, thus constructing an accurate and low-redundancy attack source graph.

[0110] Step 6: Using the attack attribution subgraph obtained in Step 5, perform attack attribution based on the semantics of technical and tactical tags from the attribution log edges to generate an alert APT lifecycle chain.

[0111] Step 6.1: Using the technical and tactical labels of the source tracing log edges, and based on expert knowledge, map the technical and tactical labels to summarize the APT lifecycle stage of each node in the attack source tracing subgraph obtained in Step 5.

[0112] Step 6.2: Merge the APT lifecycle stages of the nodes obtained in Step 6.1 to obtain an attack tracing subgraph with a further reduced node and edge size.

[0113] Step 6.3: Take the attack tracing subgraph obtained in Step 6.2, perform a legality check according to the temporal logical relationship of the APT lifecycle stages, eliminate illegal stage nodes, and obtain the final alarm APT lifecycle chain that conforms to the APT lifecycle theory. Based on the node threat score, calculate the threat level of the alarm APT lifecycle chain.

[0114] Step 7: Based on the attack tactics and lifecycle stage tags of the APT attack lifecycle chain obtained in Step 6, use generative artificial intelligence tools such as GPT, LlaMa, and Qwen to generate attack description text, and further generate a standardized APT attack report.

[0115] Step 7.1: Use an open-source threat intelligence dataset to train and fine-tune GPT in the field of security threat intelligence, so that it can understand the predictive features of security-related text.

[0116] Step 7.2: Using prompts and standardized APT attack reports contained in open-source threat intelligence datasets, further fine-tune generative AI tools such as GPT to enable them to understand the basic components, content organization, and key descriptive points of APT attack reports. Prompts might look like this: "The following shows the basic content of a standardized APT attack report. Please study the content structure features of this report and master the characteristics of the information composition, content description, and organizational structure of a standardized APT attack report."

[0117] Step 7.3: For each node in each stage of the APT attack lifecycle chain, a script automatically generates a prompt word containing the node's own information (i.e., the five-tuple information from Step 2, the technical and tactical tags obtained in Step 4, and the lifecycle chain stage tags obtained in Step 6), and the node's context information (i.e., the aforementioned information of the node's predecessor and successor nodes, as well as the predecessor and successor stage tags of the stage the node is currently in). This prompt word enables GPT to generate an explanatory text describing the semantics of the node's behavior and adds it as a tag to the explanatory text describing the node's behavior, generating an attack description text for the node's behavior. The prompt word is in the form of: The current node performed the following operation: At a certain moment, a process performed a system call on a certain operation object, using the command line XXX. Its behavior is marked with the technical tag XX, the tactical tag XX, and the APT attack lifecycle chain stage tag XX. At the same time, the node's predecessor node performed the following operation: At a certain moment, a process performed a system call on a certain operation object, using the command line XXX. Its behavior is tagged with technical label XX, tactical label XX, and APT attack lifecycle stage label XX. The downstream nodes of this node performed the following operation: At a certain moment, a process executed a system call on a certain operation object, using the command line XXX. Its behavior is tagged with technical label XX, tactical label XX, and APT attack lifecycle stage label XX. The APT attack lifecycle stage label of this node is XX, the preceding stage label of this node's APT attack lifecycle stage is XX, and the downstream stage label of this node's APT attack lifecycle stage is XX. First, determine whether this node's behavior might belong to the attacker. If so, generate an attack description text for this node's behavior based on the above information, following the format of the attacker's behavior description text in an APT attack report.

[0118] Step 7.4: Based on the attack description text of each node in each stage above, use prompt words to generate an overall APT attack report from GPT. The prompt words are in the form of: In this attack campaign, the attacker performed the following actions in chronological order: 1) The attacker executed the XXX command line, and at a certain moment used a certain process to execute a system call on a certain object. This action was marked with the technical tag XX, the tactical tag XX, and the APT attack lifecycle stage tag XX. Its description is XXX (attack description text generated in Step 7.3) 2) ... Based on the above attack behavior information, and in accordance with the standardized organization and content requirements of an APT attack report, generate a standardized APT attack report describing the aforementioned attack behavior, summarize the involved IoCs, the attack behaviors used, analyze the possible attack harms, and generate targeted corresponding strategies and defense measures.

[0119] In summary, the automated APT detection and analysis method provided by this invention, utilizing open-source network threat intelligence, collects source tracing log datasets and processes them into a sequence of five-tuples ordered by process name and ID, and time, to construct a system-level source tracing graph. Key information four-tuples are extracted from multi-source heterogeneous open-source network threat intelligence and applied to APT detection and semantic-based attribution in system source tracing logs, making the APT detection results readable. Through matching optimization, attack source tracing subgraph construction optimization, and rapid attack attribution, efficient APT detection and attribution analysis are achieved. This invention collects system-level source tracing logs and extracts key information four-tuples from multi-source heterogeneous open-source network threat intelligence, completely eliminating manual intervention and avoiding extensive manual rule formulation. Simultaneously, it enriches the interpretability of APT detection results, providing security professionals with a more intuitive and efficient way to analyze and understand APT attack alerts in the form of APT attack lifecycle chains and corresponding standardized attack reports. This achieves highly interpretable APT detection and analysis using automated open-source network threat intelligence.

[0120] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for APT detection analysis based on network threat intelligence knowledge, characterized in that, The method comprises: acquiring an application data set, the application data set comprising: multi-source heterogeneous network threat intelligence data and related system trace logs; constructing an attack trace graph based on the related system trace logs, a trace log node in the attack trace graph representing a system entity in the system trace log, and a trace log edge representing a causal relationship between the trace log nodes; constructing a threat intelligence automated utilization meta-database composed of encoding results of key information four-tuples according to the multi-source heterogeneous network threat intelligence data, wherein elements in the key information four-tuple include: specific description of an entity performing an operation, operation relationship, specific description of an entity being operated on, and compromise indicator; encoding the trace log edges and obtaining attack-related trace logs by matching the encoding results of the key information four-tuples; constructing an attack trace subgraph according to the attack-related trace logs; performing attack attribution based on technology and tactic label semantics on the attack trace subgraph using technology and tactic labels of the trace log edges to generate an alert APT life cycle chain.

2. The method of claim 1, wherein, The method comprises: preprocessing the related system trace logs to generate a five-tuple sequence with process name and number as units and time as sequence; wherein each tuple in the five-tuple sequence is (S, P, O, C, T), S is the type of system call, P is the process name and number issuing the system call, O is the action object of the system call, C is the command line to which the system call belongs, and T is the time when the system call occurs; constructing an attack trace graph according to the five-tuple sequence.

3. The method of claim 1, wherein, The method comprises: Step 3.1: identifying whether the network threat intelligence data has a technique and tactic label description, and jumping to step 3.2 if the network threat intelligence data has a technique and tactic label description; jumping to step 3.5 if the network threat intelligence data does not have a technique and tactic label description; Step 3.2: constructing a three-tuple sequence containing technique and tactic labels based on attack description labels and technique and tactic labels in the network threat intelligence data, with text sentences as units, each tuple in the three-tuple sequence being (Sen, Tac, Tech), Sen being a text sentence, Tac being the tactic label corresponding to the text sentence, and Tech being the technique label corresponding to the text sentence; Step 3.3: extracting key information from the text sentence to obtain a key information four-tuple; Step 3.4: encoding the key information four-tuple using a fast encoding model to obtain an encoding result of the key information four-tuple; wherein the fast encoding model is pre-trained using network security text; Step 3.5: identifying attack description labels in the network threat intelligence text, and segmenting the attack description text into text sentences according to the symbols contained in the text; Step 3.6: extracting key information from the text sentence to obtain a key information four-tuple; Step 3.7: information coding of the key information quadruple by using a fast coding model to obtain the coding result of the key information quadruple.

4. The method of claim 3, wherein, The key information quadruple is obtained by extracting the key information in the text sentence. The prefix modifier and the suffix modifier of the subject and the object entity in the text sentence are expanded by using the syntax analysis technology and the lexical analysis technology to obtain the entity with complete modifiers; And / or, The key information of the supplementary clause in the text sentence is extracted by using the Stanford natural language technology to identify the supplementary clause in the text sentence; And / or, For the text sentence with incomplete information, the key information is extracted by constructing an incomplete tuple; And / or, The IoC detection and identification of the path, file, registry, and command line type are performed by extending the IoCParser tool; And / or, The passive sentence is identified based on the lexical analysis technology and the syntax analysis technology, and the passive sentence is converted into an active sentence for key information extraction.

5. The method of claim 1, wherein, The attack-related traceability log is obtained by matching the coding result of the key information quadruple after the traceability log edge is coded, including: A specific part in the traceability log edge is obtained, and the specific part includes: the type of system call, the process name and number of the system call, the action object of the system call, the command line to which the system call belongs, and the time when the system call occurs; The specific part in the traceability log edge and the specific part in the quadruple information are calculated for similarity after coding, and the similarity of the traceability log edge and the quadruple information is used as the matching score of the log edge and the current quadruple, and the matching score of all quadruples in each technology / tactic and the traceability log edge is used as the threat score of the current technology / tactic and the log; The top specific number of technology labels, tactic labels, and corresponding threat scores with the highest threat scores are retained to generate the attack-related traceability log; On the benign traceability analysis log data set, the entries in the threat intelligence automated exploitation meta-database are matched, and the detection minimum threshold is obtained based on the Gibbs test; When the matching score of the application data set and the entries in the threat intelligence automated exploitation meta-database is higher than the detection minimum threshold, the corresponding information is retained in the attack-related traceability log.

6. The method of claim 5, wherein, The specific part in the traceability log edge and the specific part in the quadruple information are calculated for similarity after coding, including: The specific description of the entity of the operation, the operation relationship, and the specific description of the entity being operated in each key information quadruple in the threat intelligence automated exploitation meta-database are clustered by using a clustering algorithm based on two-band optimization, and a cluster center vector is calculated; The cluster center vector and the information of the traceability log edge are matched by using the vector matching calculation acceleration library in the numpy library and the graphic processor to accelerate the coding vector matching calculation.

7. The method of claim 1, wherein, The attack traceability subgraph is constructed according to the attack-related traceability log, including: The attack-related traceability log is used to generate a set of Steiner trees in a candidate traceability graph by using a traceability graph construction technology based on a Steiner tree optimization algorithm; According to the Steiner tree, an attack traceability subgraph is obtained.

8. The method of claim 1, wherein, The attack traceability subgraph is attributed based on the technical and tactical label semantics of the technical and tactical labels of the traceability log edges, and an alarm APT life cycle chain is generated, including: Based on expert knowledge, the technical and tactical labels of the traceability log edges are mapped and summarized to obtain the APT life cycle phase of each node in the attack traceability subgraph; The APT life cycle phases of the nodes are merged to obtain an attack traceability subgraph with reduced node and edge scales; For the attack traceability subgraph with reduced node and edge scales, a legality check is performed according to the time sequence logical relationship of the APT life cycle phases to obtain an alarm APT life cycle chain that conforms to the APT life cycle theory, and a node threat score is calculated to calculate the threat degree of the alarm APT life cycle chain.

9. The method according to any one of claims 1 to 8, characterized in that, After the attack traceability subgraph is attributed based on the technical and tactical label semantics of the technical and tactical labels of the traceability log edges to generate an alarm APT life cycle chain, the following steps are further included: Based on the attack technical and tactical labels and life cycle phase labels of the alarm APT life cycle chain, a generative artificial intelligence tool is used to generate attack description text and further generate a standardized APT attack report; wherein the attack description text is generated based on the attack technical and tactical labels and life cycle phase labels of the alarm APT life cycle chain using the generative artificial intelligence tool, and a standardized APT attack report is further generated, including: Using an open source threat intelligence dataset to fine-tune the generative artificial intelligence tool in the field of security threat intelligence, the generative artificial intelligence tool includes: GPT, LLaMa, Claude or Qwen; Using prompt words and standardized APT attack reports included in the open source threat intelligence dataset to further fine-tune the generative artificial intelligence tool; For each node in each phase of the alarm APT attack life cycle chain, a prompt word containing information about the node itself and context information is automatically generated using a script, so that the fine-tuned generative artificial intelligence tool generates a behavior semantic interpretation text segment for the node in the phase based on the prompt word, and based on the behavior semantic interpretation text segment of the node and the label of the node, an attack description text for the node in the phase is obtained. According to the attack description text of each node in each phase, a prompt word is used to make the generative artificial intelligence tool generate an overall APT attack report. 10.A system for APT detection and analysis based on network threat intelligence knowledge, characterized in that, The system includes: A dataset acquisition module is configured to acquire an application dataset, the application dataset including: multi-source heterogeneous network threat intelligence data and related system traceability logs; An attack traceability graph construction module is configured to construct an attack traceability graph based on the related system traceability logs, wherein a traceability log node in the attack traceability graph represents a system entity in a system traceability log, and a traceability log edge represents a causal relationship between traceability log nodes; A metadata construction module is configured to construct a threat intelligence automated utilization metadata database composed of coding results of key information quadruples according to the multi-source heterogeneous network threat intelligence data, wherein elements in the key information quadruples include specific descriptions of entities performing operations, operation relationships, specific descriptions of entities being operated, and compromise indicators; An attack-related traceability log generation module is configured to code the traceability log edges and obtain attack-related traceability logs by matching the coding results of the key information quadruples; An attack traceability subgraph construction module is configured to construct attack traceability subgraphs according to the attack-related traceability logs; A detection analysis result generation module is configured to perform attack attribution based on technology and tactic label semantics on attack traceability subgraphs by using technology and tactic labels of traceability log edges, and generate an alarm APT life cycle chain.

Images