A method for fine-grained attack behavior detection in large-scale networks

Abstract

This invention proposes a fine-grained attack behavior detection method for large-scale networks, solving the problem that existing technologies struggle to handle large-scale networks and generate fine-grained alarms for network attacks. The method includes: log acquisition; HST model construction; data classification based on the HST model; log processing based on KG graphs; anomaly node detection based on HST; semantic extraction and behavior summarization based on anomaly nodes; behavior semantic aggregation based on event IDF attributes; and behavior aggregation based on the HCA algorithm. This invention proposes a hierarchical event tree structure and a knowledge graph structure, using KG graphs to capture deep information embedded within the graph. This invention achieves fine-grained attack detection, a low false alarm rate, adaptability to large-scale data detection, and the ability to identify new network threats. This invention helps enterprises and network users better protect data security, contributing to network security protection and creating a more benign and secure network environment.

Description

Technical Field

[0001] This invention belongs to the field of network detection technology, and mainly relates to network abnormal traffic detection. Specifically, it is a fine-grained behavior detection method for attack behavior in large-scale networks, which is used for targeted network protection and improves network security. Background Technology

[0002] With the development of the internet and the increase in internet users, large-scale cybersecurity incidents are on the rise globally, with large-scale and complex attacks occurring frequently. A Capital One report states that the credit card information of 106 million customers was exposed due to unauthorized database access, and a recent Twitter attack resulted in multiple prominent accounts displaying fraudulent information to tens of millions of followers, the scale and audacity of which are considered unprecedented. These advanced persistent attacks are typically carried out by attackers with sophisticated networking capabilities, requiring detection systems to perform source analysis from massive amounts of data, consuming significant network bandwidth and computing resources.

[0003] Early anomaly detection methods relied on label-based strategies, query graphs, and TTP specifications. These solutions identified advanced behaviors through label propagation or graph matching. For example, the MORSE model required experts to traverse system entities and initialize their confidentiality and integrity labels for label propagation. This prior knowledge-dependent approach struggled to address novel threats. Statistical methods for APT anomaly detection typically depended on analyzing the statistical characteristics and patterns of data to identify anomalous behavior. For instance, the CA model used clustering algorithms to group data and detect anomalous points far removed from the normal behavior group. This model also struggled to address new threats. Deep learning-based methods utilized various deep learning (DL) techniques to simulate APT patterns or system behavior and performed APT detection in a classification or anomaly detection manner. For example, KAIROS used an encoder-decoder architecture for DL ​​techniques to learn graph representation learning, learning the information structure in benign graphs to distinguish anomalies. The MAGIC model used masked graph representation learning to model benign system entities and behaviors, effectively extracting deep features and abstracting the structure of the source graph. However, this model produced coarse-grained detection results and incurred extremely high resource consumption on large-scale networks.

[0004] Current intrusion detection methods have the following main drawbacks:

[0005] (1) Coarse-grained: Current mainstream detection methods can usually only detect whether an anomaly has occurred and provide a sub-graph of the abnormal behavior as feedback to the security analyst. It is difficult to detect what the abnormal behavior actually is, and the detection granularity has not yet achieved the ideal effect.

[0006] (2) High false alarm rate: Malicious traffic intrusion detection systems often generate a large number of false alarms, that is, misclassifying legitimate traffic as malicious traffic. This may cause system administrators to overlook real security threats because they cannot distinguish which alerts are genuine threats.

[0007] (3) Difficulty in handling large-scale networks: With the development of the Internet, log auditing tools will capture extremely high levels of log information. Current anomaly detection methods adopt a centralized detection architecture, sending all system audit logs to the server for processing and feedback, resulting in intolerable data transmission, data storage, and computing costs.

[0008] (4) Inability to Adapt to New Threats: Intrusion detection systems typically identify malicious traffic based on known threat patterns or signatures. However, if attackers use new attack techniques or vulnerabilities, the system may fail to identify these new threats, resulting in missed detections. Simultaneously, attackers may take circumvention measures against known detection rules, preventing the system from correctly identifying the attack. For example, they may modify the characteristics of the attack payload, use encrypted communication, or exploit protocol vulnerabilities to evade detection.

[0009] This invention solves the problems of difficulty in identifying abnormal traffic in large-scale networks, coarse detection granularity, high false alarm rate, and difficulty in adapting to new network threats. Summary of the Invention:

[0010] The purpose of this invention is to address the shortcomings and problems of the prior art by proposing a fine-grained attack behavior detection method for large-scale networks with low false alarm rate and adaptability to new threats.

[0011] This invention is a fine-grained attack behavior detection method for large-scale networks. Its key features include: collecting audit logs on the host side, classifying and analyzing the data using the HST model, processing the audit logs using a KG graph, and performing semantic extraction and behavior summarization of anomalous nodes, with the semantic extraction using the TransE model; after semantic extraction and behavior summarization of anomalous nodes, performing behavior semantic aggregation; and extracting behavior signatures on the server side. The server side processes the anomalous node information to generate anomalous behavior signatures and sends them to the host side. The host side and server side collaborate to detect anomalous behavior. The method includes the following steps:

[0012] Training phase: The training phase is conducted on the host machine.

[0013] 1. Log Acquisition: System logs are collected on each host using a log collection tool. All collected system logs are used for training. Each host has a global signature library, which is used to identify anomalies after behavioral semantic aggregation is completed and is initialized to empty.

[0014] 2. Constructing the HST Model: The constructed HST model is a tree structure model. The vertical trunk is divided into three branches: processes, files, and networks. Each of the three branches has its own sub-branches, and each branch represents a choice for data classification. The horizontal layers are connected sequentially to the event type layer, operation layer, process layer, and attribute layer. The HST model uses a multi-level tree to classify audit events according to their attributes. Each layer represents a specific set of attributes for the events of interest. The input to the HST model is the system log stream collected from each host. Before performing data classification based on the HST model, the HST model is first set to an empty tree.

[0015] 3. Data classification based on the HST model: Given the collected system logs, the HST model makes selections at the tree branches based on the system log information, finding a matching node at each level to represent the corresponding attribute value of the node event. If a matching attribute value is found, the matching node is a benign node; if no matching attribute value is found, a new node is created, completing the data classification processing of that system log. This process is iterated until all the system log data streams used for training are processed, completing the data classification based on the HST model.

[0016] 4. KG Graph-Based Log Processing for Triple Encoding: The KG graph is initialized as an empty graph, taking system log data streams classified by the HST model as input. After initialization, the KG graph encodes the classified system log data streams from head to tail, encoding the classified system log data streams into KG... Figure 3 The output of log processing based on KG graphs is an encoded system log data stream in tuple form; at this point, the training phase on the host side is complete, and the detection phase begins.

[0017] Detection phase: First, on the host side, abnormal node detection, abnormal node semantic aggregation, abnormal behavior summary and behavior semantic aggregation are performed using the HST model; then, behavior aggregation based on the HCA algorithm is completed on the server side.

[0018] Host side:

[0019] 5. Anomaly node detection based on the HST model:

[0020] During the detection phase, the anomaly detection based on the HST model has two inputs: one is the system log stream from the host, which is encoded into triples using KG graph processing; the other is the system log data stream processed from the training phase using KG graph processing. The original system log data stream from the host, encoded into triples, is first input into the HST tree for anomaly detection based on the HST model, and the system log data stream processed from the training phase is then input into the HST tree for anomaly detection based on the HST model. Anomaly detection based on the HST model is performed at branch points according to... The system log attributes are used to make selections. For each system log in the system log stream, at a certain branch of the HST model, a choice is made to enter a certain horizontal layer based on the system log attributes. If no choice can be made, the node will be treated as an abnormal system log, and the abnormal node detection for this system log will be completed. If a choice can be made, a horizontal layer will be selected to enter. If the horizontal layer is not the last layer, the selection will continue at the same branch based on the system log attributes, and a horizontal layer will be entered. The above operation is repeated until a choice cannot be made at a certain branch and the node is treated as an abnormal system log or the last layer is reached. The node is then marked as a benign system log, and the abnormal node detection for one original system log is completed.

[0021] After iteratively processing all system log data streams in sequence, all searched abnormal system logs are obtained; the output of the abnormal node detection based on the HST model is the detected abnormal system logs.

[0022] 6. Semantic extraction and behavior summarization based on abnormal nodes: Performed on the host side:

[0023] The input for semantic extraction and behavior summarization based on abnormal nodes is the detected abnormal system logs. For the detected abnormal system logs, two separate detection processes are performed simultaneously: one for semantic extraction of abnormal nodes and the other for behavior summarization of abnormal nodes. Semantic extraction of abnormal nodes takes the abnormal system logs detected by HST as input and uses the TransE embedding model to extract the semantic flow from the abnormal system logs. A loss function L is designed to optimize the TransE model. Behavioral summarization of abnormal nodes takes the abnormal system log flow detected by the HST model and performs an adaptive forward depth-first search (DFS) on the abnormal system log flow. Since DFS is a backward traversal, it cannot obtain information about ancestor nodes. The DFS system log information obtained from the traversal is encoded into a KG subgraph that conforms to the KG triple specification. The semantic flow of the abnormal system logs obtained from semantic extraction and the KG subgraph obtained from behavior summarization are respectively used as the outputs of semantic extraction and behavior summarization of abnormal nodes, and both are handed over to behavior semantic aggregation processing.

[0024] 7. Event-based IDF attribute behavior semantic aggregation: performed on the host side:

[0025] The host maintains a global signature library to store the behavior representations after semantic aggregation. After behavior extraction and semantic summarization based on abnormal nodes, it receives the semantic stream and KG subgraph of abnormal system logs obtained from behavior extraction and semantic summarization as input. Each system log represents a specific event. The next step is to aggregate the semantics of abnormal behavior and use the IDF attribute of the event to give less common system logs more distinguishing ability. Behavioral semantic aggregation quantifies abnormal events by aggregating the semantic stream of the extracted abnormal system logs to obtain the semantic representation stream of behavior instances. Then, the extracted semantic representation stream is compared with the global signature library. If the comparison is successful, the abnormality is identified and the specific information of the abnormality is obtained. If the comparison fails, it means that no such abnormality has been detected before. The semantic representation stream of the abnormal behavior and the KG subgraph are sent to the server for behavior aggregation processing.

[0026] Server-side:

[0027] 8. Behavior aggregation and signature extraction based on HCA algorithm: The server-side behavior aggregation module receives the semantic representation stream of abnormal behavior and KG subgraph from the host as input. The HCA algorithm is used to cluster the semantic representation streams of abnormal behavior, that is, behavior instances with similar abnormal information streams. Each abnormal behavior belongs to its own cluster. HCA iteratively calculates the cosine similarity between the semantic representation streams of abnormal behavior and combines the semantic representation streams of the two closest abnormal behaviors until the maximum similarity is lower than the merging threshold, thus completing the abnormal behavior aggregation.

[0028] Once the anomalous behavior aggregation is complete, the behavior signature of each cluster is extracted based on the representativeness of the behavior instance, and the behavior signature is returned to each host. Each host stores the returned behavior signature in a signature database, which stores a set of global behavior signatures. The next time the host completes the semantic extraction of the anomalous event, it directly compares it with the behavior signature database. If the comparison is successful, malicious behavior can be identified. If the malicious behavior does not exist in the global signature database, the relevant anomalous data is sent to the server for processing. The above process repeats until all the data to be detected has been processed.

[0029] Compared with the prior art, the technical advantages of this invention are:

[0030] Fine-grained attack detection is achieved: This invention maintains a global signature database on the host side and compares the aggregated behavioral semantics with the global signature database during the detection phase, thus achieving fine-grained anomaly detection. It can detect specific abnormal behaviors, rather than detecting anomalies at the graph or edge level.

[0031] Low false alarm rate: The host side of this invention proposes an HST model in the abnormal node detection module, and combines it with the behavior signature sent from the server side. This behavior signature is the marking information of abnormal nodes. After the behavior semantics are aggregated, the anomalies are detected with high accuracy, which improves the efficiency of abnormal node detection and achieves a lower false alarm rate.

[0032] Adaptable to large-scale data network detection: This invention reduces the resource consumption of the host alone in dealing with large-scale network data by delegating the tasks on the host side to the server side, and enables the host to complete anomaly detection under large-scale network data.

[0033] Capable of identifying new network threats: The abnormal node semantic extraction of this invention is based on the neural network TransE. After the behavioral semantics are aggregated, the behavioral signature sent by the server is combined for detection, and new threats are handed over to the server for collaborative processing, thus making it more adaptable to new threats. Attached image description:

[0034] Figure 1 This is an overall flowchart of the present invention;

[0035] Figure 2 This is a flowchart of the training phase of the present invention;

[0036] Figure 3 This is a flowchart of the detection stage of the present invention;

[0037] Figure 4 This is a schematic diagram of the HST model in this invention;

[0038] Figure 5 This is an attached flowchart illustrating the overall process of this invention. Detailed implementation method:

[0039] Example 1:

[0040] Deep learning-based methods utilize various deep learning (DL) techniques to simulate advanced persistent attack (APT) patterns or system behaviors, and perform APT detection through classification or anomaly detection. These methods have advantages in capturing complex patterns and improving detection accuracy, but they also have some challenges. For example, the Kairos model suffers from high computational resource consumption, high false positive rate, coarse detection granularity, and insufficient adaptability to new threats when dealing with complex and large amounts of data. This invention explores and studies these problems and proposes a fine-grained attack behavior detection method for large-scale networks.

[0041] This invention is a fine-grained attack behavior detection method for large-scale networks. The invention collects audit logs on the host side, classifies and analyzes the data using the HST model, processes the audit logs using a KG graph, and performs semantic extraction and behavior summarization of anomalous nodes. The semantic extraction uses the TransE model. After semantic extraction and behavior summarization of anomalous nodes, behavioral semantic aggregation is performed. Behavioral signatures are extracted on the server side. The server processes the anomalous node information, generates anomalous behavior signatures, and sends them to the host side. The host and server sides collaborate to detect anomalous behavior. See [link to relevant documentation]. Figure 1 , Figure 1 This is an overall flowchart of the present invention; it includes the following steps:

[0042] Training Phase: The training phase takes place on the host machine; see [link to training phase]. Figure 2 , Figure 2 This is a flowchart of the training phase of the present invention.

[0043] Log Acquisition: System logs are collected on each host using a log collection tool. All collected system logs are used for training. Each host has a global signature library, which is used to identify anomalies after behavioral semantic aggregation is completed and is initialized to empty.

[0044] The Trusted Computing Foundation (TCB) of the log collection tool includes the underlying operating system, auditing framework, and analysis code for the detection method. It does not consider kernel-level attacks, uses existing system hardening techniques to mitigate any potential compromises to the auditing framework, ensures the integrity of the output data from the auditing framework, and guarantees the integrity of the logs through existing secure tracing systems and anti-tampering log technologies, and detects any malicious interference with the tracing logs.

[0045] 2. Construct the HST model: See Figure 4 , Figure 4 This is a schematic diagram of the HST model. The HST model constructed in this invention is a tree structure model. The vertical trunk is divided into three branches: process, file, and network. Each of the three branches has its own new branches, and each branch represents a choice for data classification. The horizontal layers are connected sequentially to the event type layer, operation layer, process layer, and attribute layer. The HST model uses a multi-layer tree to classify audit events according to their attributes. Each layer represents a specific set of attributes for the events of interest. The input of the HST model is the system log stream collected from each host. The output during the training phase is the system log stream collected from each host, and the output during the detection phase is the abnormal system log stream. Before performing data classification based on the HST model, the HST model is first set to an empty tree.

[0046] 3. Data classification based on the HST model: Given the collected system logs, the HST model makes selections at the tree branches based on the system log information, finding a matching node at each level to represent the corresponding attribute value of the node event. If a matching attribute value is found, the matching node is a benign node; if no matching attribute value is found, a new node is created, completing the data classification processing of that system log. This process is iterated until all the system log data streams used for training are processed, completing the data classification based on the HST model.

[0047] 4. KG Graph-Based Log Processing for Triple Encoding: The KG graph is initialized as an empty graph, taking system log data streams classified by the HST model as input. After initialization, the KG graph encodes the classified system log data streams from head to tail, encoding the classified system log data streams into KG... Figure 3 The output of log processing based on KG graphs is an encoded system log data stream in tuple form; after completing the training phase on the host side, the detection phase begins.

[0048] Testing phase: See Figure 3 , Figure 3 This is a flowchart of the detection phase of the present invention. The present invention first performs abnormal node detection, abnormal node semantic aggregation, abnormal behavior summary and behavior semantic aggregation on the host side using the HST model; then, behavior aggregation based on the HCA algorithm is completed on the server side.

[0049] Host side:

[0050] Anomaly node detection based on the HST model: performed on the host side.

[0051] During the detection phase, the anomaly detection based on the HST model has two inputs: one is the system log stream from the host, which is encoded into triples using KG graph processing; the other is the system log data stream processed from the training phase using KG graph processing. The original system log data stream from the host, encoded into triples, is first input into the HST tree for anomaly detection based on the HST model, and the system log data stream processed from the training phase is then input into the HST tree for anomaly detection based on the HST model. Anomaly detection based on the HST model is performed at branch points according to... The system log attributes are used to select which horizontal layer to enter for each system log in the system log stream at a certain branch of the HST model. If no selection can be made, the node is treated as an abnormal system log, and the abnormal node detection for this system log is completed. If a selection can be made, it enters a horizontal layer. If this horizontal layer is not the last layer, the selection is continued at the same branch based on the system log attributes, and the node enters a horizontal layer. This process is repeated until no selection can be made at a certain branch, and the node is treated as an abnormal system log or the last layer is reached. At this point, the node is marked as a benign system log, and the abnormal node detection for one original system log is completed. All system log data streams are processed sequentially and iteratively to obtain all the searched abnormal system logs. The output of the abnormal node detection based on the HST model is the detected abnormal system logs.

[0052] The complexity of finding a node in the tree is limited to hashing the attribute values ​​at four levels, thus requiring very low computational resources and memory consumption to meet the client's need for low overhead.

[0053] Semantic extraction and behavior summarization based on abnormal nodes: performed on the host side:

[0054] The input for semantic extraction and behavior summarization based on abnormal nodes is the detected abnormal system logs. For the detected abnormal system logs, two separate detection processes are performed simultaneously: one for semantic extraction of abnormal nodes and the other for behavior summarization of abnormal nodes. Semantic extraction of abnormal nodes takes the abnormal system logs detected by HST as input and uses the TransE embedding model to extract the semantic flow from the abnormal system logs. A loss function L is designed to optimize the TransE model. Behavioral summarization of abnormal nodes takes the abnormal system log flow detected by the HST model and performs an adaptive forward depth-first search (DFS) on the abnormal system log flow. Since DFS is a backward traversal, it cannot obtain information about ancestor nodes. The DFS system log information obtained from the traversal is encoded into a KG subgraph that conforms to the KG triple specification. The semantic flow of the abnormal system logs obtained from semantic extraction and the KG subgraph obtained from behavior summarization are respectively used as the outputs of semantic extraction and behavior summarization of abnormal nodes, and both are handed over to behavior semantic aggregation processing.

[0055] Event-based IDF attribute-based behavioral semantic aggregation: performed on the host side:

[0056] The host maintains a global signature database to store the behavior representations after semantic aggregation. After behavior extraction and semantic summarization based on abnormal nodes, it receives the semantic stream and KG subgraph of the abnormal system logs obtained from behavior extraction and semantic summarization as input. Each system log in the semantic stream of the abnormal system logs represents a specific event. To improve the efficiency of behavior semantic aggregation, two types of noise events are defined in the system log behavior:

[0057] Redundant events: In behavioral instances, deleting certain events does not change the result of behavioral semantic aggregation. To identify these redundant events, a framework is built on shadow events, which are file operations whose causal relationships have already been represented by other key events. Specific files that do not introduce explicit information flow in causal analysis, such as temporary files created by many processes during execution to store intermediate results, interact with only a single process during their lifecycle. They do not affect or contribute to behavioral semantic aggregation.

[0058] Secular events: File operations that are performed periodically for a specific purpose. These events include those used for caching file editing history and those used for shell program settings. We classify secular events as noise events because they are associated with system routines.

[0059] After removing these two types of noisy events, the next step is to aggregate the semantics of abnormal behavior. By utilizing the IDF attribute of the event, less common system logs are given more discriminative power. Behavioral semantic aggregation quantifies abnormal events by aggregating the semantic stream of the extracted abnormal system logs to obtain the semantic representation stream of the behavioral instance. The extracted semantic representation stream is then compared with the global signature database. If the comparison is successful, the abnormality is identified and the specific information of the abnormality is obtained. If the comparison fails, it means that this type of abnormality has not been detected before. The semantic representation stream of the abnormal behavior and the KG subgraph are sent to the server for behavior aggregation processing.

[0060] Server-side:

[0061] 8. Behavior aggregation and signature extraction based on HCA algorithm: The server-side behavior aggregation module receives the semantic representation stream of abnormal behavior and KG subgraph from the host as input. The HCA algorithm is used to cluster the semantic representation streams of abnormal behavior, that is, behavior instances with similar abnormal information streams. Each abnormal behavior belongs to its own cluster. HCA iteratively calculates the cosine similarity between the semantic representation streams of abnormal behavior and combines the semantic representation streams of the two closest abnormal behaviors until the maximum similarity is lower than the merging threshold, thus completing the abnormal behavior aggregation.

[0062] Once the anomalous behavior aggregation is complete, the behavior signature of each cluster is extracted based on the representativeness of the behavior instance, and the behavior signature is returned to each host. Each host stores the returned behavior signature in a signature database, which stores a set of global behavior signatures. The next time the host completes the semantic extraction of the anomalous event, it directly compares it with the behavior signature database. If the comparison is successful, malicious behavior can be identified. If the malicious behavior does not exist in the global signature database, the relevant anomalous data is sent to the server for processing. The above process repeats until all the data to be detected has been processed.

[0063] Existing deep learning-based anomaly traffic detection methods suffer from high computational resource consumption, high false positive rates, coarse detection granularity, and insufficient adaptability to new threats when handling complex and large amounts of data. The difficulty in addressing these problems and shortcomings lies in the following: Intrusion detection is a complex field involving knowledge of network protocols, security vulnerabilities, and malicious behavior. Designing and implementing an accurate and efficient intrusion detection system requires a deep understanding of this knowledge and the comprehensive application of various technical means. Security threats are constantly evolving, and attackers are continuously developing new attack techniques and vulnerability exploitation methods. Therefore, intrusion detection systems need to be dynamically adaptable, able to update detection rules and models in a timely manner to address emerging threats. With the continuous expansion of network scale and the increase in data volume, intrusion detection needs to process large-scale data and make accurate decisions in a short time. Therefore, algorithms and technologies for handling large amounts of data are the key and challenging aspects of solving this problem. More importantly, model training relies heavily on suitable datasets. While open-source datasets on the internet are limited, and although log auditing tools can capture large amounts of data, selecting suitable data as training datasets remains a challenging task.

[0064] This invention addresses the above-mentioned problems by collecting and preprocessing audit logs on the host side, classifying and analyzing the data using the HST model, processing the audit logs using KG graphs, and summarizing and aggregating abnormal behaviors. An abnormal node semantic extraction module is based on a neural network model. On the server side, behavioral signatures are extracted, and the server processes the abnormal node information to generate abnormal behavior signatures. This invention is a comprehensive technical solution that aims to accurately identify complex attack behaviors through refined behavioral analysis and multi-layered model construction. It can effectively handle abnormal traffic in large-scale network environments, provide targeted security protection, and improve network security. It has the following advantages:

[0065] Low false alarm rate: By utilizing fine-grained behavioral analysis and multi-dimensional data features, normal and abnormal behaviors can be distinguished more accurately, significantly reducing the false alarm rate.

[0066] Comprehensive behavioral coverage: It covers a variety of kernel objects and system events, enabling the capture of a wider range of system behaviors, thus allowing for more comprehensive detection of potential threats.

[0067] Data-driven model updates: Based on a large amount of collected log data, the detection model can be continuously updated and optimized to maintain the advanced nature and effectiveness of the detection system.

[0068] Dynamic adjustment capability: It can dynamically adjust and optimize based on new abnormal data, improving the ability to identify and adapt to new types of attacks.

[0069] Example 2

[0070] A method for fine-grained attack behavior detection in large-scale networks is similar to that in Embodiment 1. In step 1 of this invention, log acquisition is performed. When performing fine-grained attack behavior detection in large-scale networks, traditional techniques often rely on static signature or rule matching methods. The main drawbacks of these methods include: they are mainly based on known attack features and rules for detection, and cannot identify attack patterns that have never been seen before, making it difficult to deal with new and unknown attack methods; signature or rule matching is easily affected by noise and normal behavior, resulting in a high false positive rate, which is particularly significant in large-scale network environments.

[0071] The log acquisition described in step 1 of this invention involves collecting system logs on each host using a log collection tool. Based on the large amount of collected log data, this invention can continuously update and optimize the detection model, maintaining the advanced nature and effectiveness of the detection system. The system logs cover three kernel objects (process, file, socket) and nine system events (process read / write, file read / write, open / execute, socket send / receive). Furthermore, during the training phase, using healthy system logs effectively avoids false positives and improves the accuracy of the detection model. This invention covers a variety of kernel objects and system events, enabling the capture of a wider range of system behaviors, thus providing a more comprehensive detection of potential threats.

[0072] Example 3

[0073] A method for fine-grained attack behavior detection in large-scale networks is similar to Embodiments 1-2. The construction of the HST model described in step 2 of this invention is described in [reference needed]. Figure 4 The HST model constructed in this invention is a tree model. The main branches include processes, files, and networks. The process branches include start and end branches. The file branches include start, delete, adjust, and process branches. The network branch includes process branches, and the process branches include attribute tag package branches. The horizontal layers of the HST model of this invention are connected from top to bottom as follows: event type layer, operation layer, process layer, and attribute layer.

[0074] Traditional methods for detecting APT anomalies typically rely on analyzing the statistical characteristics and patterns of data to identify anomalous behavior. For example, the CA model uses clustering algorithms to group data and detect anomalies that are far removed from the normal behavior group. However, this approach suffers from a high false positive rate and difficulty in dealing with emerging threats. The HST model constructed in this invention, through a multi-layered structure of event type, operation, process, and attribute layers, can deeply decompose and analyze different types of attack behavior. This detailed layering helps to accurately identify known attacks and also allows for rapid adaptation to and detection of new threats. By recording and analyzing system log attributes in detail, the HST model can better distinguish between normal and anomalous behavior, thereby significantly reducing the false positive rate.

[0075] Example 4

[0076] A method for fine-grained attack behavior detection in large-scale networks is similar to embodiments 1-3. The data classification based on the HST model described in step 3 of this invention involves a given system log (i.e., system logs collected from each host). The HST model makes selections at tree branches based on the system log information, finding a matching node at each layer to represent the corresponding attribute value of the node event. If a matching attribute value is found, the matching node is a benign node; if no matching attribute value is found, a new node is created, completing the processing of one system log. This process iterates continuously for the collected system log stream until all the system log data streams used for training are processed, thus completing the data classification based on the HST model.

[0077] Traditional methods typically employ linear or simple tree structures for data classification. The main drawbacks of these methods include: low processing efficiency, especially in large-scale data environments, leading to excessive latency and resource consumption; difficulty in accurately classifying multi-dimensional and complex log data, resulting in high false positive and false negative rates; inability to effectively adapt to dynamic changes in system behavior; and vulnerability to overlooking new or complex attack methods. This invention classifies log data using a hierarchical branching structure based on the HST model, enabling more efficient processing of large-scale data streams and reducing computational resource consumption. The HST model performs attribute matching at each branch level, improving classification accuracy and enabling a more precise distinction between benign and anomalous system logs.

[0078] Example 5

[0079] A fine-grained attack behavior detection method for large-scale networks is described in embodiments 1-4. Step 4 of this invention involves log processing based on the KG graph to obtain triple encodings. (See also...) Figure 2 , Figure 2 This is a flowchart of the training phase of the present invention. The KG graph of the present invention is a set of semantic triples. Each triple corresponds to a system log. A triple consists of three elements. The three elements of the triple encode the semantic relationship between the head and the tail in the form of head, relation, and tail. The head and tail are any type of system entity. The relation accepts any system operation performed on the tail.

[0080] Existing log processing technologies commonly employ keyword matching and rule detection. Keyword matching relies on specific words or phrases to identify anomalies, making it susceptible to spelling errors and word variations, leading to false positives or false negatives. Rule detection analyzes log data using predefined rules and can detect known attack patterns, but its adaptability to new or unknown threats is limited. These methods generally fail to fully capture the deep semantic information and contextual relationships within log data, making them inadequate in dealing with advanced and complex attacks. This invention addresses this issue with KG graph-based log processing, which represents system logs as a set of semantic triples. Each triple consists of a head, a relation, and a tail, accurately encoding the semantic relationships between the head and tail. This overcomes the lack of deep semantic understanding in traditional methods. KG graphs can capture the rich semantic connections and dynamic changes between different entities in the system, providing higher detection accuracy and adaptability to new attacks compared to traditional static rules and limited keyword matching.

[0081] Example 6

[0082] A method for fine-grained attack behavior detection in large-scale networks is similar to embodiments 1-5. Step 5 of this invention, based on the HST model for anomaly node detection, includes the following steps:

[0083] 5.1 HST Model Branch Selection: In the detection phase, the anomaly detection based on the HST model has two inputs. One input is the system log stream from the host. The raw system log stream from the host is processed using KG graphs to obtain a triplet encoding. The other input is the system log data stream processed using KG graphs, output from the training phase. The raw system log data stream from the host, encoded into triplet form, is first input into the HST tree for anomaly detection based on the HST model. The system log data stream processed using KG graphs, output from the training phase, is then input into the HST tree for further processing. For anomaly node detection in the HST model: Each system log from the two input system log streams is processed in the HST model according to the order of the original system log stream from the host and the system log data stream of KG graph-based log processing output during the training phase, following the main branch, the first branch, the second branch, the third branch, and the fourth branch under the main branch. At a certain branch, a choice is made to enter a certain horizontal layer. If no choice can be made, the node will be regarded as an anomaly system log. If a choice can be made, it will enter a certain horizontal layer. If the horizontal layer is not the last layer, the choice will continue to be made at the branch to enter a certain horizontal layer.

[0084] 5.2 Repeated Branch Selection: Repeat the above operation until a branch cannot be selected and the node is marked as an abnormal system log or the last level is reached, marking the node as a benign system log, and processing one original system log.

[0085] 5.3 Iterative Processing: After processing one raw system log, start processing the next raw system log from the top branch of the HST model. After processing all raw system log streams from the host, obtain all the searched abnormal system logs. All the searched abnormal system logs are output and processed by the semantic extraction and behavior summary of the abnormal nodes.

[0086] Traditional anomaly detection typically relies on rule-based methods and statistical modeling to identify anomalous behavior. The former uses predefined rules, while the latter is based on statistical models; both have limited ability to detect novel attacks. This invention's anomaly detection based on the HST model, through hierarchical analysis of system logs, repeated branch selection, and iterative processing, not only automates the processing of large amounts of data but also captures anomalous behavior patterns more precisely, improving detection accuracy and adaptability. It demonstrates a stronger responsiveness to new and unknown threats.

[0087] Example 7

[0088] A fine-grained attack behavior detection method for large-scale networks is described in the same way as in Examples 1-6. In step 5 of this invention, the selection is made at the branch based on the attributes of the system log. The HST model constructed in this invention focuses on only a specific set of attributes of the system log in each horizontal layer: the event type layer focuses on the event type attribute of the system log, the operation layer focuses on the event operation attribute of the system log, the process layer focuses on the event process attribute of the system log, and the attribute layer focuses on the attribute label of the event described by the system log. At the same time, the complexity of finding nodes for system events is limited to hash checking of the attribute values ​​of the four layers, and the detection of abnormal nodes is transformed into hash lookup of the hierarchical event tree.

[0089] Existing methods for detecting abnormal nodes identify anomalous behavior by manually defining rules, which can easily miss undefined rules or fail to handle complex attacks. Step 5 of this invention selects based on the attributes of the system logs at the branch point, dividing the system log analysis into four horizontal layers. Each layer focuses on only a specific set of attributes of the system logs. This hierarchical and attribute-specific analysis method not only simplifies complexity but also improves search efficiency through hash checking, transforming it into a fast hash lookup of the hierarchical event tree, thus improving detection accuracy and significantly reducing computational costs and false alarm rates.

[0090] Example 8

[0091] A fine-grained attack behavior detection method for large-scale networks is similar to that in Examples 1-7. In step 6 of this invention, based on the semantic extraction and behavior summary of abnormal nodes, the detected abnormal system logs are taken as input and the detection is performed simultaneously in two ways: one is the semantic extraction of abnormal nodes, and the other is the behavior summary of abnormal nodes.

[0092] 6.1 Semantic Extraction of Anomaly Nodes Based on the Neural Network TransE Model: The semantic extraction of anomaly nodes accepts the anomaly system logs obtained from HST detection as input. The TransE embedding model is used to extract the semantic flow from the anomaly system logs. The TransE model optimizes the element vector space by minimizing the transformation distance of triples existing in the KG graph while maximizing the transformation distance of triples not existing in the KG graph. Broken triples are generated by replacing the Head or Tail of triples in the KG graph with random elements, ensuring that the new triples do not exist in the KG graph. The loss function L used to optimize the TransE model is:

[0093]

[0094] Where ||·|| represents the L1-norm distance function, h, r, and t represent the Head, Relation, and Tail of the triple, and e x Representing elements in the embedding space, the TransE model uses γ to distinguish whether a triple is located in the KG graph. The result of TransE is an n×m embedding matrix that maps n-dimensional one-dimensional encoded elements to an m-dimensional embedding space, connects the embeddings of the head, relation, and tail of the constituent elements, and generates a multi-dimensional vector. The output of the TransE model is the semantic stream of the abnormal system logs obtained by semantic extraction.

[0095] 6.2 DFS-based Anomaly Behavior Summary: This approach receives anomaly system logs detected by the HST model as input. An adaptive forward depth-first search (DFS) is performed on the anomaly system log stream. Since DFS is a backward traversal, it cannot obtain ancestor node information. Ancestor node information is obtained by merging one-hop edges reaching system entities during the DFS process. The DFS system log information obtained is encoded into a KG subgraph conforming to the KG triple specification. The semantic stream of the anomaly system logs obtained through semantic extraction and the KG subgraph obtained through behavior summary processing are respectively used as the outputs of anomaly node semantic extraction and behavior summary, both of which are then processed by behavior semantic aggregation.

[0096] Existing methods for semantic extraction and behavior summarization of anomalous nodes, employing only a single processing path, fail to fully capture the complexity and diversity of anomalous behavior. Traditional methods are insufficient in both semantic extraction and behavior summarization, struggling to accurately identify complex anomalous behavior patterns and unable to dynamically adjust and optimize based on new data. This results in limited ability to identify new types of attacks. Furthermore, the DFS algorithm in anomalous behavior summarization, lacking information on ancestor nodes, may miss crucial contextual information, affecting detection accuracy. This invention addresses this issue by processing anomalous system logs through two paths: semantic extraction and behavior summarization, comprehensively capturing the complexity and diversity of anomalous behavior. Semantic extraction of anomalous nodes is based on the TransE neural network model, dynamically adjusted and optimized according to new anomalous data, improving the ability to identify and adapt to new types of attacks. Anomalous behavior summarization uses an improved DFS algorithm, merging one-hop incoming edges reaching system entities to obtain ancestor node information, compensating for the inability to obtain ancestor node information and improving information utilization efficiency.

[0097] Example 9

[0098] A fine-grained attack behavior detection method for large-scale networks is similar to embodiments 1-8. Step 7 of this invention, based on the behavioral semantic aggregation of event-specific IDF attributes, employs term-weighted IDF attributes from information retrieval to impart greater discriminative power to less common system logs. The IDF attribute calculation formula is as follows:

[0099]

[0100] Where e represents a specific audit event, S represents the number of sessions, and S e This represents the number of sessions containing event e. The relative importance of the different events that make up the behavior is determined by the IDF attribute, and the importance of the different events that make up the behavior is ranked.

[0101] Existing behavioral semantic aggregation methods use simple frequency counting to evaluate the importance of events in system logs. These methods cannot effectively distinguish the relative importance of different events, especially when dealing with large-scale and diverse datasets, easily leading to high false positive and false negative rates. The event-based IDF attribute-based behavioral semantic aggregation described in this invention assigns higher weights to infrequent system log events through an IDF attribute formula, reducing false positives caused by infrequent events. This allows the system to more accurately identify important anomalous behavioral events on large-scale datasets, improving detection accuracy.

[0102] Example 10

[0103] A fine-grained attack behavior detection method for large-scale networks is described in the same way as in Examples 1-9. The behavior aggregation and signature extraction based on the HCA algorithm described in step 8 of this invention includes the following steps:

[0104] 8.1 Calculating Behavioral Cluster Similarity: The HCA algorithm is used to cluster the semantic representation streams of anomalous behaviors. The HCA algorithm takes the semantic representation streams of anomalous behaviors as input and iteratively calculates the cosine similarity between behavioral clusters. The cosine similarity S(F) m ,F n The calculation formula is as follows:

[0105]

[0106] Wherein, F m ,F n e is a semantic representation stream of two sequentially flowing behavioral instances participating in clustering. i e j For elements that make up the semantic representation stream of abnormal behavior, calculate the cosine similarity of two sequentially flowing semantic representation streams, and group those with a cosine similarity below a set threshold into one cluster, and those with a cosine similarity above the set threshold into different clusters.

[0107] 8.2 Combinatorial Clustering: Calculate the semantic representation stream of abnormal behavior for all classes obtained from the behavioral clustering similarity, combine each class, and extract the unique semantic representation stream as the representative of the cluster.

[0108] Existing methods for behavior aggregation and signature extraction typically rely on specific single features or indicators to detect anomalous behavior. These methods lack multi-dimensional and multi-perspective comprehensive analysis, which may lead to misjudgment or missed detection of anomalous behavior. They also struggle to capture complex patterns and contextual relationships of anomalous behavior, exhibiting limited pattern representation capabilities, especially when facing complex network attack patterns. The behavior aggregation based on the HCA algorithm described in this invention achieves multi-perspective comprehensive analysis by iteratively calculating the cosine similarity between the semantic representation streams of anomalous behavior and merging the closest anomalous behaviors. This enhances pattern representation capabilities. The HCA algorithm learns from system logs, aggregates similar anomalous behaviors, and updates clustering results, providing a comprehensive analytical perspective, improving the ability to detect complex attacks, reducing manual intervention, and enabling rapid response to emerging threats.

[0109] This invention provides a fine-grained attack behavior detection method for large-scale networks. It achieves fine-grained, low false alarm rate, adaptability to large-scale data and new threats, and abnormal traffic intrusion detection. It can help enterprises and network users better protect data security and create a more benign and secure network environment.

[0110] Example 11

[0111] A fine-grained attack behavior detection method for large-scale networks is similar to Examples 1-10, with the training phase performed on the host side:

[0112] 1. System log retrieval:

[0113] System audit logs are collected on each host using log collection tools (such as Windows ETW, Linux Audit, and CamFlow). This method considers three kernel objects: process, file, and socket; and nine system events: process read / write, file read / write, open / execute, and socket send / receive. All logs collected during the training phase must be from a benign dataset.

[0114] The Trusted Computing Foundation (TCB) used includes the underlying operating system, auditing framework, and analysis code for the detection method. It does not consider kernel-level attacks and assumes that existing system hardening techniques are used to mitigate any potential compromises to the auditing framework, while also assuming that the output data from the auditing framework is intact. Existing secure tracing systems and tamper-proof log technologies ensure log integrity and detect any malicious interference with the tracing logs.

[0115] 2. Data classification based on the HST model

[0116] The HST (Hierarchical System Event Tree) model consists of four layers of nodes: event type, operation, procedure, and attribute. Each layer focuses on a specific set of attributes of the event. Given a system event, HST finds a matching node in each layer to represent the attribute value of the event. If no such value is found, a new node is created.

[0117] For captured system log events, they are categorized according to their nature, and a hierarchical system event tree (HST) is constructed. The HST model consists of four layers of nodes: event type, operation, process, and attribute. Each layer focuses on a specific set of attributes of the event, and the frequency of each event is recorded in the corresponding attribute node. Convergence is considered achieved when the proportion of newly created nodes is less than 0.1% for two consecutive days. Experiments show that most systems converge within 3 days. Furthermore, the complexity of finding nodes for system events is limited to hash checks of the four attribute values, thus requiring very low computational resources and memory consumption, meeting the low-overhead requirements of the client.

[0118] 3. Raw log processing based on KG graph

[0119] To analyze the contextual semantics of events, a unified representation is needed to homogeneously represent heterogeneous events. Using a KG graph, a log-based KG graph is a collection of semantic triples that can capture more relationships (e.g., metafile information, permissions, and owner). Each triple corresponds to an audit event and consists of three elements that encode the semantic relationship between the head and tail in the form of (Head, Relation, Tail). The head and tail can be any type of system entity, the relation can accept any system operation performed on the tail, and the type of the system entity in the triple should be consistent with the type in the audit event.

[0120] The detection phase first completes abnormal node detection, abnormal node semantic aggregation, abnormal behavior summary and behavior semantic aggregation on the host side, and then completes behavior aggregation based on the HCA algorithm on the server side.

[0121] Host side:

[0122] 4. HST-based anomaly node detection

[0123] When the hierarchical system event tree (HST) construction converges, given a system event, the HST is searched to find if there is a node that matches the event's attributes. If no match is found, the event is reported as an alert. In other words, any event not observed during the learning period is considered an anomaly, and the system will detect as many anomalies as possible.

[0124] Existing intrusion detection algorithms suffer from coarse-grainedness, high false alarm rates, and difficulty in handling large-scale networks.

[0125] It has drawbacks such as being unable to adapt to new types of threats.

[0126] 5. Semantic Extraction and Behavior Summarization Based on Anomaly Nodes

[0127] 5.1 Semantic Extraction of Abnormal Nodes Based on the TransE Neural Network Model

[0128] Understanding the semantics of audit events is the first step in abstracting high-level behaviors. A single audit event comprises three elements (Head, Relation, and Tail), each contributing to the event's semantics. Performing semantic analysis at the element level, with the context of each element clearly defined, provides a more detailed view. The semantics of an audit event can be obtained by merging the three constituent elements, and the semantics of a behavioral instance can be obtained by merging the events that define it.

[0129] The translation-based embedding model TransE is used to learn the mapping from elements to the embedding space. This is achieved by transforming each element into a vector, where smaller distances between elements (e.g., L1 / L2 norm distance) represent similar semantics, and larger distances represent opposite semantics. In TransE, the transformations in the embedding space describe the semantic relationship between Head + Relation and Tail. Specifically, the embedding space has the property that, given a triple (Head, Relation, Tail), the position of Tail is the position of Head after a translation of Relation (i.e., Head + Relation ≈ Tail). The guiding principle for choosing TransE is that its translation-based model perfectly matches our understanding of the semantic context of audit events.

[0130] TransE optimizes the element vector space by minimizing the transformation distance of triples present in the KG graph while maximizing the transformation distance of triples not present in the KG graph. It generates corrupted triples by replacing the Head or Tail elements in triples in the KG graph with random elements, ensuring that the new triples do not exist in the KG graph. The loss function used to optimize the embedding model is:

[0131]

[0132] Where ||·|| represents the L1-norm distance function, h, r, and t represent the Head, Relation, and Tail of the triple, and e x Representing elements in the embedding space, TransE uses γ to distinguish whether a triple is located in the KG graph. The result of ransE is an n×m embedding matrix that maps n-dimensional one-dimensional encoded elements to an m-dimensional embedding space. To further infer the semantics of audit events, we concatenate the embeddings of their constituent elements (Head, Relation, and Tail) to generate a multidimensional vector.

[0133] 5.2 Summary of Abnormal Behaviors Based on Depth-First Search

[0134] Anomalous behavior is a sequence of audit events that operate on anomalous event nodes and are interconnected through information flow. To extract a subgraph summarizing instances of anomalous behavior, an adaptive forward depth-first search (DFS) is performed on the session KG based on anomalous nodes. To prevent dependency explosion, heuristics are applied to specify system entities that may trigger dependency explosion (e.g., .bash_history and firefox) as termination conditions in the DFS, thus mitigating their impact. Simultaneously, during graph traversal, constraints are enforced: the timestamp of each subsequent edge must monotonically increase from all preceding edges to ensure correct temporal dependencies. Furthermore, the ancestors of system entities often contain crucial contextual information, which is unavailable in the DFS because they are traversed backwards. Therefore, during graph traversal, one-hop edges reaching system entities are further merged to obtain a more complete anomalous behavior subgraph.

[0135] 6. Event-based IDF attribute behavior semantic aggregation

[0136] After summarizing the behavioral instances, the next step is to extract the semantics of the anomalous behavioral instances. One approach to obtain a semantic representation of a behavioral instance is to sum the vectors of its constituent events. However, this approach is only effective under the assumption that all constituent events contribute equally to the semantics of the behavioral instance. In practice, this assumption typically does not hold because events have different relative importance in reflecting the semantics of the behavior and the impact of noisy events.

[0137] The relative importance of the events:

[0138] Events unrelated to behavior are more prevalent in a session because they recur across different behaviors, while events actually related to behavior occur less frequently. Therefore, event frequency is used as a measure of its importance. Inverse Document Frequency (IDF) is used to determine the importance of a specific event to the overall behavior. IDF, as a term-weighting technique, is widely used in information retrieval. Its principle is to assign more discriminative power to less common terms in documents.

[0139]

[0140] Where e represents a specific audit event, S represents the number of sessions, and S e This represents the number of sessions containing event e, and IDF is used to determine the relative importance of the different events that make up the behavior.

[0141] Noise cancellation events:

[0142] The low-level and verbose nature of audit logs makes noisy events a major challenge affecting the effectiveness of detection. This study primarily investigates two types of noisy events:

[0143] Redundant events: In behavioral instances, deleting certain events does not alter data transmission. To identify these redundant events, a concept is built upon shadow events, which refer to file operations whose causal relationships are already represented by other key events. At a higher level, analysts list specific files that do not introduce explicit information flows in causal analysis. For example, many processes create temporary files during execution to store intermediate results. Because such files interact with only a single process during their lifecycle, they do not affect data transmission or contribute to behavioral abstractions.

[0144] Mundane events: File operations performed periodically for a specific purpose. These include events for file editing history caching (vim, write, .viminfo) and events for shell program settings (bash, read, / etc / profile). We categorize mundane events as noisy events because they are associated with system routines.

[0145] After removing noisy events, the semantic representation of the behavior instance is obtained by aggregating its constituent vectorized events. Each host sends the semantic representation of the abnormal behavior, the KG subgraph of the abnormal behavior, and the overall KG graph containing the abnormal behavior to the server for processing, and then waits for the server's feedback.

[0146] Server-side:

[0147] 7. Behavior aggregation based on HCA algorithm

[0148] Instances are often variations of how high-level behaviors are implemented. In other words, a behavior can be considered as a set of similar instances. In this way, only a few automatically selected signatures need to be studied for behavior matching, rather than the entire cluster space. Given the vector representations of behavior instances, we use cosine similarity to compute their semantic relationships:

[0149]

[0150] Among them, F m ,F n Two vector representations related to behavioral instances.

[0151] After receiving the data, the server performs the following steps for processing:

[0152] The HCA algorithm is used to cluster similar behavioral instances, with each instance belonging to its own cluster. HCA iteratively calculates the cosine similarity between clusters and combines the two closest clusters until the maximum similarity is below a merging threshold. Centroid links are selected as the criterion for determining cluster similarity. In other words, cluster similarity estimation depends on the centroids (arithmetic mean positions) within the clusters.

[0153] Once a cluster is identified, a behavioral signature for each cluster is extracted based on the representativeness of the instances. The representativeness of each instance in the cluster is quantified by calculating the average similarity between each behavioral instance and other instances. The instance with the highest similarity is selected as the signature, and the behavioral signature is returned to each host. Each host stores the returned behavioral signature in a signature database, which stores a set of global signatures. The next time a host completes semantic extraction of anomalies, it can directly compare the signature with the behavioral signature database. If the comparison is successful, malicious behavior is identified. If the malicious behavior does not exist in the global signature database, the relevant data is sent to the server for processing.

[0154] In summary, this invention proposes a fine-grained behavior detection method for large-scale networks, belonging to the field of network detection technology. It solves the problem that existing technologies struggle to handle large-scale networks and generate fine-grained alarms for network attacks. This invention collects audit logs on the host side, classifies and analyzes the data using the HST model, processes the audit logs using a KG graph, and performs semantic extraction and behavior summarization of abnormal nodes. Semantic extraction uses the TransE model. After semantic extraction and behavior summarization of abnormal nodes, behavioral semantic aggregation is performed. Behavioral signature extraction is performed on the server side. The server processes the abnormal node information to generate abnormal behavior signatures and sends them to the host side. The host and server sides collaborate to detect abnormal behavior. Its implementation includes: 1) system log acquisition; 2) HST model construction; 3) HST model-based data classification; 4) KG graph-based log processing; 5) HST-based abnormal node detection; 6) abnormal node semantic extraction and behavior summarization; 7) event-based IDF attribute-based behavioral semantic aggregation; and 8) HCA algorithm-based behavioral aggregation. This invention proposes a hierarchical event tree structure and a knowledge graph structure, transforming the detection of abnormal nodes into graph traversal. It uses a KG graph to capture deep information embedded within the graph, providing fine-grained alarm information even under large-scale network traffic. This invention achieves fine-grained attack detection, a low false alarm rate, adaptability to large-scale data detection, and the ability to identify new network threats. This invention helps enterprises and network users better protect data security, contributing to network security protection and creating a more robust and secure network environment.

Claims

1. A method for attack fine-grained behavior detection in large-scale network, characterized in that, The collection of the audit log is completed at the host end, the HST model data is classified and analyzed, the audit log is processed by the KG graph, and the semantic extraction of the abnormal node and the behavior summary are completed, the semantic extraction uses the TransE model; after the semantic extraction of the abnormal node and the behavior summary, the behavior semantic aggregation is completed, the behavior signature extraction is completed at the server end, the abnormal node information is processed to generate the abnormal behavior signature and is sent to the host end, the host end and the server end cooperatively complete the detection of the abnormal behavior, including the following steps: Training phase: the training phase is performed at the host end: Step 1: log acquisition: the system log of the host is collected by the log collection tool at each host end, and all the collected system logs are used for training; each host end is provided with a global signature library for identifying abnormalities after the behavior semantic aggregation is completed, and is initialized as empty; Step 2: HST model construction: the HST model constructed is a tree structure model, the vertical trunk is divided into three branches, which are process, file and network, and the three branches have new branches respectively, each branch represents a selection for data classification; the horizontal layer is connected with the event type layer, the operation layer, the flow layer and the attribute layer in turn; the HST model uses a multi-layer tree to classify the audit events according to the attributes thereof; the input of the HST model is the system log stream collected by each host end, before the data classification based on the HST model is performed, the HST model is first set as an empty tree; Step 3: data classification based on the HST model: given the collected system log, the HST model makes a selection at the tree branch according to the system log information, finds a matching node in each layer to represent the corresponding attribute value of the node event, if the corresponding attribute value is found, the matching node is a benign node; if the corresponding attribute value is not found, a new node is created, and the data classification processing of the given collected system log is completed; the iteration is continuously performed until all the system log data streams used for training are processed, and the data classification based on the HST model is completed; Step 4: encoding of the triplets based on the KG graph log processing: the system log data stream classified by the HST model is accepted as the input of the KG graph log processing, the KG graph is initialized as an empty graph, after the initialization, the KG graph starts to encode the classified system log data stream from the head to the tail, encodes the classified system log data stream into the form of the KG graph triplet, and the output of the KG graph log processing is the encoded system log data stream; at this time, the training phase of the host end is completed, and the detection phase is entered; Detection phase: first, the abnormal node detection, abnormal node semantic aggregation, abnormal behavior summary and behavior semantic aggregation are performed at the host end by the HST model; then, the behavior aggregation based on the HCA algorithm is completed at the server end; Host end: Step 5: abnormal node detection based on the HST model: In the detection stage, the abnormal node detection based on the HST model has two inputs. One input is the system log stream from the host end. The original system log stream from the host end is encoded into triples based on the KG graph log processing. The other input is the system log data stream based on the KG graph log processing output in the training stage. The original system log data stream from the host end encoded into triples is first input into the HST tree for abnormal node detection based on the HST model. The system log data stream based on the KG graph log processing output in the training stage is then input into the HST tree for abnormal node detection based on the HST model. The abnormal node detection based on the HST model selects a branch according to the attributes of the system log. Each system log in the system log stream is selected to enter a certain horizontal layer according to the attributes of the system log at a certain branch of the HST model. If the selection cannot be made, the node is treated as an abnormal system log, and the abnormal node detection of the system log is completed. If the selection can be made, the node is selected to enter a certain horizontal layer. If the horizontal layer is not the last layer, the selection is made again according to the attributes of the system log at the branch and the node is selected to enter a certain horizontal layer. The above operation is repeated until the selection cannot be made at a certain branch and the node is treated as an abnormal system log or the last layer is reached, and the node is marked as a benign system log. The abnormal node detection of the original system log is completed. All system log data streams are processed in sequence, and all abnormal system logs searched are obtained. The abnormal node detection based on the HST model outputs the abnormal system logs detected. Step 6: Semantic extraction and behavior summary based on abnormal nodes: performed on the host end. The input of the semantic extraction and behavior summary based on abnormal nodes is the abnormal system logs detected. For the detected abnormal system logs, two separate detections are performed simultaneously. One is the semantic extraction of abnormal nodes, and the other is the behavior summary of abnormal nodes. The semantic extraction of abnormal nodes accepts the abnormal system logs detected by HST as the input of abnormal semantic extraction and uses the embedding model TransE to extract the semantic stream in the abnormal system logs. An optimized loss function L of the TransE model is designed. The behavior summary of abnormal nodes receives the abnormal system log stream detected by the HST model and performs adaptive forward depth-first search DFS on the abnormal system log stream. DFS is a backward traversal and cannot obtain the information of ancestor nodes. The DFS system log information obtained by traversal is encoded into a KG subgraph conforming to the KG triple specification. The abnormal system log semantic stream obtained by semantic extraction and the KG subgraph obtained by behavior summary processing are respectively output as the outputs of abnormal node semantic extraction and behavior summary and are both input into the behavior semantic aggregation processing. Step 7: Behavior semantic aggregation based on the IDF attributes of events: performed on the host end. The host end maintains a global signature library for storing semantic aggregation behavior representation. After the abnormal node-based behavior extraction and semantic summary, the semantic stream and KG subgraph of the abnormal system log obtained by the behavior extraction and semantic summary are received as input. Each system log represents a specific event. The next step is to aggregate the semantic of abnormal behavior. The IDF attribute of the event is used to give less common system logs more distinguishing ability. The behavior semantic aggregation quantifies abnormal events by collecting the semantic stream of the extracted abnormal system log to obtain the semantic representation stream of the behavior instance. The extracted semantic representation stream is compared with the global signature library. If the comparison is successful, the abnormality is identified, and the specific information of the abnormality is obtained. If the comparison fails, it means that the abnormality is not identified. The semantic representation stream of the abnormal behavior and the KG subgraph are sent to the server end for behavior aggregation processing at the same time. The server end: Step 8: Behavior aggregation and signature extraction based on HCA algorithm: The behavior aggregation of the server end accepts the semantic representation stream of the abnormal behavior and the KG subgraph transmitted from the host end as the input of the behavior aggregation module. The HCA algorithm is used to cluster the semantic representation stream of the abnormal behavior, i.e., the behavior instances with similar abnormal information streams. Each abnormal behavior has its own cluster. HCA iteratively calculates the cosine similarity between the semantic representation streams of the abnormal behaviors and combines the two closest semantic representation streams of the abnormal behaviors until the maximum similarity is lower than the merging threshold, and the abnormal behavior aggregation is completed. After the completion of the abnormal behavior aggregation, the behavior signature of each cluster is extracted according to the representativeness of the behavior instance, and the behavior signature is returned to each host end. Each host end stores the returned behavior signature in the signature library. The signature library stores a set of global behavior signatures. The next time the host end completes the semantic extraction of the abnormal event, it directly compares with the behavior signature library. If the comparison is successful, the malicious behavior can be identified. If the malicious behavior does not exist in the global signature library, the related abnormal data is sent to the server for processing. The above procedures are repeated until all the to-be-detected data are processed.

2. The method according to claim 1, wherein, In step 1, the log acquisition is performed by collecting system logs through a log collection tool at each host end. The system logs use three kernel objects: process, file, and socket. The system logs use nine system events: process read, write, file read, write, open, and execute, socket send and receive. The system logs collected in the training stage are all benign.

3. The method of claim 1, wherein the method is characterized by: In step 2, the HST model is constructed. The HST tree model has a main branch including process, file, and network. The process branch includes start and end branches. The file branch includes start, delete, adjust, and process branches. The network branch includes process branch, and the process branch includes attribute label package branch. The HST model has event type layer, operation layer, flow layer, and attribute layer connected in sequence in the horizontal layer.

4. The method of claim 1, wherein the method is characterized by: The KG graph-based log processing in step 4 obtains the encoding of the triplets, the KG graph is set as a set of semantic triplets, each triplet corresponds to a system log, and a triplet is composed of three elements, the three elements of the triplet encode the semantic relationship between the head and the tail in the form of head, relation, and tail, the head and the tail are any type of system entity, and the relation accepts any system operation performed on the tail.

5. The method of claim 1, wherein the method is characterized by: The abnormal node detection based on the HST model in step 5 includes the following steps: 5.1 HST model branch selection: detection stage, the abnormal node detection based on the HST model has two inputs, one input is the system log stream from the host end, and the other input is the system log data stream processed based on the KG graph in the training stage, the original system log stream from the host end is encoded into triplets based on the KG graph log processing, and the original system log data stream from the host end is first input into the HST tree for abnormal node detection based on the HST model, and the system log data stream processed based on the KG graph in the training stage is then input into the HST tree for abnormal node detection based on the HST model; Each system log in the two input system log streams is sequentially selected in the HST model according to the trunk branch, the first branch, the second branch, the third branch, and the fourth branch under the trunk, and a selection into a certain horizontal layer is made at a certain branch, if the selection cannot be made, the node will be an abnormal system log, if the selection can be made, a certain horizontal layer is entered, and if the horizontal layer is not the last layer, the selection is continued at the branch to enter a certain horizontal layer; 5.2 Repeat branch selection: repeat the above operation until a selection cannot be made at a certain branch and the node is marked as an abnormal system log or the last layer is reached, and the node is marked as a benign system log, and one original system log is processed; 5.3 Iterative processing: after processing one original system log, the next original system log is processed from the top branch selection of the HST model, all original system log streams from the host end are processed, all searched abnormal system logs are obtained, and all searched abnormal system logs are output to the abnormal node semantic extraction and behavior summary for processing.

6. The method of claim 1, wherein the method is characterized by: The abnormal node detection based on the HST model in step 5, the HST model only focuses on a certain group of attributes of the system log at each horizontal layer: the event type layer focuses on the event type attribute of the system log, the operation layer focuses on the event operation attribute of the system log, the process layer focuses on the event process attribute of the system log, and the attribute layer focuses on the attribute label of the event described by the system log; at the same time, the complexity of searching for the node of the system event is limited to the hash check of the four attribute values, and the detection of the abnormal node is converted into the hash search of the layered event tree.

7. The method of claim 1, wherein the method is characterized by: The semantic extraction and behavior summary based on the abnormal node in step 6 accepts the detected abnormal system log as input, and simultaneously performs detection in two ways, one for semantic extraction of abnormal nodes, and the other for behavior summary of abnormal nodes; 6.1 Abnormal node semantic extraction based on neural network TransE model: The abnormal node semantic extraction accepts the abnormal system log detected by the HST as input, and uses the embedding model TransE to extract the semantic flow in the abnormal system log. The TransE model optimizes the vector space of elements by minimizing the conversion distance of triples existing in the KG graph and maximizing the conversion distance of triples not existing in the KG graph. The damaged triples are generated by replacing the Head or Tail in the triples in the KG graph with random elements, and ensuring that the new triples do not exist in the KG graph. The loss function L used to optimize the TransE model is: ， wherein, represents the L1-norm distance function, h, r, and t represent the Head, Relation, and Tail of the triple, represents the elements in the embedding space, the TransE model uses γ to distinguish whether the triple is located in the KG graph or not, the result of TransE is an n × m embedding matrix, which maps the n-dimensional one-dimensional coded elements into the m-dimensional embedding space, connects the embeddings of the elements head, relation, and tail, and generates a multi-dimensional vector, the output of the TransE model is the semantic stream of the abnormal system log obtained by semantic extraction; 6.2 Abnormal behavior summary based on DFS: The abnormal behavior summary based on depth-first search receives the abnormal system log detected by the HST model as input, and performs adaptive forward depth-first search DFS on the abnormal system log stream. DFS is a backward traversal and cannot obtain ancestor node information. The ancestor node information is obtained by merging the one-hop inbound edge of the system entity during the DFS graph search. The DFS system log information obtained by traversal is encoded into a KG subgraph conforming to the KG triple specification. The abnormal system log semantic flow obtained by semantic extraction and the KG subgraph obtained by behavior summary processing are respectively output as the outputs of the abnormal node semantic extraction and behavior summary, and are both subjected to behavior semantic aggregation processing.

8. The method of claim 1, wherein the method is characterized by: Step 7 describes the behavior semantic aggregation based on the event-based IDF attribute, which uses system log to assign more discriminative IDF attributes for calculation. The calculation formula is: ， where e represents a certain audit event, S represents the number of sessions, S represents the number of sessions containing event e, the relative importance of different events constituting the behavior is determined by the IDF attribute, and the importance of different events constituting the behavior is sorted.

9. The method of claim 1, wherein the method is characterized by: Step 8 describes the behavior aggregation and signature extraction based on the HCA algorithm, which includes the following steps: 8.1 Compute behavior cluster similarity: use HCA algorithm to cluster the semantic representation streams of abnormal behaviors, the HCA algorithm takes the semantic representation streams of abnormal behaviors as input, iteratively computes the cosine similarity between behavior clusters, the cosine similarity The calculation formula is as follows: ， Wherein, in the formula is the semantic representation flow of two behavior instances participating in clustering in sequence, is the element of the semantic representation flow constituting abnormal behavior, the cosine similarity of two semantic representation flows flowing in sequence is calculated, and the cosine similarity lower than the set threshold is classified into one cluster, and the cosine similarity higher than the set threshold is classified into different clusters. 8.2 Combined clustering: Calculate the abnormal behavior semantic representation flow of all classes obtained in the behavior clustering similarity, combine each class, and extract a unique semantic representation flow as the representative of the class.

Images