Threat detection system and method based on behavior analysis
By installing a log collector on a network node to generate behavior sequences and using deep learning models to learn behavior timing correlation, the problem of APT detection in massive log data is solved, real-time and accurate threat detection and model generalization capabilities are improved.
Patent Information
- Application Number
- CN202510675526.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2025-07-18
AI Technical Summary
The existing technology is difficult to detect and locate advanced persistent threats (APTs) from massive log data in a timely manner, and it is very stressful to directly analyze massive log data, and the training data set lacks behavioral sequence-related data.
By installing a log collector on a network node, using rule matching to generate behavior sequences, and using deep learning models to learn behavior timing correlation for threat detection, combining CNN and LSTM to build a model, and using PCA and random sampling to reduce dimensionality to improve model generalization capabilities.
Real-time detection of network threats is realized, the pressure of analyzing massive log data is reduced, the detection efficiency and accuracy are improved, and the generalization ability of the model is enhanced.
Smart Images
Figure CN120342756A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network security, and more specifically, to a threat detection system and method based on behavior analysis. Background Art
[0002] In the era of rapid development of informatization, security problems in the network environment emerge in an endless stream. Among them, the Advanced Persistent Threat (APT) is difficult to prevent and detect due to its characteristics such as customization, organization, and long latency. In order to timely discover and accurately locate threats in the network environment, there is an urgent need for a method to mine and extract potential threat behaviors from normal log data, and obtain threat detection results through further analysis. Summary of the Invention
[0003] The purpose of the present invention is to overcome the deficiencies of the prior art, and provide a threat detection system and method based on behavior analysis, which realizes the collection of log data of multiple network nodes, can perform behavior analysis on multi-source logs, and detect threats existing in the network environment based on the behavior analysis.
[0004] The purpose of the present invention is achieved through the following solutions: A threat detection system based on behavior analysis includes a log collection module, a data aggregation and storage module, a behavior analysis module, a data processing module, and a threat detection module; The log collection module is used to collect the log data generated by network nodes, and transmit the collected data to the data aggregation and storage module for storage; The data aggregation and storage module is used to collect the data reported by the log collection module, and filter, transform, and reorganize the data to form logs in a unified format for indexing and storage; The behavior analysis module is used to implement rule matching for log data by writing a rule library to form a behavior sequence; The data processing module is used to preprocess the behavior sequence and input it into the threat detection module for detection; The threat detection module is used to input the behavior sequence processed by data into a pre-trained deep learning model for threat detection.
[0005] Further, the log data includes: process logs, registry logs, network connection logs, Powershell execution logs, command line execution logs, driver logs, file operation logs, service logs, system kernel call logs, and image logs.
[0006] Further, the log data generated by the collection network node specifically includes the following sub-steps: collect log data through the event hook mechanism, set a hook at the operating system kernel level to intercept selected types of system calls or events, and form log data based on the intercepted information.
[0007] Further, in the log collection module, there is also a log collector deployment step. The collector installer completes the installation action of the log collection module; during the installation process, the program will be registered as a system service. Whenever the device is powered on, the collector will start and record log data; an initialization configuration file will be generated locally, and the file is written in xml format to define log collection types, filtering conditions, and output formats.
[0008] Further, in the behavior analysis module, the ways to match log data include field matching, regular expression matching, data comparison, and list matching.
[0009] Further, the rule matching of log data is implemented by writing a rule library to form a behavior sequence, which specifically includes the following sub-steps: Perform rule matching on the log data, and abstract the log data into behavior data in combination with the ATT&CK model. The rule library is written in sigma format and classified according to log types. The final behavior data is arranged in chronological order to form a behavior sequence.
[0010] Further, in the threat detection module, the deep learning model is constructed by combining CNN and LSTM; by introducing PCA and random sampling to reduce the dimensionality of feature vectors and improve the anti-noise and generalization capabilities of the model, and at the same time using CNN for feature extraction to improve the running efficiency of the recurrent neural network while ensuring accuracy.
[0011] A threat detection method based on behavior analysis, based on the threat detection system based on behavior analysis described in any one of the above, includes the following steps: When the behavior analysis function is running, first write a rule library and save it in a selected format one; at the same time, add threat behavior labels mapped to the ATT&CK model to the rule file for subsequent classification of threat behaviors; secondly, the program sequentially matches the log data according to log types, field names, and matching conditions. After successful matching, the threat behavior labels in the corresponding rule and the original log will be combined according to the specified fields to form a behavior sequence in the selected format two; During the threat detection runtime, the threat detection module determines whether it is a real threat based on the result obtained by inputting the behavior sequence into the deep learning model; the judgment requires the user to preset a detection threshold, and the value range of the threshold is from 0 to 1. The closer the value is to 1, the stricter the detection is, that is, the more difficult it is to judge as true. If it is judged as true, the reported alarm result includes the associated original log data, the behavior sequence set, and the judgment result. If it is judged as false, the detection module continues to run until the next detection is triggered.
[0012] A threat detection method based on behavior analysis, based on the threat detection system based on behavior analysis described above, performs the following steps: Collect log data by deploying log collectors from various network nodes; then analyze threat behaviors from the original log data through rule matching; and then learn the temporal relationship between threat behaviors through a deep learning model to give a threat detection result.
[0013] Further, the collecting log data by deploying log collectors from various network nodes; then analyzing threat behaviors from the original log data through rule matching; and then learning the temporal relationship between threat behaviors through a deep learning model to give a threat detection result specifically includes the following sub-steps: Step ①: When a network node generates a log, the log collector first determines whether this type of log meets the collection conditions. If it does not meet the conditions, no operation is performed. If it meets the collection conditions, the log is combined into a selected format two according to the specified fields and transmitted to the data aggregation and storage module; Step ②: When the data aggregation and storage module receives the log data, the behavior analysis module will load the corresponding log data and match it according to the pre-written rule library. First, traverse all the matched rule files, parse the file content, and generate a list of dictionaries; then perform basic verification on the rules to ensure that they meet the requirements of the rule format. In addition, standardize the rule unique identifier, matching keyword, and matching value field. Finally, create a rule set, add the list of dictionaries that meet the rule format to the rule set. By matching the log data with the rule set, if the match is successful, generate a behavior sequence according to the specified format; if the match fails, continue to match the next log data until all log data is matched; Step ③: The data processing module performs outlier judgment on the behavior sequence generated according to the parsing result. If there are outliers, correct and fill them; after the judgment is completed, vectorize the behavior sequence data and input it into the threat detection module; Step ④: The threat detection module first determines whether the current working state is training or detection. When it is in the training state, the threat detection module reads the local dataset to train the deep learning model; when it is in the detection state, the threat detection module receives the vectorized data from the data processing module and performs detection through the trained model. The output result of the model is compared with the preset threshold according to the probability given by the normalization function. If it is greater than the threshold, an alarm is issued; if it is less than the threshold, no operation is performed.
[0014] The beneficial effects of the present invention include: (1) The method of the present invention can record and analyze log data in real time, ensuring the real-time nature of threat detection.
[0015] (2) The method of the present invention can more accurately describe the network environment through behavior analysis while reducing the pressure of directly analyzing a large amount of raw log data, improving the detection efficiency.
[0016] (3) The method of the present invention provides a behavioral sequence dataset and a construction process for training deep learning, enabling the model to effectively capture the temporal correlation between behaviors and improving the detection accuracy.
[0017] (4) The method of the present invention improves the generalization ability and operation efficiency of the model by introducing random sampling and convolutional neural network on the basis of the original deep learning model. Description of the Drawings
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0019] Figure 1 It is the overall architecture diagram of the embodiment of the present invention; Figure 2 It is the log collection function flow chart of the embodiment of the present invention; Figure 3 It is the behavior analysis function flow chart of the embodiment of the present invention; Figure 4 It is the threat detection function flow chart of the embodiment of the present invention. Detailed Embodiments
[0020] All the features disclosed in all the embodiments in this specification, or all the steps in the methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or extended and replaced in any way.
[0021] In view of the current situation in the background, the inventors of the present application further discovered the following technical problems: To achieve threat detection based on behavior analysis, the main technical problems to be solved are how to timely discover and locate threats from massive log data. Further technical problems to be solved are as follows: 1) Log data collection and storage problem: To provide data support for threat detection, appropriate methods are needed to collect log data in the network environment.
[0022] 2) Massive log data analysis problem: Since the amount of log data is often huge in the actual environment, it is extremely difficult to directly analyze the original data, and it will also consume more resources.
[0023] 3) Context correlation analysis problem: Although APT attacks will use a variety of methods and tools for attacks, there is a correlation between each step in the attack process. How to use this correlation for threat detection.
[0024] 4) Training dataset construction problem: Existing datasets rarely contain data related to behavior sequences.
[0025] In the concept of the present invention, mainly based on collecting log data in the network environment, obtaining behavior sequences through rule matching, and detecting the temporal correlation of the behavior analysis results through a deep learning model to determine whether there is a network threat. Specifically for the above problems, first, install a log collector on network nodes such as hosts and servers for data collection, and then use the rule matching method to abstract the original log data into behavior sequences for further analysis later, thus avoiding directly analyzing the original data. Then, learn the temporal correlation between behaviors through training a deep learning model for threat detection. Finally, construct a dataset for model training through attack reproduction. At the same time, the attack means used in the attack reproduction are derived from the disclosed APT attack reports, so that the data is more authentic.
[0026] More specifically, as the first aspect of the present invention, a threat detection system based on behavior analysis is provided, as Figure 1 shown, including a log collection module, a data aggregation and storage module, a behavior analysis module, a data processing module, and a threat detection module.
[0027] Log collection module: Used to collect log data generated by hosts, servers, etc., including process logs, registry logs, network connection logs, Powershell execution logs, command line execution logs, driver logs, file operation logs, service logs, system kernel call logs, image logs, and network packets, etc. And transmit the collected data to the data aggregation and storage module for storage.
[0028] It should be noted that log data is collected through the event hooking mechanism. It sets hooks at the operating system kernel level to intercept specific types of system calls or events. For example, when a new process is created, the collector captures this event and records relevant information such as the timestamp, IP, PID, process name, and username.
[0029] In the implementation method of the log collector deployment, the collector installer completes the installation action of the log collection module of this method. During the installation process, the program will be registered as a system service. Whenever the device is powered on, the collector will start and record log data; an initialization configuration file will be generated locally, which is written in XML format and used to define the types of collected logs, filtering conditions, output formats, etc.
[0030] Data aggregation and storage module: Collects the data reported by the log collection module, filters, transforms, and reorganizes the data, and finally forms logs in a unified format for indexing and storage.
[0031] Behavior analysis module: Implements rule matching for log data by writing a rule library to form a behavior sequence. The ways to match log data include field matching, regular expression matching, data comparison, and list matching.
[0032] It should be noted that the generation of the behavior sequence is achieved by performing rule matching on the log data and abstracting the log data into behavior data in combination with ATT&CK (Adversary Tactics, Techniques, and Common Knowledge). The rule library is written in sigma format and classified according to the types of logs. The final behavior data is arranged in chronological order to form a behavior sequence.
[0033] Data processing module: Preprocesses the behavior sequence, including operations such as correction of outliers, vectorization, and normalization, and finally inputs it into the threat detection module for detection.
[0034] Threat detection module: Inputs the behavior sequence that has undergone data processing into a pre-trained deep learning model for threat detection.
[0035] It should be noted that the deep learning model is constructed by combining CNN and LSTM, and the model structure is as Figure 4 . By introducing PCA and random sampling to reduce the dimensionality of the feature vectors and improve the anti-noise and generalization capabilities of the model, and at the same time using CNN for feature extraction to improve the running efficiency of the recurrent neural network while ensuring the accuracy.
[0036] Based on the above system, when the behavior analysis function runs, first, a rule library is compiled and the file is saved in yaml format. At the same time, threat behavior tags mapped to ATT&CK are added to the rule file to facilitate subsequent classification of threat behaviors. Secondly, the program matches the log data in sequence according to the log type, field name, and matching conditions. After successful matching, the threat behavior tags in the corresponding rule and the original log are combined according to the specified fields to form a behavior sequence in json format.
[0037] Based on the above system, when threat detection runs, the threat detection module determines whether it is a real threat based on the result obtained by inputting the behavior sequence into the deep learning model. The judgment requires the user to preset a detection threshold, and the value range of the threshold is from 0 to 1. The closer the value is to 1, the stricter the detection, that is, the more difficult it is to judge as true. If it is judged to be true, the reported alarm result includes the associated original log data, the set of behavior sequences, and the judgment result. If it is judged to be false, the detection module continues to run until the next detection is triggered.
[0038] As the second aspect of the present invention, a threat detection method based on behavior analysis is provided, as Figure 2 、 Figure 3 and Figure 4 shown, mainly including the following processes: Step ① When logs are generated by hosts, servers, etc., the log collector first determines whether this type of log meets the collection conditions. If not, no operation is performed. If it meets the collection conditions, the log is combined into JSON format according to the specified fields and transmitted to the data aggregation and storage module.
[0039] Step ② When the data aggregation and storage module receives the log data, the behavior analysis module loads the corresponding log data and matches it according to the pre-compiled rule library. First, it traverses all the matched rule files, parses the content of the yaml file such as the fields corresponding to the rule id, description, and detection, and generates a list of dictionaries. Then, basic verification is performed on the rules to ensure that they meet the requirements of the Sigma rule format. In addition, some fields (such as the rule unique identifier, matching keyword, matching value, etc.) are standardized, for example, the field names are unified to adapt to different log sources. Finally, a rule set is created, and the list of dictionaries that meet the rule format is added to the rule set. By matching the log data with the rule set, if the matching is successful, a behavior sequence is generated according to the specified format; if the matching fails, the next log data is continued to be matched until all the log data is matched.
[0040] Step ③ The data processing module judges the outliers based on the behavior sequence generated by the parsing result. If any exist, they are corrected and filled. After the judgment is completed, the behavior sequence data is vectorized and input into the threat detection module.
[0041] In step ④, the threat detection module first determines whether the current working state is training or detection. When it is in the training state, the module reads the local data set to train the deep learning model. When it is in the detection state, the module receives the vectorized data from the data processing module and performs detection through the trained model. The output result of the model is compared with a preset threshold according to the probability given by the normalization function (softmax function). If it is greater than the threshold, an alarm is given; if it is less than the threshold, no operation is performed.
[0042] The units involved in the embodiments of the present invention can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation to the units themselves in some cases.
[0043] According to one aspect of the embodiments of the present invention, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above various optional implementation manners.
[0044] As another aspect, the embodiments of the present invention further provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist alone without being assembled into the electronic device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by an electronic device, the electronic device implements the methods described in the above embodiments.
Claims
1. A threat detection system based on behavior analysis, characterized in that, It includes a log collection module, a data aggregation and storage module, a behavior analysis module, a data processing module, and a threat detection module; The log collection module is used to collect the log data generated by network nodes and transmit the collected data to the data aggregation and storage module for storage; The data aggregation and storage module is used to collect the data reported by the log collection module, filter, transform, and reorganize the data to form logs in a unified format for indexing and storage; The behavior analysis module is used to implement rule matching for log data by writing a rule library to form a behavior sequence; The data processing module is used to preprocess the behavior sequence and input it into the threat detection module for detection; The threat detection module is used to input the behavior sequence processed by data into a pre-trained deep learning model for threat detection.
2. The threat detection system based on behavior analysis according to claim 1, wherein The log data includes: process logs, registry logs, network connection logs, Powershell execution logs, command line execution logs, driver logs, file operation logs, service logs, system kernel call logs, and image logs.
3. The threat detection system based on behavior analysis according to claim 1, wherein The collection of the log data generated by network nodes specifically includes sub-steps: collecting log data through an event hook mechanism, setting a hook at the operating system kernel level to intercept selected types of system calls or events, and forming log data according to the intercepted information.
4. The threat detection system based on behavior analysis according to claim 1, wherein In the log collection module, there is also a log collector deployment step. The collector installer completes the installation of the log collection module; during the installation process, the program will be registered as a system service. Whenever the device is powered on, the collector will start and record log data; an initialization configuration file will be generated locally, and the file is written in xml format to define log collection types, filtering conditions, and output formats.
5. The threat detection system based on behavior analysis according to claim 1, wherein In the behavior analysis module, the methods of matching log data include field matching, regular expression matching, data comparison, and list matching.
6. The threat detection system based on behavior analysis according to claim 1, wherein The implementation of rule matching for log data by writing a rule library to form a behavior sequence specifically includes sub-steps: Performing rule matching on log data, abstracting the log data into behavior data in combination with the ATT&CK model. The rule library is written in sigma format and classified according to log types. The final behavior data is arranged in chronological order to form a behavior sequence.
7. The threat detection system based on behavior analysis according to claim 1, wherein In the threat detection module, the deep learning model is constructed by combining CNN and LSTM; by introducing PCA and random sampling to reduce the dimensionality of feature vectors and improve the anti-noise and generalization abilities of the model, and at the same time using CNN for feature extraction to improve the running efficiency of the recurrent neural network while ensuring accuracy.
8. A threat detection method based on behavior analysis, characterized in that, Based on the threat detection system based on behavior analysis according to any one of claims 1 to 7, it includes the following steps: When the behavior analysis function is running, first write the rule library and save it in the selected format 1; at the same time, add threat behavior tags mapped to the ATT&CK model to the rule file for subsequent classification of threat behaviors; secondly, the program sequentially matches the log data according to the log type, field name, and matching conditions. After successful matching, the threat behavior tags in the corresponding rule and the original log will be combined according to the specified fields to form a behavior sequence in the selected format 2. When the threat detection is running, the threat detection module determines whether it is a real threat based on the result obtained by inputting the behavior sequence into the deep learning model; the judgment requires the user to preset a detection threshold, and the value range of the threshold is from 0 to 1. The closer the value is to 1, the stricter the detection is, that is, the more difficult it is to judge as true; if it is judged to be true, the reported alarm result includes the associated original log data, the set of behavior sequences, and the judgment result; if it is judged to be false, the detection module continues to run until the next detection is triggered.
9. A threat detection method based on behavior analysis, characterized in that, The threat detection system based on behavior analysis according to claim 1 performs the following steps: Collect log data by deploying log collectors from various network nodes; then analyze threat behaviors from the original log data through rule matching; and then learn the temporal relationship between threat behaviors through a deep learning model to give a threat detection result.
10. The threat detection method based on behavior analysis according to claim 9, wherein, The step of collecting log data by deploying log collectors from various network nodes; then analyzing threat behaviors from the original log data through rule matching; and then learning the temporal relationship between threat behaviors through a deep learning model to give a threat detection result specifically includes the following sub-steps: Step ①: When a network node generates a log, the log collector first determines whether this type of log meets the collection conditions. If it does not meet the conditions, no operation is performed. If it meets the collection conditions, the log is combined according to the specified fields into the selected format 2 and transmitted to the data aggregation and storage module. Step ②: When the data aggregation and storage module receives the log data, the behavior analysis module will load the corresponding log data and match it according to the pre-written rule library. First, traverse all the matching rule files, parse the file content, and generate a list of dictionaries; then perform basic verification on the rules to ensure that they meet the requirements of the rule format; in addition, standardize the rule unique identifier, matching keyword, and matching value field. Finally, create a rule set, add the list of dictionaries that meet the rule format to the rule set. By matching the log data with the rule set, if the match is successful, a behavior sequence is generated according to the specified format; if the match fails, continue to match the next log data until all log data is matched. Step ③: The data processing module performs outlier judgment on the behavior sequence generated according to the parsing result. If there are outliers, correct and fill them; after the judgment is completed, vectorize the behavior sequence data and input it into the threat detection module. Step ④: The threat detection module first determines whether the current working state is training or detection. When it is in the training state, the threat detection module reads the local dataset to train the deep learning model; when it is in the detection state, the threat detection module receives the vectorized data from the data processing module and performs detection through the trained model. The output result of the model is compared with the pre-set threshold according to the probability given by the normalization function. If it is greater than the threshold, an alarm is given; if it is less than the threshold, no operation is performed.