Network anomaly detection method and system based on isolation forest fusion zero-positive anomaly detection

Through the isolated forest fusion zero-positive anomaly detection method, the problem of insufficient recognition capabilities for emerging complex attacks and unknown threats in the existing technology is solved, and efficient and flexible network anomaly detection is achieved, adapting to rapidly changing network environments, reducing resource consumption and Improve detection accuracy and response speed.

CN118573487BActive Publication Date: 2024-11-12INFORMATION & COMMNUNICATION BRANCH STATE GRID JIANGXI ELECTRIC POWER CO +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411052561.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-12
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

Existing network anomaly detection methods are difficult to effectively identify emerging complex attacks and unknown threats, and are resource-consuming, lack flexibility and adaptability, and cannot quickly deal with rapidly changing network environments.

Method used

The isolated forest fusion zero-positive anomaly detection method is adopted to identify abnormal points in network traffic through unsupervised learning, combine zero-positive anomaly detection strategy, dynamically adjust the detection strategy and model, real-time monitoring and response are achieved, and the detection model is optimized through continuous learning.

Benefits of technology

It improves the comprehensiveness, accuracy and efficiency of network abnormal behavior detection, reduces system resource consumption, and realizes rapid identification and response to emerging threats, which is suitable for the identification of dynamic network environments and unknown attack types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118573487B_ABST
    Figure CN118573487B_ABST
Patent Text Reader

Abstract

The present invention discloses a network anomaly detection method and system of isolated forest fusion zero-positive anomaly detection, and the method includes the following steps: 1. data collection and preprocessing; 2. feature extraction and analysis; 3. model integration and training; 4. threat level assessment and adaptive strategy formulation; 5. real-time monitoring and dynamic response; 6. model update and continuous learning; 7. performance evaluation and optimization; the system includes a data collection and preprocessing module, a feature extraction and analysis module, a model integration and training module, a threat level assessment and adaptive strategy formulation module, a real-time monitoring and dynamic response module, a model update and continuous learning module, and a performance evaluation and optimization module. The present invention improves the comprehensiveness, accuracy and efficiency of abnormal network behavior detection, reduces system resource consumption, realizes real-time anomaly detection, and ensures that new threats can be quickly discovered and effective measures can be taken.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of network anomaly detection, and in particular relates to a network anomaly detection method and system for an isolation forest fusion zero-positive anomaly detection. Background Art

[0002] In the field of network security, anomaly detection technology is a key link in protecting information systems from unauthorized access and attacks. With the development of technology and the complexity of the network environment, network attackers use increasingly complex and covert means, such as distributed denial of service (DDoS) attacks and advanced persistent threats (APT). These attacks are not only difficult to detect, but may also cause serious economic losses and service interruptions, bringing unprecedented challenges to information system security. Especially in terms of 0day or other unknown attacks, traditional anomaly detection methods are often unable to effectively identify emerging complex attack methods because they mainly rely on known attack patterns and characteristics. These attacks may lead to data leakage, service interruption or even complete system paralysis, causing huge economic losses and reputation damage. Nowadays, anomaly detection has become one of the hot spots in the field of network security research, and the current research status mainly includes the following aspects:

[0003] (1) Rule-based anomaly detection method

[0004] The rule-based anomaly detection method is a method to identify and respond to abnormal behaviors in the network through predefined rules. These rules are usually based on known abnormal patterns and network behavior characteristics, such as specific request patterns, traffic anomalies, or unusual user behaviors. Therefore, by performing abnormal rule matching analysis on network traffic, abnormal behaviors can be detected and identified. The rule-based anomaly detection method usually includes the following steps:

[0005] ① Rule formulation: Analyze known abnormal behaviors and network attack patterns and define a series of detection rules. These rules may include specific traffic patterns, abnormal access frequencies, abnormal request types, etc.

[0006] ② Traffic monitoring and rule comparison: Continuously monitor network traffic and analyze captured data to check whether there are any abnormal behaviors that comply with preset rules; for example, the rules may involve detecting frequent access from a certain IP address or abnormal data transmission volume.

[0007] ③ Abnormal behavior identification and processing: Once behavior that meets the rules is detected, the system will mark it as potential abnormal activity and take predetermined response measures; this may include isolating suspicious traffic, generating security warnings, or conducting further analysis and processing.

[0008] ④ Rule update and maintenance: Regularly review and update detection rules to ensure that they can adapt to emerging abnormal patterns and attack strategies; adjust and optimize rules by learning the latest security threats and network behaviors to improve detection accuracy and reduce false alarm rates.

[0009] (2) Anomaly detection method based on machine learning

[0010] The anomaly detection method based on machine learning is a method that uses machine learning algorithms to identify abnormal behaviors. It is applicable to a wide range of application scenarios. This method learns the characteristics of abnormal behaviors from a large amount of network traffic data and builds a classifier to determine whether the data exhibits abnormal behavior. Its core idea is to achieve automatic detection of abnormal behaviors through steps such as feature extraction, feature selection, classifier training, and classifier testing. Specifically, the anomaly detection method based on machine learning usually includes the following key steps:

[0011] ① Feature extraction: Extract useful features from network traffic data, such as the number of requests, request frequency, request type, request data volume, etc.

[0012] ②Feature selection: Through the feature selection algorithm, the most discriminative features are screened out.

[0013] ③Dataset construction: The filtered features and corresponding labels (normal or abnormal) are combined into a dataset for training and testing classifiers.

[0014] ④Classifier training: Use machine learning algorithms to train the data set to build an effective classifier.

[0015] ⑤Classifier testing: Use the test dataset to evaluate the performance and accuracy of the classifier.

[0016] (3) Anomaly detection method based on deep learning

[0017] The anomaly detection method based on deep learning is a method that uses deep learning algorithms such as deep neural networks (DNN) and convolutional neural networks (CNN) to learn the characteristics of abnormal behavior from a large amount of network traffic data and build a model to determine whether the data exhibits abnormal behavior. The anomaly detection method based on deep learning usually includes the following key steps:

[0018] ① Data preprocessing: preprocess the data, including data cleaning, standardization, sampling, etc.

[0019] ② Feature extraction: Use deep learning algorithms such as deep neural networks or convolutional neural networks to extract useful features from preprocessed data.

[0020] ③Model construction: Based on the extracted features, build a deep learning model, such as a multi-layer perceptron (MLP), convolutional neural network (CNN) or recurrent neural network (RNN).

[0021] ④Model training: Use large-scale labeled data to train the constructed deep learning model.

[0022] ⑤Model testing: Use test datasets to evaluate the performance and accuracy of deep learning models.

[0023] Rule-based anomaly detection methods cannot detect attacks outside the rules and require constant updating of normal behavior models. Machine learning-based anomaly detection methods require a large amount of network traffic data for learning, and the generalization ability of the classifier is limited, which may not be able to effectively respond to new or changing attack patterns. Deep learning-based anomaly detection methods also require a large amount of network traffic data and have high demands on computing resources. In this context, traditional one-size-fits-all anomaly detection methods often lack flexibility and adaptability, and are difficult to effectively respond to rapidly changing threat environments; moreover, a single anomaly detection strategy often leads to excessive consumption of resources and cannot be adaptively adjusted according to the network environment and attack characteristics, which increases the cost and complexity of system operation.

[0024] In summary, the existing anomaly detection methods are no longer sufficient to meet current security needs, and there is an urgent need to provide an innovative solution for the increasingly complex network security environment. Summary of the invention

[0025] The technical problem to be solved by the present invention is to provide a network anomaly detection method that integrates isolated forest and zero-positive anomaly detection in order to address the deficiencies in the above-mentioned prior art, which improves the comprehensiveness, accuracy and efficiency of abnormal network behavior detection, reduces system resource consumption, realizes real-time anomaly detection, and ensures that new threats can be quickly discovered and effective measures can be taken.

[0026] In order to solve the above technical problems, the technical solution adopted by the present invention is: a network anomaly detection method of isolation forest fusion zero-positive anomaly detection, comprising the following steps:

[0027] Step 1: Data collection and preprocessing: Capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data;

[0028] Step 2: Feature extraction and analysis: Deeply explore the key features in network traffic data, including abnormal request frequency, time series patterns, and response data volume;

[0029] Step 3: Model integration and training: Unsupervised training of the isolation forest model to identify potential anomalies in the data, and the use of zero positive anomaly detection strategy to identify and respond to new threats that do not conform to known patterns, and then combining the isolation forest model with zero positive anomaly detection to obtain an integrated detection model;

[0030] Step 4: Threat level assessment and adaptive strategy formulation: Based on the detection results of the integrated detection model, classify the threat level of each detection event, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy;

[0031] Step 5: Real-time monitoring and dynamic response: Conduct real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors;

[0032] Step 6: Model update and continuous learning: The system updates the isolation forest model and zero-positive anomaly detection strategy using the latest network traffic data according to a predetermined period to maintain the sensitivity and accuracy of the integrated detection model;

[0033] Step 7: Performance evaluation and optimization: The system conducts regular comprehensive performance reviews to measure detection accuracy, false alarm rate, and response time, and optimizes and adjusts detection parameters and strategies based on the evaluation results to ensure efficient and stable operation of the entire anomaly detection system.

[0034] In the above-mentioned network anomaly detection method of isolation forest fusion zero-positive anomaly detection, the network traffic data in step 1 includes IP address, port information, traffic type and request frequency; the preprocessing operation includes screening, format conversion and purification processing.

[0035] In the above-mentioned network anomaly detection method of isolation forest fusion zero-positive anomaly detection, the isolation forest algorithm in step 3 recursively divides the data space by randomly selecting data features and segmentation values ​​until each data point is isolated or the tree reaches its depth limit; the anomaly score of the data point to be evaluated is calculated through the constructed forest to determine whether it is an anomaly point; the anomaly score of the isolation forest model is expressed as:

[0036] ,

[0037] in, is the data point to be evaluated, is the sample size, express The path length in the isolation tree, express The expected path length among all isolated trees, The sample size is The average search path length of the binary sorted tree when is expressed as:

[0038] ,

[0039] in, For the harmonic numbers, and , is Euler's constant;

[0040] The closer the anomaly score is to 1, the higher the possibility of being judged as an outlier. When the anomaly score is much smaller than 0.5, it is judged not to be an outlier. When the anomaly scores of sample points are all close to 0.5, it is judged that the sample does not contain obvious outliers.

[0041] The above-mentioned network anomaly detection method of isolated forest fusion zero-positive anomaly detection, the zero-positive anomaly detection in step 3 adopts the method based on statistical analysis The scoring method quantifies the degree of deviation of a data point from the normal behavior pattern. The anomaly score calculation formula for zero positive anomaly detection is:

[0042] ,

[0043] in, is the anomaly score for zero positive anomaly detection, is the observed value, is the mean value of the data set, is the standard deviation;

[0044] The larger the absolute value of , the more likely the observation is abnormal.

[0045] The above-mentioned network anomaly detection method of isolation forest fusion zero positive anomaly detection, in step 3, combines the isolation forest model with zero positive anomaly detection to obtain an integrated detection model expressed as:

[0046] ,

[0047] in, represents the comprehensive score of anomaly detection, Represents the anomaly score of the isolation forest model The weight of Indicates the score of zero positive anomaly detection The weight of =1.

[0048] The above-mentioned network anomaly detection method of isolation forest fusion zero-positive anomaly detection, and The value of is:

[0049] Step A1: Define a total quantity of Key features of;

[0050] Step A2: Check whether each key feature matches a known security threat feature to identify whether it is a known feature or an unknown feature. When a feature exists in the known threat database, it is marked as a known feature and the number of known features is recorded. ; When a feature does not exist in the known threat database, it is marked as an unknown feature and the number of unknown features is recorded as ;

[0051] Step A3: Calculate the ratio of known features to , and dynamically adjust The value of ; Calculate the number ratio of unknown features as , and dynamically adjust The value of ;

[0052] Step A4: For unknown features, once detected, they will be recorded in the known threat database, and the key feature database will be regularly adjusted and updated.

[0053] In the above-mentioned network anomaly detection method of the isolation forest fusion zero-positive anomaly detection, the specific process of classifying the threat level of each detection event according to the detection results of the integrated detection model in step 4 is as follows:

[0054] Step 401: Comprehensively score the anomaly detection From the minimum value of the anomaly detection comprehensive score To the maximum value of the anomaly detection comprehensive score Divided into 5 areas , ),[ , ),[ , ),[ , ),[ , ];in, , , , All are comprehensive scores of anomaly detection The values ​​in are arranged from small to large;

[0055] Step 402: Determine The overall score of anomaly detection for each test In which area is it located? The threat level of this detection; specifically:

[0056] when lie in[ , ) area, determine the The threat level of this detection is normal, indicating no abnormal activities or only safe and expected network behaviors;

[0057] when lie in[ , ) area, determine the The threat level of this detection is suspicious behavior, which means that the behavior deviates from the normal pattern, but there is not enough evidence to indicate that it is a malicious attack;

[0058] when lie in[ , ) area, determine the The threat level of this detection is a mild attack, which means that the attack behavior is clear but the impact is limited, such as an attempted intrusion activity;

[0059] when lie in[ , ) area, determine the The threat level of this detection is medium attack, which means the attack is more serious and may pose a greater threat to system security, such as successful SQL injection;

[0060] when lie in[ , ] area, judge the The threat level of this detection is severe attack, which means a high-risk attack that poses a direct and serious threat to system or data security, such as a large-scale DDoS attack.

[0061] The above-mentioned network anomaly detection method of isolated forest fusion zero-positive anomaly detection, in step A1, a total number of defined groups is When the key features are detected, two detection modes are defined. The first detection mode is the basic detection mode. The key features include URL features, HTTP methods, request body content, access time, source IP address, The value of is 5; the second detection mode is the enhanced detection mode, and the key features include URL features, HTTP methods, request body content, access time, source IP address, Headers request header content, access frequency, data packet length, attack source address and access port; The value of is 10;

[0062] When setting the dual-modal anomaly detection strategy according to the threat level and dynamically adjusting the anomaly detection strategy as described in step 4, lie in[ , ),[ , ) area, adjust the anomaly detection strategy to the basic detection mode; when lie in[ , ),[ , ),[ , ] area, adjust the anomaly detection strategy to enhanced detection mode.

[0063] The present invention also discloses a network anomaly detection system for implementing the above-mentioned network anomaly detection method by integrating isolation forest and zero-positive anomaly detection, comprising:

[0064] Data collection and preprocessing module: used to capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data;

[0065] Feature extraction and analysis module: used to deeply mine the key features in network traffic data, including abnormal request frequency, time series pattern and response data volume, providing data basis for subsequent model training combining isolation forest algorithm and zero positive anomaly detection;

[0066] Model integration and training module: used to train the isolation forest model and the zero positive anomaly detection model, and then combine the isolation forest model with the zero positive anomaly detection model to obtain an integrated detection model;

[0067] Threat level assessment and adaptive strategy formulation module: used to classify the threat level of each detection event according to the detection results of the integrated detection model, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy;

[0068] Real-time monitoring and dynamic response module: used to perform real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors;

[0069] Model update and continuous learning module: used to update the isolation forest model and zero-positive anomaly detection strategy according to the scheduled period using the latest network traffic data to maintain the sensitivity and accuracy of the integrated detection model;

[0070] Performance evaluation and optimization module: used to conduct regular comprehensive performance reviews of the system, measure key indicators such as detection accuracy, false alarm rate, and response time, and optimize and adjust detection parameters and strategies in a timely manner based on the evaluation results to ensure efficient and stable operation of the entire anomaly detection system.

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] 1. The core advantage of the isolation forest algorithm lies in its unsupervised learning ability and linear time complexity, which enables it to quickly process and identify abnormal behaviors in a large amount of network traffic data, and is particularly suitable for real-time analysis of large-scale data; and by combining the zero-positive anomaly detection technology, the ability to identify emerging threats can be further enhanced, because zero-positive anomaly detection does not rely on historical attack data and can identify attack behaviors without pre-defined features; the present invention combines the isolation forest algorithm with zero-positive anomaly detection to propose a new and efficient network anomaly detection technology, which provides a new perspective for the field of network security, especially in the face of the growing threat of 0day attacks; it can effectively identify 0day attacks, other unknown abnormal behaviors and known anomalies, improve the comprehensiveness and accuracy of network abnormal behavior detection, and improve the efficiency of anomaly detection, which is particularly suitable for dynamic network environments and the identification of unknown attack types.

[0073] 2. The present invention combines the isolation forest algorithm with zero positive anomaly detection to enhance the system's ability to identify emerging threats without sacrificing detection speed; it can identify unknown or previously unseen threats in a constantly changing network environment in real time, and can improve the accuracy of zero positive anomaly detection in identifying true anomalies in large amounts of network traffic data by combining the unsupervised characteristics of the isolation forest algorithm with zero positive anomaly detection.

[0074] 3. The present invention proposes a threat level determination mechanism based on data behavior analysis, which realizes a fine-grained assessment of network security threats and provides security analysts with clear threat information, facilitating rapid understanding and response to potential security incidents.

[0075] 4. The present invention realizes a dynamic adaptive mechanism through the dynamic allocation of the weights of the isolation forest model and the weights of the zero-positive anomaly detection, as well as a method for dynamically adjusting the anomaly detection strategy. The mechanism can adjust the detection parameters and strategies based on the changes in the real-time monitored network conditions and traffic behaviors; in this way, the system can effectively improve the recognition accuracy of emerging threats and abnormal behaviors, and ensure the efficient operation of the security system.

[0076] 5. The present invention improves the adaptability of network anomaly detection by continuously learning and updating the detection model, so that the system can adapt to changes in network behavior in a timely manner and dynamically adjust security policies according to changes in network behavior, ensuring the system's rapid response and continuous learning to new threats, optimizing the use of system resources, and reducing system resource consumption.

[0077] 6. The present invention adopts an innovative dual-modal anomaly detection strategy. Under normal circumstances, the system runs in basic detection mode to maintain efficiency and resource economy. Once the threat level is detected to be elevated, the system automatically switches to enhanced detection mode to increase the depth and scope of detection to improve response efficiency. After the threat subsides, the system intelligently switches back to basic detection mode. This adaptive switching mechanism not only improves the ability to respond to network threats, but also effectively balances performance and resource consumption, thereby achieving optimal system performance.

[0078] 7. The present invention realizes real-time anomaly detection, ensuring that new threats can be quickly discovered and effective measures can be taken.

[0079] 8. The present invention not only performs well in solving technical problems, but also provides an effective and flexible solution for network security.

[0080] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a method flow chart of the network anomaly detection method of the present invention that integrates isolation forest and zero-positive anomaly detection. DETAILED DESCRIPTION Example 1

[0082] like Figure 1 As shown, the network anomaly detection method of the isolation forest fusion zero-positive anomaly detection in this embodiment includes the following steps:

[0083] Step 1: Data collection and preprocessing: Capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data;

[0084] In this embodiment, the network traffic data in step one includes key indicators such as IP address, port information, traffic type and request frequency that can characterize the key features of the difference between normal and abnormal behaviors; the preprocessing operations include screening, format conversion and purification processing, aiming to extract effective information that can reflect potential risks, remove redundant and non-critical information, and ensure the quality and efficiency of subsequent analysis.

[0085] During specific implementation, the network traffic data may be derived from public resources or real data collected directly from Web server applications.

[0086] The core of the Isolation Forest algorithm is to use linear time complexity to "isolate" abnormal data points, assuming that these points are easier to identify because they are different. This method does not rely on historical attack data, making it particularly suitable for detecting unknown or previously unseen threats. In addition, the Isolation Forest algorithm is highly computationally efficient and can quickly process large-scale data sets, which is crucial for real-time network traffic analysis. In addition, the Isolation Forest algorithm is relatively insensitive to parameter settings, which allows the algorithm to run stably in a variety of environments without the need for complex tuning processes.

[0087] Step 2: Feature extraction and analysis: Deeply explore the key features in network traffic data, including abnormal request frequency, time series patterns, and response data volume. These features help to clearly distinguish the boundaries between normal and abnormal behaviors, and provide a data basis for subsequent model training combining the isolation forest algorithm with zero positive anomaly detection;

[0088] Step 3: Model integration and training: Unsupervised training of the isolation forest model to identify potential anomalies in the data, and the use of zero positive anomaly detection strategy to identify and respond to new threats that do not conform to known patterns, and then combining the isolation forest model with zero positive anomaly detection to obtain an integrated detection model;

[0089] By combining the Isolation Forest Algorithm with Zero Positive Anomaly Detection, it is possible to identify unknown or previously unseen threats in a constantly changing network environment in real time. The unsupervised nature of the Isolation Forest Algorithm can be used to improve the accuracy of Zero Positive Anomaly Detection in identifying true anomalies in large amounts of network traffic data, and enhance the system's ability to identify new and unknown attack types.

[0090] In this embodiment, the isolation forest algorithm in step 3 recursively divides the data space by randomly selecting data features and segmentation values ​​until each data point is isolated or the tree reaches its depth limit; the anomaly score of the data point to be evaluated is calculated through the constructed forest to determine whether it is an anomaly point; the anomaly score of the isolation forest model is expressed as:

[0091] ,

[0092] in, is the data point to be evaluated, is the sample size, express The path length in the isolation tree, express The expected path length among all isolated trees, The sample size is The average search path length of the binary sorted tree (BST) when , used to sort the data points The expected path length is normalized and expressed as:

[0093] ,

[0094] in, For the harmonic numbers, and , is Euler's constant;

[0095] When implementing it specifically, Approximately equal to 0.57722;

[0096] The closer the anomaly score is to 1, the higher the possibility of being judged as an outlier. When the anomaly score is much smaller than 0.5, it is judged not to be an outlier. When the anomaly scores of sample points are all close to 0.5, it is judged that the sample does not contain obvious outliers.

[0097] Zero-shot Anomaly Detection is an advanced unsupervised learning method that is specifically used to discover unprecedented anomaly patterns that may be related to security threats from a large amount of unlabeled data. It is particularly important in the field of network security. It is mainly used to deal with zero-day attacks, which have not yet been captured by security defense systems due to their novelty and do not have known characteristic signatures or behavior patterns, making it difficult for traditional signature-based protection methods to effectively identify them.

[0098] In practical applications, zero-positive anomaly detection does not rely on training with previous anomaly samples, but rather builds a model that deeply understands normal behavior patterns through deep statistical analysis of network traffic, system behavior, or other related data under normal conditions. The model aims to reveal the inherent structural relationships and typical behavioral characteristics of the data, and once a significant deviation from the normal pattern is encountered, it can be marked as a potential anomaly or attack activity.

[0099] The core challenge is how to accurately distinguish true abnormal behaviors from a large amount of network traffic data, which requires the algorithm to be highly robust and accurate. This involves understanding and extracting the effectiveness of key features, fine-tuning model performance, and balancing the delicate relationship between detection sensitivity and false alarm rate. At the same time, given the dynamic evolution of the network environment, the zero-positive anomaly detection system must also have the ability to self-update and continuously learn in order to adapt to new threat situations in a timely manner.

[0100] In this embodiment, the zero-positive anomaly detection in step 3 adopts a method based on statistical analysis. The scoring (standard score) method quantifies the degree of deviation of a data point from the normal behavior pattern. The anomaly score calculation formula for zero positive anomaly detection is:

[0101] ,

[0102] in, is the anomaly score for zero positive anomaly detection, is the observed value, is the mean value of the data set, is the standard deviation;

[0103] The larger the absolute value of , the more likely the observation is abnormal.

[0104] In this embodiment, the isolation forest model is combined with zero positive anomaly detection in step 3 to obtain an integrated detection model represented as:

[0105] ,

[0106] in, represents the comprehensive score of anomaly detection, Represents the anomaly score of the isolation forest model The weight of Indicates the score of zero positive anomaly detection The weight of =1.

[0107] For example, The value is 10, when the number of known features is 7, the number of unknown features When the number is 3, dynamically adjust The value is 70%, which is adjusted dynamically The value is 30%.

[0108] The above method realizes the dynamic allocation of the weights of the isolation forest model and the weights of zero-positive anomaly detection, which can ensure the accuracy and interpretability of the anomaly detection results. It can not only detect anomalies, but also give a certain degree of explanation for the type of anomaly. For example, the higher the weight of zero-positive anomaly detection, the more likely it is to be a zero-day vulnerability.

[0109] In this embodiment, and The value of is:

[0110] Step A1: Define a total quantity of Key features of;

[0111] Step A2: Check whether each key feature matches a known security threat feature to identify whether it is a known feature or an unknown feature. When a feature exists in the known threat database, it is marked as a known feature and the number of known features is recorded. ; When a feature does not exist in the known threat database, it is marked as an unknown feature and the number of unknown features is recorded as ;

[0112] Step A3: Calculate the ratio of known features to , and dynamically adjust The value of ; Calculate the number ratio of unknown features as , and dynamically adjust The value of ;

[0113] Step A4: For unknown features, once detected, they will be recorded in the known threat database, and the key feature database will be regularly adjusted and updated.

[0114] Step 4: Threat level assessment and adaptive strategy formulation: Based on the detection results of the integrated detection model, classify the threat level of each detection event, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy;

[0115] In this embodiment, the specific process of classifying the threat level of each detection event according to the detection results of the integrated detection model in step 4 is as follows:

[0116] Step 401: Comprehensively score the anomaly detection From the minimum value of the anomaly detection comprehensive score To the maximum value of the anomaly detection comprehensive score Divided into 5 areas , ),[ , ),[ , ),[ , ),[ , ];in, , , , All are comprehensive scores of anomaly detection The values ​​in are arranged from small to large;

[0117] When implementing it specifically, , , , Average distribution in arrive between;

[0118] Step 402: Determine The overall score of anomaly detection for each test In which area is it located? The threat level of this detection; specifically:

[0119] when lie in[ , ) area, that is, When The threat level of this detection is normal, which means that the network traffic and behavior patterns conform to the preset normal behavior model, with no abnormal activities or only safe and expected network behaviors;

[0120] when lie in[ , ) area, that is, When The threat level of this detection is suspicious behavior, which means that the behavior pattern deviates slightly from the normal model, but does not reach the clear attack characteristics; the behavior deviates from the normal pattern, but there is not enough evidence to indicate that it is a malicious attack;

[0121] when lie in[ , ) area, that is, When The threat level of this detection is a mild attack, which means that behaviors with attack characteristics are detected, but the impact on the system or data is small; the attack behavior is clear, but the impact is limited, such as attempted intrusion activities;

[0122] when lie in[ , ) area, that is, When The threat level of this detection is moderate attack, which means that the attack behavior is confirmed and the attack has successfully broken through a certain security line, posing a significant threat to the system; the attack is more serious and may pose a greater threat to system security, such as successful SQL injection;

[0123] when lie in[ , ] area, that is, When The threat level of this detection is severe attack, which means large-scale, high-intensity attack behavior that may cause serious system damage or data leakage; high-risk attack poses a direct and serious threat to system or data security, such as large-scale DDoS attacks.

[0124] Threat level determination is a key part in modern network security systems, especially in complex and dynamic network environments. Accurately identifying and classifying network threat levels is crucial for effectively defending against attacks. The threat level determination method proposed in the present invention monitors and analyzes network activities in real time and divides them into five threat levels: normal, suspicious behavior, mild attack, moderate attack and severe attack. This method can improve the security system's ability to identify various types of attacks, and can also take corresponding defense measures according to different threat levels, thereby improving the security of the entire network.

[0125] The above threat level classification method is expressed in a table as follows:

[0126] Table 1 Threat level classification standards

[0127] ,

[0128] In practice, the system will analyze and evaluate these activities based on the real-time monitored network activity data. Once suspicious or malicious behavior is detected, it will be classified into the corresponding threat level according to the severity of the behavior, and appropriate response measures will be taken. In this way, the system can respond to various network attacks more flexibly and efficiently, thereby greatly improving the overall effectiveness of network security defense.

[0129] In this embodiment, the total number of a group defined in step A1 is When the key features are detected, two detection modes are defined. The first detection mode is the basic detection mode. The key features include URL features, HTTP methods, request body content, access time, source IP address, The value of is 5; the second detection mode is the enhanced detection mode, and the key features include URL features, HTTP methods, request body content, access time, source IP address, Headers request header content, access frequency, data packet length, attack source address and access port; The value of is 10;

[0130] When setting the dual-modal anomaly detection strategy according to the threat level and dynamically adjusting the anomaly detection strategy as described in step 4, lie in[ , ),[ , ) area, adjust the anomaly detection strategy to the basic detection mode; when lie in[ , ),[ , ),[ , ] area, adjust the anomaly detection strategy to enhanced detection mode.

[0131] The present invention adopts an innovative dual-modal anomaly detection strategy. Under normal circumstances, the system runs in basic detection mode to maintain efficiency and resource economy. Once an increased threat level is detected, the system automatically switches to enhanced detection mode to increase the depth and scope of detection to improve response efficiency. After the threat subsides, the system intelligently switches back to basic detection mode. This adaptive switching mechanism not only improves the response capability to network threats, but also effectively balances performance and resource consumption, thereby achieving optimal system performance.

[0132] Step 5: Real-time monitoring and dynamic response: Conduct real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors; this includes generating security alerts, isolating suspicious traffic, etc.

[0133] Step 6: Model update and continuous learning: To ensure that the model can effectively respond to the evolving threat situation, the system updates the isolation forest model and zero positive anomaly detection strategy with the latest network traffic data according to a predetermined period to maintain the sensitivity and accuracy of the integrated detection model;

[0134] By continuously learning and updating the detection model, the adaptability of network anomaly detection is improved, so that the system can adapt to changes in network behavior in a timely manner, improve the response speed, adaptability and accuracy to new threats, and ensure the efficient operation of the security system;

[0135] Step 7: Performance evaluation and optimization: The system conducts regular comprehensive performance reviews to measure key indicators such as detection accuracy, false alarm rate, and response time, and optimizes and adjusts detection parameters and strategies based on the evaluation results to ensure that the entire anomaly detection system operates efficiently and stably. Example 2

[0136] The network anomaly detection system of the isolation forest fusion zero-positive anomaly detection of this embodiment includes:

[0137] Data collection and preprocessing module: used to capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data;

[0138] Feature extraction and analysis module: used to deeply mine key features in network traffic data, including abnormal request frequency, time series pattern and response data volume. These features help to clearly distinguish the boundary between normal and abnormal behaviors, and provide a data basis for subsequent model training combining isolation forest algorithm and zero positive anomaly detection;

[0139] Model integration and training module: used to train the isolation forest model and the zero positive anomaly detection model, and then combine the isolation forest model with the zero positive anomaly detection model to obtain an integrated detection model;

[0140] Threat level assessment and adaptive strategy formulation module: used to classify the threat level of each detection event according to the detection results of the integrated detection model, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy;

[0141] Real-time monitoring and dynamic response module: used to perform real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors;

[0142] Model update and continuous learning module: used to update the isolation forest model and zero-positive anomaly detection strategy according to the scheduled period using the latest network traffic data to maintain the sensitivity and accuracy of the integrated detection model;

[0143] Performance evaluation and optimization module: used to conduct regular comprehensive performance reviews of the system, measure key indicators such as detection accuracy, false alarm rate, and response time, and optimize and adjust detection parameters and strategies in a timely manner based on the evaluation results to ensure efficient and stable operation of the entire anomaly detection system.

[0144] During the specific implementation, a system information display module and a system control module are also set up. In the system information display module, various important information of the adaptive anomaly detection system for threat level determination will be displayed to the user, mainly including: model parameters, historical versions, abnormal event logs, threat level distribution, security alarm customization, adaptive strategy settings and real-time detection accuracy and other system information; the system control module includes: model parameter update, historical version rollback, abnormal event processing, dynamic threat level adjustment, anomaly detection strategy adjustment and dual-mode anomaly detection model setting and other functions; with these control functions, users can easily use this module in the user space to modify important system data at any time.

[0145] In response to rapidly evolving network threats, anomaly detection adaptive technology is crucial to enhancing the response speed and accuracy of network security systems. Faced with increasingly complex network attacks, traditional anomaly detection methods based on fixed rules or preset models are often difficult to adapt to the challenges of new or unknown attacks. Therefore, it is particularly urgent to develop a system that can dynamically adjust the detection strategy according to the real-time changes in the network environment.

[0146] In summary, the present invention proposes a network anomaly detection method and system that integrates an isolated forest and zero-positive anomaly detection. First, the isolated forest algorithm is combined with the zero-positive anomaly detection method, and then the potential threat level in the network traffic is dynamically evaluated to automatically adjust the detection strategy and response mechanism. The present invention can automatically adjust the detection accuracy and resource allocation according to the severity of the threat, thereby reducing the resource consumption of the system, and does not require pre-labeled data. It can improve the recognition ability of new attacks by applying it to real-time network traffic. The adaptive anomaly detection technology of the present invention can not only cope with complex attack threats, but also provide more reliable security protection for cloud computing and container technology. It is expected that this technology can significantly reduce the economic losses and service interruptions caused by network attacks, and provide an innovative solution for the current increasingly complex network security environment.

[0147] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0148] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0151] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A network anomaly detection method based on isolation forest fusion zero-positive anomaly detection, characterized by: The following steps are involved: Step 1: Data collection and preprocessing: Capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data; Step 2: Feature extraction and analysis: Deeply explore the key features in network traffic data, including abnormal request frequency, time series patterns, and response data volume; Step 3: Model integration and training: Unsupervised training of the isolation forest model to identify potential anomalies in the data, and the use of zero positive anomaly detection strategy to identify and respond to new threats that do not conform to known patterns, and then combining the isolation forest model with zero positive anomaly detection to obtain an integrated detection model; Step 4: Threat level assessment and adaptive strategy formulation: According to the detection results of the integrated detection model, the threat level of each detection event is divided, and the dual-modal anomaly detection strategy is set according to the threat level and the anomaly detection strategy is dynamically adjusted; the first detection mode is the basic detection mode, and the second detection mode is the enhanced detection mode; when the dual-modal anomaly detection strategy is set according to the threat level and the anomaly detection strategy is dynamically adjusted in step 4, lie in[ , ),[ , ) area, adjust the anomaly detection strategy to the basic detection mode; when lie in[ , ),[ , ),[ , ] area, adjust the anomaly detection strategy to enhanced detection mode; Step 5: Real-time monitoring and dynamic response: Conduct real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors; Step 6: Model update and continuous learning: The system updates the isolation forest model and zero-positive anomaly detection strategy using the latest network traffic data according to a predetermined period to maintain the sensitivity and accuracy of the integrated detection model; Step 7: Performance evaluation and optimization: The system conducts regular comprehensive performance reviews to measure detection accuracy, false alarm rate, and response time, and optimizes and adjusts detection parameters and strategies based on the evaluation results to ensure efficient and stable operation of the entire anomaly detection system.

2. The network anomaly detection method of Isolation Forest Fusion Zero-Positive Anomaly Detection according to claim 1, characterized in that: The network traffic data in step one includes IP address, port information, traffic type and request frequency; the preprocessing operation includes screening, format conversion and purification.

3. The network anomaly detection method of Isolation Forest Fusion Zero-Positive Anomaly Detection according to claim 1, characterized in that: The isolation forest algorithm in step 3 recursively divides the data space by randomly selecting data features and segmentation values ​​until each data point is isolated or the tree reaches its depth limit; the anomaly score of the data point to be evaluated is calculated through the constructed forest to determine whether it is an anomaly point; the anomaly score of the isolation forest model is expressed as: , in, is the data point to be evaluated, is the sample size, express The path length in the isolation tree, express The expected path length among all isolated trees, The sample size is The average search path length of the binary sorted tree when is expressed as: , in, For the harmonic numbers, and , is Euler's constant; The closer the anomaly score is to 1, the higher the possibility of being judged as an outlier. When the anomaly score is much smaller than 0.5, it is judged not to be an outlier. When the anomaly scores of sample points are all close to 0.5, it is judged that the sample does not contain obvious outliers.

4. The network anomaly detection method of Isolation Forest Fusion Zero-Positive Anomaly Detection according to claim 3, characterized in that: The zero positive anomaly detection described in step 3 adopts the statistical analysis based The scoring method quantifies the degree of deviation of a data point from the normal behavior pattern. The anomaly score calculation formula for zero positive anomaly detection is: , in, is the anomaly score for zero positive anomaly detection, is the observed value, is the mean value of the data set, is the standard deviation; The larger the absolute value of , the more likely the observation is abnormal.

5. The network anomaly detection method of Isolation Forest Fusion Zero-Positive Anomaly Detection according to claim 4, characterized in that: In step 3, the isolation forest model is combined with zero positive anomaly detection to obtain an integrated detection model expressed as: , in, represents the comprehensive score of anomaly detection, Represents the anomaly score of the isolation forest model The weight of Indicates the score of zero positive anomaly detection The weight of =1 .

6. The network anomaly detection method of Isolation Forest Fusion Zero-Positive Anomaly Detection according to claim 5, characterized in that: and The value of is: Step A1: Define a total quantity of Key features of; Step A2: Check whether each key feature matches a known security threat feature to identify whether it is a known feature or an unknown feature. When a feature exists in the known threat database, it is marked as a known feature and the number of known features is recorded. ; When a feature does not exist in the known threat database, it is marked as an unknown feature and the number of unknown features is recorded as ; Step A3: Calculate the ratio of known features to , and dynamically adjust The value of ; Calculate the number ratio of unknown features as , and dynamically adjust The value of ; Step A4: For unknown features, once detected, they will be recorded in the known threat database, and the key feature database will be regularly adjusted and updated.

7. The network anomaly detection method of Isolation Forest Fusion Zero-Positive Anomaly Detection according to claim 6, characterized in that: The specific process of classifying the threat level of each detection event according to the detection results of the integrated detection model described in step 4 is as follows: Step 401: Comprehensively score the anomaly detection From the minimum value of the anomaly detection comprehensive score To the maximum value of the anomaly detection comprehensive score Divided into 5 areas , ),[ , ),[ , ),[ , ),[ , ];in, , 、 、 All are comprehensive scores of anomaly detection The values ​​in are arranged from small to large; Step 402: Determine The overall score of anomaly detection for each test In which area is it located? The threat level of this detection; specifically: when lie in[ , ) area, determine The threat level of this detection is normal, indicating no abnormal activities or only safe and expected network behaviors; when lie in[ , ) area, determine The threat level of this detection is suspicious behavior, which means that the behavior deviates from the normal pattern, but there is not enough evidence to indicate that it is a malicious attack; when lie in[ , ) area, determine The threat level of this detection is a mild attack, which means that the attack behavior is clear, but the impact is limited; when lie in[ , ) area, determine The threat level of this detection is medium attack, which means the attack is relatively serious and may pose a greater threat to system security; when lie in[ , ] area, judge the The threat level of this detection is severe attack, indicating a high-risk attack that poses a direct and serious threat to system or data security.

8. The network anomaly detection method of Isolation Forest Fusion Zero-Positive Anomaly Detection according to claim 7, characterized in that: In step A1, a total quantity is defined as When the key features are detected, two detection modes are defined. The first detection mode is the basic detection mode. The key features include URL features, HTTP methods, request body content, access time, source IP address, The value of is 5; the second detection mode is the enhanced detection mode, and the key features include URL features, HTTP methods, request body content, access time, source IP address, Headers request header content, access frequency, data packet length, attack source address and access port; The value of is 10.

9. A network anomaly detection system that implements the network anomaly detection method according to claim 1 by integrating isolation forest and zero-positive anomaly detection, characterized in that: include: Data collection and preprocessing module: used to capture network traffic data from the network interface in real time and perform preprocessing operations on the network traffic data; Feature extraction and analysis module: used to deeply mine key features in network traffic data, including abnormal request frequency, time series pattern and response data volume, providing data basis for subsequent model training combining isolation forest algorithm and zero positive anomaly detection; Model integration and training module: used to train the isolation forest model and the zero positive anomaly detection model, and then combine the isolation forest model with the zero positive anomaly detection model to obtain an integrated detection model; Threat level assessment and adaptive strategy formulation module: used to classify the threat level of each detection event according to the detection results of the integrated detection model, set the dual-modal anomaly detection strategy according to the threat level, and dynamically adjust the anomaly detection strategy; Real-time monitoring and dynamic response module: used to perform real-time monitoring in the network environment, detect anomalies through integrated detection models, and implement immediate responses to detected abnormal behaviors; Model update and continuous learning module: used to update the isolation forest model and zero positive anomaly detection strategy according to the scheduled period using the latest network traffic data to maintain the sensitivity and accuracy of the integrated detection model; Performance evaluation and optimization module: used to conduct regular comprehensive performance reviews of the system, measure key indicators such as detection accuracy, false alarm rate, and response time, and optimize and adjust detection parameters and strategies in a timely manner based on the evaluation results to ensure efficient and stable operation of the entire anomaly detection system.