Implementation method for network anomaly detection

Abstract

The invention provides a method for realizing network anomaly detection. The method comprises the following steps of: capturing network flow data; the captured data are preprocessed, and effective loads are extracted; extracting semantic features and statistical features from the payload; fusing the extracted features into a comprehensive feature vector; and classifying the comprehensive feature vectors by using a random forest algorithm to identify abnormal traffic. And the classification unit performs anomaly detection on the comprehensive feature vector by using a random forest algorithm. The system has the advantages of improving the detection accuracy, enhancing the generalization ability, improving the processing efficiency, reducing false alarm and missing alarm, and being easy to operate and maintain and high in adaptability. Through real-time monitoring and quick response, the system provides reliable guarantee for network security.

Description

Technical Field

[0001] The present invention relates to the field of network security applications, and more specifically to a method for implementing network anomaly detection. Background Art

[0002] With the rapid development of information technology, especially the popularity of the Internet of Things (IoT), network infrastructure and information security have become increasingly important. Cyber ​​attacks occur frequently, posing a serious threat to personal privacy and corporate security. Traditional network security solutions, such as firewalls and intrusion detection systems, mainly focus on defending against known attack patterns and have limited detection capabilities for new attacks and unknown threats.

[0003] In recent years, anomaly-based network intrusion detection systems (NIDS) have received widespread attention due to their ability to detect new types of attacks. These systems identify potential attacks by analyzing abnormal behaviors in network traffic. However, most existing NIDS focus on analyzing packet header information and ignore valuable information in packet payloads. As attackers increasingly exploit payloads to conduct covert attacks, the limitations of this approach have become apparent.

[0004] In addition, traditional rule-based systems require security experts to manually write rules to detect known attack patterns, which is not only time-consuming but also error-prone. Although machine learning-based systems can automatically learn patterns of network traffic and detect anomalies, they often face the following challenges:

[0005] 1. Limited ability to identify unknown attacks: Machine learning models usually require a large amount of labeled data for training, but samples of unknown attacks are often difficult to obtain.

[0006] 2. Vulnerable to deceptive attacks: Attackers can deceive the model and make it misjudge by constructing specific network traffic.

[0007] 3. Inefficient processing of large-scale network traffic: With the rapid growth of network traffic, how to quickly and accurately process and analyze large amounts of data has become a major challenge.

[0008] Existing deep learning-based network intrusion detection systems have made progress in some aspects, but they usually only consider statistical features when processing network traffic payloads, while ignoring the semantic information in the payload. This leads to some problems, such as difficulty in distinguishing attacks with similar statistical features but different payloads, and difficulty in detecting new attacks hidden in the payload.

[0009] Therefore, existing technologies have certain limitations and deficiencies in extracting and utilizing network traffic payload features. Summary of the invention

[0010] In view of the limitations and shortcomings of the existing technology in extracting and utilizing the network traffic payload features, in order to overcome these limitations and improve the detection capability of new attacks, the present invention proposes a method for realizing network anomaly detection, which is characterized by using an anomaly detection method based on a convolutional neural network and a random forest anomaly detection system, comprising the following steps: capturing network traffic data; preprocessing the captured data to extract the payload; extracting semantic features and statistical features from the payload; fusing the extracted features into a comprehensive feature vector; using a random forest algorithm to classify the comprehensive feature vector to identify abnormal traffic, aiming to make full use of the semantic information in the network traffic payload, and combining statistical features for comprehensive analysis and classification, thereby improving the accuracy and sensitivity of intrusion detection, and specifically proposes to realize it through an anomaly detection system based on a convolutional neural network and a random forest, including:

[0011] a data pre-processing unit for processing network traffic data and extracting payload;

[0012] a feature extraction unit for extracting semantic features and statistical features from the payload;

[0013] a feature fusion unit, used for combining the semantic features and statistical features into a comprehensive feature vector;

[0014] A classification unit is used to receive the comprehensive feature vector and perform anomaly detection classification using a random forest algorithm.

[0015] The implementation method is characterized in that the feature extraction unit comprises:

[0016] A word embedding layer to convert byte data into word vectors;

[0017] A convolutional neural network is used to extract semantic features from the word vectors.

[0018] The implementation method is characterized in that the statistical features include: protocol type, source port, destination port, number of data packets, number of bytes, transmission speed and time interval.

[0019] The implementation method is characterized in that the classification unit includes: a random forest model for processing high-dimensional data and performing accurate classification.

[0020] The implementation method is characterized in that the system can adapt to different network environments and can continuously optimize the accuracy of its anomaly detection through continuous learning.

[0021] The implementation method is characterized in that it also includes a user interface that allows network security experts to define detection parameters and view detection results.

[0022] The anomaly detection system provided by the present invention has the following technical effects:

[0023] 1. Improve detection accuracy: By combining semantic features and statistical features, the system can more accurately identify abnormal traffic, including unknown and new attacks.

[0024] 2. Enhance the generalization ability of the system: Using deep learning technology, the system can automatically learn and adapt to new attack patterns, improving the ability to detect unknown attacks.

[0025] 3. Improve processing efficiency: The system design takes into account the needs of large-scale data processing and can effectively process and analyze large amounts of network traffic data.

[0026] 4. Reduce false positives and false negatives: Through precise feature extraction and advanced classification algorithms, the system significantly reduces false positives and false negatives, and improves the reliability of network security protection.

[0027] 5. Easy to operate and maintain: The system provides an intuitive user interface, which is convenient for operators to monitor and manage, reducing the dependence on professional skills.

[0028] 6. Strong adaptability: The system is flexibly designed to adapt to different network environments and security requirements and has broad application prospects.

[0029] In summary, the present invention not only improves the accuracy and efficiency of network traffic anomaly detection, but also provides a more reliable guarantee for network security through real-time monitoring and rapid response. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the anomaly detection system architecture diagram;

[0031] Figure 2 This is a schematic diagram of the working principle of Skip-Gram;

[0032] Figure 3 This is a schematic diagram of the working principle of convolutional neural network. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The implementation of the present invention involves an anomaly detection system based on convolutional neural networks and random forests. The system uses deep learning and machine learning techniques to monitor network traffic in real time and detect anomalies. The following are the detailed implementation steps:

[0035] 1. System architecture design

[0036] 1.1 System Components:

[0037] The system mainly consists of the following components:

[0038] Data capture module: responsible for capturing real-time traffic data from the network.

[0039] Data preprocessing module: cleans and formats the captured data for subsequent processing.

[0040] Feature extraction module: includes semantic feature extraction based on CNN and statistical feature extraction based on traditional methods.

[0041] Feature fusion module: fuses the extracted semantic features and statistical features into a comprehensive feature vector.

[0042] Classifier module: Use the random forest algorithm to classify the fused feature vector.

[0043] User interface: provides an interface for system operation and monitoring.

[0044] 1.2 System deployment:

[0045] The system is deployed at key nodes of the network, such as behind a firewall or in front of an intrusion detection system, to monitor network traffic in real time. The system can be deployed on a physical server or a virtual machine and configured according to the network size and needs.

[0046] 2. Data capture and preprocessing

[0047] 2.1 Data Capture:

[0048] Use network packet capture tools (such as Wi reshark, Tcpdump, etc.) to capture network traffic. The captured data includes but is not limited to IP packets, TCP / UDP data segments, and application layer data.

[0049] 2.2 Data preprocessing: Data preprocessing steps include:

[0050] Data cleaning: Remove invalid or erroneous data packets.

[0051] Data formatting: Format the data packets into a unified format for easy subsequent processing.

[0052] Payload extraction: Extract the payload from the data packet as input to the deep learning model.

[0053] 3. Feature extraction

[0054] 3.1 Semantic feature extraction: Use convolutional neural network (CNN) to extract semantic features of the payload. The specific steps include:

[0055] First, the payload in the network flow is taken as text data, and word embedding technology is used to convert each byte into a continuous vector representation. Skip-Gram is applied to the byte embedding task here: Step 1, generate the input vector of the central word; Step 2, get the embedding vector of the central word; Step 3, for each surrounding word, generate a score vector, which is then converted to probability. Therefore, a softmax output is obtained, indicating the window size; Step 4. Match the generated probability vector with the true probability, which is the vector of the actual output. The difference between the generated probability and the true probability is the loss function for optimizing the parameters.

[0056] 3.2 Statistical feature extraction: Extract statistical features of network traffic, including: Basic statistical features: such as protocol type, source port, destination port, packet size, etc. Time series features: such as packet transmission speed, time interval, session duration, etc.

[0057] First, some basic features are extracted, such as protocol, source port, and destination port, while IP addresses are not included because they vary in different networks and therefore cannot generalize the characteristics of the attack. Then, some statistical properties are calculated, such as the number of packets, the number of bytes, and the number of TCP flags. After that, some time-based statistics are also extracted, such as transmission speed and the time interval between two packets. These features are key signs for detecting attacks such as probing, DoS, DDoS, scanning, U2R, and U2L, which have unique traffic patterns. That is, key statistical features are extracted from the network flow, including the packet header fields and the statistical properties of the entire flow. These statistical features can provide basic information about the network flow, such as protocol type, number of packets, number of bytes, etc.

[0058] 4. Feature fusion: The semantic features and statistical features extracted by CNN are fused to form a comprehensive feature vector. The fusion method can be a simple splicing or a more complex weighted fusion strategy to highlight important features.

[0059] 5. Classifier Design

[0060] 5.1 Random Forest Model: Use the Random Forest algorithm to build a classifier. Random forest consists of multiple decision trees, and each tree uses a randomly selected subset of features during training to improve the generalization ability of the model and reduce overfitting.

[0061] 5.2 Training and Optimization:

[0062] Training data preparation: Collect labeled normal traffic and abnormal traffic data as training sets.

[0063] Parameter Tuning: Adjust the parameters of random forest like number of trees, maximum depth etc. to get the best performance.

[0064] Cross-validation: Use cross-validation method to evaluate the accuracy and robustness of the model.

[0065] 6. Real-time monitoring and anomaly detection

[0066] The system runs in real time on the network, extracts and classifies the passing data packets, and detects abnormal behavior in real time. Once abnormal traffic is detected, the system will trigger an alarm and take corresponding protective measures, such as blocking the connection and recording logs.

[0067] 7. User Interface. Develop a user-friendly interface that enables cybersecurity experts to:

[0068] Configure system parameters, such as feature extraction method, classifier parameters, etc.

[0069] View real-time monitoring data and historical records.

[0070] Adapt anomaly detection strategies to new security threats.

[0071] 8. System testing and evaluation

[0072] 8.1 Test environment establishment: Simulate various network traffic in a controlled environment, including normal traffic and various attack traffic, to test the performance of the system.

[0073] 8.2 Performance Evaluation:

[0074] Accuracy: The ratio of correctly identified traffic to total traffic.

[0075] Recall rate: The ratio of correctly identified abnormal traffic to actual abnormal traffic.

[0076] Response time: The time from when traffic passes to when detection results are generated.

[0077] 8.3 Security Assessment: Evaluate the system’s performance under different attack scenarios to ensure that it can effectively identify and defend against various network attacks.

[0078] 9. System optimization and iteration: Continuously optimize system performance based on test results and user feedback, including:

[0079] Optimize the feature extraction algorithm to improve the expressiveness of features.

[0080] Adjust classifier parameters to improve classification accuracy.

[0081] Improve the user interface and enhance the user experience.

[0082] 10. Deployment and maintenance

[0083] 10.1 System deployment: Deploy the system in the actual network environment to ensure stable operation of the system.

[0084] 10.2 System Maintenance: Regularly update the system, including software updates, signature library updates, and parameter adjustments to adapt to new network environments and threats.

[0085] Figure 1 This is an anomaly detection system architecture diagram, which shows the different stages of the anomaly detection system architecture. The first is statistical feature extraction. Some key statistical features are extracted from each network flow. These features include the packet header field and the statistical properties of the entire flow. The second is payload feature extraction. Use word embedding to map each byte in the payload to a word vector, and then use a convolutional neural network to extract significant features of the payload. The third is classification through random forest. The statistical features and payload features are concatenated together, and then the random forest algorithm is applied to classify the generated new data set.

[0086] Figure 2 This is a schematic diagram of the working principle of Skip-Gram. The figure records the working principle of Skip-Gram and shows the working principle of Skip-Gram. First, the A-dimensional input vector of the central word is generated in the input layer. Second, the B-dimensional embedding vector of the central word is obtained by learning in the embedding layer. Third, 2m A-dimensional score vectors are generated for the surrounding words and converted into softmax probability outputs, where m represents the window size. Fourth, the generated probability vector is matched with the true probability, and the difference between the generated probability and the true probability is used as the loss function of the optimization parameter.

[0087] Figure 3 This is a schematic diagram of the working principle of convolutional neural networks. The figure records the working principle of convolutional neural networks and shows the working principle of convolutional neural networks. Three types of convolution kernels are used here, and the convolution kernel sizes are 2, 3, and 4 respectively. A total of 6 convolution kernels extract 6 feature vectors, and then 6 higher-level feature vectors are obtained through maximum pooling and connected in series. Finally, a fully connected layer is used to map the concatenated feature vectors to the label domain, and the probability of the text belonging to each category is obtained through the softmax function.

[0088] Detailed improvements to convolutional neural networks:

[0089] ① Statistical feature extraction: Extract key statistical features from each network flow, including the packet header fields and the statistical attributes of the entire flow. The statistical attributes include protocol, source port, destination port, number of URG flags, number of ACK flags, number of PSH flags, number of RST flags, number of SYN flags, number of FIN flags, total number of packets in the flow, total number of bytes in the flow, total number of forward (backward) packets, total number of forward (backward) bytes, minimum length of forward (backward) packets, maximum length of forward (backward) packets, average length of forward (backward) packets, standard deviation length of packets in the forward (backward) direction, flow duration, etc.

[0090]

[0091] ② Payload feature extraction: Map the bytes in the payload of each network flow into word vectors, and use the word embedding technology Skip-Gram to obtain the continuous vector representation of the bytes.

[0092] Step 1: Generate one-hot input vector: For each center word (i.e., one-hot vector for each byte), generate its input vector.

[0093] Step 2: Get the embedding vector of the central word: Use a matrix W (i.e., embedding matrix) to get the embedding vector of the central word.

[0094] Step 3: Generate score vectors for surrounding words: For each central word, generate score vectors for its surrounding words, where is a matrix of another. Convert the score vector to a probability distribution by passing it through a softmax function.

[0095] Step 4: Calculate the loss function: Match the generated probability vector with the true probability, which is the one-hot vector of the actual output.

[0096] The difference between the generated probability and the true probability is calculated as the loss function for optimizing the parameters, using the cross entropy loss function.

[0097]

[0098] ③ Text Convolutional Neural Network (Text-CNN): Text-CNN is used to extract the features of the payload from the word embedding vector of the byte. Text-CNN captures the spatial patterns and important features in the payload through one-dimensional convolution operations and maximum pooling operations.

[0099]

[0100]

[0101] The above disclosure is only one embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment can be implemented, and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. A method for implementing network anomaly detection, characterized in that The following steps are involved: Capture network traffic data; pre-process the captured data to extract the payload; extract semantic features and statistical features from the payload; fuse the extracted features into a comprehensive feature vector; classify the comprehensive feature vector using the random forest algorithm to identify abnormal traffic; also includes the following modules: a data pre-processing unit for processing network traffic data and extracting payload; a feature extraction unit for extracting semantic features and statistical features from the payload; a feature fusion unit, used for combining the semantic features and statistical features into a comprehensive feature vector; A classification unit is used to receive the comprehensive feature vector and perform anomaly detection classification using a random forest algorithm.

2. The implementation method according to claim 1, characterized in that: The feature extraction unit comprises: A word embedding layer to convert byte data into word vectors; A convolutional neural network is used to extract semantic features from the word vectors.

3. The implementation method according to claim 1 or 2, characterized in that: The statistical features include: protocol type, source port, destination port, number of data packets, number of bytes, transmission speed and time interval.

4. The implementation method according to claim 1, characterized in that: The classification unit includes: a random forest model for processing high-dimensional data and performing accurate classification.

5. The implementation method according to claim 1, characterized in that: The system can adapt to different network environments and continuously optimize the accuracy of its anomaly detection through continuous learning.

6. The implementation method according to claim 1, characterized in that: Also included is a user interface that allows network security experts to define detection parameters and view detection results.

Images