A malware detection method based on meta-learning multi-level feature fusion

Abstract

The application provides a multi-level feature fusion malware detection method based on meta learning. First, a network flow data set is divided into flows according to five tuples, and the original byte information of the first N packets of each flow is reserved; bytes in the flow are extracted as flow-level features, and bytes in each packet are extracted as packet-level features; then, MAML is used as a learning method of a multi-level convolutional neural network CNN to extract flow-level spatial features and packet-level spatial features, and the flow-level spatial features and the packet-level spatial features are fused; the fused features and corresponding malware or benign software labels are input into a full connection layer of the CNN to complete learning of a software classification task, and finally, the trained CNN is used for malware detection. The feature extraction of the application combines the fusion feature extraction of flow-level spatial features and packet-level spatial features, so that the feature expression is more abundant at the network level, and the best detection effect is obtained.

Description

Technical Field

[0001] This invention relates to information security technology, specifically to a multi-level feature fusion malware detection technology based on meta-learning. Background Technology

[0002] Any software that intentionally executes malicious payloads on a victim's machine (computer, mobile device, etc.) is considered malware. Due to the rapid development of the internet, criminals are increasingly committing crimes online. They typically create and distribute malware online to launch cyberattacks against victim devices, and malware has had a significant impact on the real world. Since 1988, security incidents such as computer intrusions have been on the rise. Malware detection primarily involves analyzing samples of malware families such as Trojans, viruses, rootkits, ransomware, and spyware in isolated environments. Various methods are used to understand the attacker's motives, objectives, and the type and function of malware based on their behavior, and models are built for detection.

[0003] Currently, the mainstream malware detection methods are based on machine learning and deep learning. (1) Machine learning methods first extract manually designed features from network traffic and then classify them using traditional machine learning algorithms. MLDriod uses four different machine learning algorithms in parallel: Farthest First Clustering, Y-MLP, DeepLearning Algorithm, and Nonlinear Ensemble Decision Tree Forest Approach. It has achieved a detection rate of 98.8% on more than 500,000 Android applications. Bo Liu extracts features from network traffic and performs feature dimensionality reduction, then uses an improved SVM algorithm to detect malware. Although machine learning methods can achieve good results in some specific scenarios, it is undeniable that current machine learning is data-driven statistical machine learning, which relies heavily on feature engineering and a large amount of prior expert knowledge. (2) Deep learning methods use neural networks to automatically extract features from the raw bytes of malware network traffic, which has greatly promoted the development of malware detection technology. For example, IMCEC uses a CNN architecture to convert network traffic into images to achieve malware detection. The performance improvement of deep learning depends heavily on the size of the dataset; therefore, deep learning typically requires a large amount of data. Insufficient and effective training often leads to overfitting. However, in the internet environment where malware iterates frequently, obtaining large amounts of training data and manually labeling it with real data is a difficult task. Summary of the Invention

[0004] The technical problem to be solved by this invention is to propose a method for detecting malicious network traffic software based on a small sample learning approach.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is a multi-level feature fusion malware detection method based on meta-learning.

[0006] Meta-learning is defined as "learning how to learn." It aims to improve the learning algorithm itself. In basic learning, the basic learning algorithm solves a task defined by the dataset and the objective. In meta-learning, an external algorithm updates an internal learning algorithm, causing the learned model to improve the external objective. Many fundamental methods of meta-learning have been proposed. The applicant chose Finn, who proposed a model-agnostic meta-learning (MAML) strategy. Its idea is to train a set of highly generalizable initialization parameters, allowing the model to achieve rapid convergence with only a small amount of data. The core idea of ​​metric-based meta-learning is similar to the nearest neighbor algorithm and kernel density estimation. This method predicts probabilities on a set of known labels as a weighted sum of sample labels in the support set. The weights are calculated by a kernel function, representing the similarity between two data samples. Snell proposed a prototype network meta-learning framework that utilizes clustering to project the support set onto a metric space. Based on Euclidean distance, it obtains the vector mean and calculates the distance from the test sample to each prototype to achieve classification.

[0007] This invention first acquires a dataset of network traffic from malware and benign software during the data collection phase. Then, the network traffic dataset is divided into flows according to a five-tuple: source IP, source port, destination IP, destination port, and transport layer protocol. The original byte information of the first N packets of each flow is retained. The first M1 bytes of the original information are extracted from each flow as flow-level features, and the first M2 bytes of each packet in the N packets are extracted as packet-level features. Both flow-level and packet-level features are converted into grayscale images. MAML is then used as the learning method for a multi-layered convolutional neural network (CNN). The grayscale images of flow-level and packet-level features are input into the CNN, and the feature extraction part of the CNN extracts the spatial features of flow-level and packet-level respectively. These spatial features are then fused. The fused features, along with the corresponding malware or benign software labels, are input into the fully connected layer of the CNN N to complete the malware and benign software classification task. Finally, the trained CNN N is used for malware detection.

[0008] In order to improve the classification performance of CNNs for MAML-based malware classification tasks, this invention combines stream-level spatial features and packet-level spatial features for feature extraction before the fully connected layers of the CNN. This makes the feature representation richer at the network level and achieves the best detection results.

[0009] Specifically, after dividing the data into flows according to the 5-tuples, if a flow has more than N packets, only the first N packets are retained; if a flow has fewer than N packets, the flow is discarded.

[0010] Specifically, when extracting the first M1 bytes of the original information from the stream, if the number of bytes of the original information in the stream is less than M1 bytes, it is filled with hexadecimal 0x00 bytes; similarly, when extracting the first M2 bytes of each packet, if the number of bytes of the original information in the packet is less than M2 bytes, it is filled with hexadecimal 0x00 bytes.

[0011] The advantages of this invention are that it is suitable for small-sample learning, meaning it can achieve good results without massive amounts of data; and the model trained by meta-learning has strong generalization ability and can be transferred to other fields besides malware detection. Therefore, compared with previous traffic detection methods, it has higher algorithm accuracy and shorter detection time, and can be applied in real-world production environments. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of step 4 in Embodiment 1 of the present invention. Detailed Implementation

[0013] Network traffic refers to the amount of data transmitted between two communication nodes, and its main characteristics are divided into packet-level characteristics, flow-level characteristics, and stream-level characteristics. Packet-level traffic classification focuses on the characteristics of data packets and their arrival process, such as packet size distribution and packet arrival time interval distribution. Flow-level traffic classification focuses on the characteristics of flows and their arrival process, which can be a TCP connection or a UDP stream; a flow is typically defined as a five-tuple consisting of source IP address, source port, destination IP address, destination port, and application protocol. Stream-level traffic classification focuses on host pairs and the application traffic between them.

[0014] A multi-level feature fusion malware detection method based on meta-learning includes the following steps:

[0015] Step 1. Collect malware traffic and normal software traffic to generate a pcap file in the corresponding network packet capture format.

[0016] Specifically, this step is best done by collecting at least 10 different software types.

[0017] Specifically, in step 1, traffic collection tools such as Wireshark or tcpdump are used.

[0018] Furthermore, the traffic data in step 1 can be collected in different environments and platforms, increasing the robustness and cross-platform compatibility of the final identification results.

[0019] Furthermore, step 1 can use publicly available datasets to facilitate comparison of performance with other publicly available methods.

[0020] Step 2. Preprocess the collected traffic. Divide the network traffic dataset into flows according to the five-tuple of source IP, source port, destination IP, destination port, and transport layer protocol, and tag each flow with the corresponding software.

[0021] Furthermore, data preprocessing also includes traffic anonymization and redundant file removal. Specifically, anonymizing the segmented streams involves randomizing MAC and IP addresses to prevent strong correlations between model learning addresses and results, thus making it suitable for wide area networks (WANs) beyond local area network environments. Finally, redundant files are removed, specifically pcap files with identical content, to prevent training bias and overfitting.

[0022] Step 3. Retain the original byte information of the first 6 packets of each stream; extract the first 784 bytes of the original flow-level information and the first 100 bytes of each packet in the packet-level information of each stream, and convert them into 28*28 grayscale images and 10*10 grayscale images respectively.

[0023] Specifically, in step 3, if a flow has more than 6 packets, only the first 6 packets are kept; if a flow has fewer than 6 packets, the flow is discarded. If the number of bytes in the original flow-level information is less than 784 bytes, it is padded with 0x00 bytes; if the number of bytes in the original packet-level information is less than 100 bytes, it is padded with 0x00 bytes.

[0024] Step 4. Use a multi-layered CNN to extract flow-level and packet-level spatial features respectively, and then fuse the features at both levels. Finally, use a fully connected layer to complete the classification task of malware and normal software.

[0025] Step 5. Use MAML as the learning method for Meta Learning to train the model from Step 4 into a generalizable learner model.

[0026] MAML is used as the meta-learning method, and a multi-level convolutional neural network (CNN) is used as the base classifier. In the multi-level CNN, the flow-level CNN extracts spatial features from the flow-level grayscale image, and the packet-level CNN extracts spatial features from the packet-level grayscale image. Finally, the spatial features from both levels are concatenated to obtain the final features used for classification. Meta-learning consists of two phases: meta-training and meta-testing. The data types in these two phases do not overlap. Both phases include many tasks, each consisting of a support set and a query set. The meta-training phase first performs an inner loop on the support set of each task, and then performs an outer loop on the query set of all tasks.

[0027] Step 6. Using the generalizable model trained in Step 5, a few fine-tuning steps on the meta-test support set can make it well applicable to the distinction between unseen malware and normal software categories, thus completing the detection of new categories of malware.

[0028] Example

[0029] A multi-level feature fusion malware detection method based on meta-learning includes the following steps:

[0030] Step 1. Collect malware traffic and normal software traffic separately to generate corresponding pcap files. This step uses the publicly available malware detection traffic datasets CICAndMal2017 and MCFP as the malware dataset source, and BEN-1 as the benign software dataset source. CICAndMal2017 and MCFP each contain 42 and 36 malware categories, respectively, while BEN-1 contains 20 benign software categories.

[0031] Step 2. Preprocess the traffic collected in Step 1. First, divide the traffic into streams using a five-tuple (source IP, source port, destination IP, destination port, transport layer protocol), and label each stream with the corresponding software tag. Next, anonymize the streams by randomizing MAC and IP addresses to prevent strong correlation between the model's learning addresses and results, thus ensuring applicability to wide area networks (WANs) beyond local area network (LAN) environments. Finally, remove redundant files, i.e., delete pcap files with identical content to prevent training bias and overfitting.

[0032] Step 3. Convert the first 784 bytes of each stream into a 28*28 flow-level grayscale image (if the number of bytes of the original flow-level information is less than 784 bytes, fill it with 0x00 bytes to 784 bytes). Consider the first 6 packets of each stream (flows with less than 6 packets are not considered). Convert the first 100 bytes of each packet into a 10*10 grayscale image (if the number of bytes of the original packet-level information is less than 100 bytes, fill it with 0x00 bytes to 100 bytes), that is, 6 10*10 grayscale images.

[0033] Step 4. Use MAML as the meta-learning method and a multi-level convolutional neural network (CNN) as the base classifier. The flow-level CNN in the multi-level CNN extracts the spatial features of the flow-level grayscale image (5×5 Conv2d, 8 filters → ReLU → Batch Normalization → 2×2 MaxPool2d → 5×5 Conv2d, 16 filters → ReLU → Batch Normalization → 2×2 MaxPool2d); the packet-level CNN extracts the spatial features of the packet-level grayscale image (3×3 Conv2d, 5 filters → ReLU → Batch Normalization → 2×2 MaxPool2d). Finally, the spatial features of the two levels are concatenated to obtain the final features used for classification. The specific process is as follows: Figure 1 As shown, meta-learning consists of two phases: meta-training and meta-testing. The data types in these two phases do not overlap. Both phases involve many tasks, each consisting of a support set and a query set. The meta-training phase first performs an inner loop on the support set of each task: the initial learner is f with parameters θ. After training on the support set of task i, the optimal model for that task is obtained. Then, an outer loop is performed on the query set of all tasks to update the learner parameters using gradient descent, resulting in model f. θ .

[0034] Step 5. After step 4, a generalizable model f is obtained. θ A few fine-tuning steps (gradient updates) on the support set of the meta-test can be well applied to the detection of new categories of malware.

[0035] Table 1

[0036]

[0037] Table 2

[0038]

[0039] Table 3

[0040]

[0041] We sampled several samples from the training set to construct an N-way K-shot task, where N is 5 and 10, and K is 1 and 5. Tables 2 and 3 show the results of malware detection tasks on the CICAndMal2017 and MCFP datasets, respectively. It can be seen that the proposed Meta-HFMD significantly outperforms other models on malware detection tasks. On the CICAndMal2017 dataset, the accuracies for 5-way 1-shot, 5-way 5-shot, 10-way 1-shot, and 10-way 5-shot are 95.59%, 98.26%, 94.62%, and 94.83%, respectively. The F1 scores are 95.04%, 95.55%, 94.22%, and 94.26%, respectively. In the MCFP dataset, the accuracies for 5-way 1-shot, 5-way 5-shot, 10-way 1-shot, and 10-way 5-shot tests reach 96.64%, 99.23%, 94.39%, and 95.25%, respectively, while the F1 scores reach 95.88%, 96.59%, 92.20%, and 94.12%, respectively. Notably, the proposed method also achieves high accuracy and F1 scores when the shot count is 1, meaning that Meta-HFMD can effectively perform malware detection tasks even with extremely limited data.

[0042] Table 3 shows the experiments conducted on cross-datasets. Although Meta-HFMD's accuracy and F1 score on cross-dataset tasks are lower than those on single datasets, they still reach over 90%, significantly higher than the other two models based on few-shot learning. Since CICAndMal2017 collects Android mobile software traffic and MCFP collects traditional internet software traffic, this demonstrates that Meta-HFMD also performs well on cross-platform malware detection tasks.

Claims

1. A malware detection method based on meta-learning multi-level feature fusion, characterized in that, The method comprises the following steps: Obtaining malware and benign software network traffic dataset in data collection stage; Dividing the network traffic dataset according to five tuples of source IP, source port, destination IP, destination port and transmission layer protocol to obtain flows, and reserving original byte information of the first N packets of each flow; extracting the first M1 bytes of original information in the flow as flow-level features, and extracting the first M2 bytes in each of the N packets as packet-level features, and converting the flow-level features and the packet-level features into grayscale images; Using MAML as a learning method of a multi-level convolutional neural network (CNN), inputting the flow-level feature grayscale image and the packet-level feature grayscale image into the CNN, extracting flow-level spatial features and packet-level spatial features by a feature extraction part of the CNN, and performing feature fusion on the flow-level spatial features and the packet-level spatial features; Inputting the fused features and corresponding malware or benign software labels into a fully connected layer of the CNN to complete learning of a malware and benign software classification task, and finally using the trained CNN for malware detection.

2. The method of claim 1, wherein, After the flows are divided according to the five tuples, if the number of packets of a flow is more than N, only the first N packets are reserved; if the number of packets of a flow is less than N, the flow is discarded.

3. The method of claim 1, wherein, When the first M1 bytes of original information in the flow are extracted, if the number of bytes of original information in the flow is less than M1, 0x00 bytes in hexadecimal are filled; similarly, when the first M2 bytes in each packet are extracted, if the number of bytes of original information in the packet is less than M2, 0x00 bytes in hexadecimal are filled.

4. The method of claim 1, wherein, After the network traffic dataset is divided into flows according to the five tuples, traffic anonymization and redundant file elimination processing are further performed.

5. The method of claim 4, wherein, The traffic anonymization specifically refers to randomization processing on MAC addresses and IP addresses of the divided flows; The redundant file elimination processing specifically refers to deleting redundant files, that is, deleting files with the same content.

Images