Method and device for open set identification of malicious traffic based on prediction of uncertainty using deep learning

Abstract

The application provides a method and device for open set identification of malicious traffic based on deep learning prediction of uncertainty. The method comprises the following steps: step 1: constructing and training a deep evidence malicious traffic open set identification model; the deep evidence malicious traffic open set identification model comprises an evidence generation module and a result derivation module; wherein the evidence generation module adopts a convolutional neural network and is used for converting an input feature vector into an evidence vector; the result derivation module adopts evidence deep learning to quantify the classification uncertainty of the input evidence vector; step 2: performing data preprocessing on a to-be-detected original traffic file to obtain a feature vector; and step 3: inputting the feature vector into the trained deep evidence malicious traffic open set identification model to obtain an identification result.

Description

Technical Field

[0001] The present invention relates to the technical field of malicious traffic identification, and in particular to a method and device for open-set identification of malicious traffic based on deep learning prediction uncertainty. Background Art

[0002] With the rapid development and widespread application of the internet, cloud computing, big data, and the Internet of Things (IoT), networks have become pervasive in our daily lives. Consequently, attacks targeting networks are becoming increasingly frequent, and networks face numerous attacks and threats. Malicious traffic generated by network attacks is currently one of the major network security threats and a key target for network security monitoring.

[0003] Malicious traffic identification aims to classify traffic containing malicious behavior into several predefined traffic categories (closed sets). Existing malicious traffic identification methods identify attack behaviors using either supervised or unsupervised learning. Supervised learning trains data-driven classifiers on traffic samples of known classes, achieving promising results but ignoring samples outside of the existing classes in the training set. Once an unfamiliar sample is submitted to the classifier, it may be incorrectly classified as a predefined class, resulting in a high false positive rate. Unsupervised learning, such as clustering, achieves the goal of traffic classification by collecting unlabeled samples from the same class in the feature space. Therefore, unsupervised learning is naturally capable of handling unknown classes. However, it lacks accuracy when processing high-dimensional traffic data, limiting its practical application. Due to the continuous emergence of new polymorphic malware and zero-day attacks, the variability of malware attacks continues to increase, and many malicious attacks remain undetected. The resulting unknown traffic poses a potential threat to network management and is a major obstacle to improving the performance of malicious traffic identification systems.

[0004] In practical applications, malicious traffic identification is essentially an open set identification problem, which requires the classifier to accurately identify known malicious traffic, and at the same time, the model can distinguish when unknown traffic appears. The core of open set identification is the ability to distinguish open set data outside the K closed set classes. The difficulty lies in how to model unknown classes in the absence of unknown class samples. It is more challenging than closed set identification and has greater significance for security-related applications. Existing work aimed at solving the problem of open set malicious traffic identification is quite limited. It is mainly based on threshold-based unknown class detection schemes, which rely on how to train a classifier and perform unknown class detection on this basis. Some studies have proposed that the maximum softmax probability value can be used as an indicator for unknown class detection, and the threshold is determined as the lower bound of the maximum softmax value of known class samples, but the normalization property of the softmax function often makes the output value artificially high, which ultimately leads to a large number of unknown class samples being incorrectly classified as known classes. Open-CNN method [1]The distance between the test sample and the known class sample in the latent space is used, and the upper bound of the distance is used as the threshold. However, the distance function learned by the classifier from the training set cannot correctly measure the test set, and cannot fully play its role in identifying unknown classes, resulting in poor results in unknown class recognition. Summary of the Invention

[0005] To address the problem that traditional closed-set traffic classification models incorrectly classify unknown classes with high confidence, the present invention applies deep learning model uncertainty assessment to the problem of malicious traffic open-set identification, and provides a malicious traffic open-set identification method and device based on deep learning prediction uncertainty, which solves the problem that traditional closed-set methods cannot effectively identify unknown attacks.

[0006] In one aspect, the present invention provides a method for identifying malicious traffic open sets based on deep learning prediction uncertainty, comprising:

[0007] Step 1: Construct and train a deep evidence malicious traffic open set identification model; the deep evidence malicious traffic open set identification model includes an evidence generation module and a result derivation module; wherein the evidence generation module uses a convolutional neural network to convert the input feature vector into an evidence vector; the result derivation module uses evidence deep learning to quantify the classification uncertainty of the input evidence vector;

[0008] Step 2: Preprocess the original traffic file to be detected to obtain a feature vector;

[0009] Step 3: Input the feature vector into the trained deep evidence malicious traffic open set recognition model to obtain the recognition result.

[0010] Furthermore, data preprocessing includes session segmentation, packet processing, and feature vectorization;

[0011] The session division refers to dividing the original traffic file into session flows according to the five-tuple information;

[0012] The packet processing includes removing the Ethernet layer, performing IP anonymization and filling the packet header;

[0013] The feature vectorization refers to processing all conversation streams into feature vectors with the same dimension.

[0014] Furthermore, the feature vectorization process specifically includes: setting the feature vector dimension that meets the model requirements to n1×n2, where n1 is the number of packets allowed to be included in each session flow, and n2 is the byte length allowed to be included in each packet, and then performing the following operations on any session flow s in the session flow set: if the number of packets it contains is not less than n1, extracting the first n1 packets of s; otherwise, padding them with 0 to n1 packets; for any packet p in the session flow sk Perform the following operation: If its length is greater than n2, then intercept p k Otherwise, use 0 to replace p k The length is padded to n2.

[0015] Furthermore, the evidence generation module includes two convolutional layers, one pooling layer and two fully connected layers from shallow to deep layers; the input feature vector passes through the convolutional layer, pooling layer and fully connected layer in sequence, and is then converted into an evidence vector.

[0016] Furthermore, the processing of the result derivation module specifically includes:

[0017] Given a sample x for K-class classification (i) , according to its corresponding evidence vector Based on subjective logic theory, the evidence vector e (i) Conversion to Dirichlet distribution with concentration parameter α (i) , that is, α (i) =e (i) +1 and record it as and Then the corresponding Dirichlet distribution Dir(p (i) |α (i) ); and then calculate the evidence vector e according to formula (2) (i) The quality of belief b k and uncertainty u (i) ; and calculate the sample x according to formula (3) (i) The expected probability of belonging to the kth category

[0018]

[0019] Among them, S (i) is the total intensity of the Dirichlet distribution,

[0020] Furthermore, in the process of training the deep evidence malicious traffic open set recognition model, the loss function shown in formula (6) is used to guide the model to train in the right direction:

[0021]

[0022] Where ψ(·) is the digamma function, y (i) is the sample x i Tags, is the sample x i is the label when it belongs to the kth category, B(·) is the multivariate beta function, and θ is the model parameter.

[0023] Furthermore, in the process of training the deep evidence malicious traffic open set identification model, it also includes: calibrating the model by using the maximum accuracy and uncertainty AvU function, specifically including: establishing the logarithmic constraint relationship L between the maximum class probability and uncertainty according to formula (8) AvU To maximize the AvU function:

[0024]

[0025] in, is the input sample x (i) The maximum class probability of

[0026] Correspondingly, the loss function shown in formula (9) is used to guide the model to train in the right direction:

[0027]

[0028] Where N represents the total number of samples in the training set.

[0029] On the other hand, the present invention provides a device for identifying malicious traffic open sets based on deep learning prediction uncertainty, comprising:

[0030] A model building module is used to build and train a deep evidence malicious traffic identification model; the deep evidence malicious traffic identification model includes an evidence generation module and a result derivation module; wherein the evidence generation module uses a convolutional neural network to convert the input feature vector into an evidence vector; the result derivation module uses evidence deep learning to quantify the classification uncertainty of the input evidence vector;

[0031] The data preprocessing module is used to preprocess the original traffic file to be detected to obtain a feature vector;

[0032] The recognition module is used to input the feature vector into the trained deep evidence malicious traffic recognition model to obtain a recognition result.

[0033] Beneficial effects of the present invention:

[0034] This paper first uses raw traffic data as features for malicious traffic identification, preserving all the characteristic information of the original data packets and avoiding information loss caused by manual feature extraction. It then builds a Deep Evidence Malicious Traffic Recognition (DEMTR) model, simultaneously quantifying multi-classification probabilities and prediction uncertainty, and uses the prediction uncertainty to distinguish known from unknown samples. The effectiveness and robustness of the proposed method are verified on a real-world traffic dataset. Experimental results show that the proposed method adapts well to open-set scenarios while maintaining high performance in traditional closed-set identification settings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a method for open set identification of malicious traffic based on prediction uncertainty of deep learning is provided for an embodiment of the present application.

[0036] Figure 2 A DEMTR model structure diagram is provided for an embodiment of the present application.

[0037] Figure 3 An example of Dirichlet distribution is provided for an embodiment of the present application (taking three classifications as an example, assuming that the sample label is the first class).

[0038] Figure 4 An uncertainty distribution histogram is provided for an embodiment of the present application.

[0039] Figure 5 The relationship between F1 score and openness is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0041] Existing machine learning-based malicious traffic identification technologies can effectively detect abnormal traffic, but they almost all focus on abnormal traffic detection in a closed set environment, that is, the training data and the test data come from the same label space. However, the real network environment is open, and malware is constantly updated and iterated. Under the existing technical conditions, it is impossible to exhaust all attack categories to train a complete identification model. When an unknown attack occurs, if the existing detection system cannot correctly identify it, it will seriously damage network security. To solve this problem, the present application establishes a network model that can not only identify known malicious traffic but also effectively handle unknown traffic. By applying the prediction uncertainty of deep learning to the unknown class identification task, the multi-classification probability and the prediction uncertainty are modeled using an evidence neural network, and a deep evidence malicious traffic open set identification model (DEMTR) is built for malicious traffic identification in an open set scenario. The deep evidence malicious traffic open set identification model learns distinguishable features for malicious traffic identification through training, and models the uncertainty of the prediction results to reject unknown samples. Traffic with high uncertainty is regarded as unknown traffic, while low-uncertainty traffic is classified according to the learned classification probability to achieve the purpose of open set identification.

[0042] Example 1

[0043] like Figure 1 As shown, an embodiment of the present invention provides a method for identifying malicious traffic open sets based on deep learning prediction uncertainty, including a training phase and a testing phase;

[0044] Training phase:

[0045] S101: Build a deep evidence-based open set identification model for malicious traffic;

[0046] Specifically, the deep evidence malicious traffic open set identification model includes an evidence generation module and a result derivation module; wherein the evidence generation module uses a convolutional neural network to convert the input feature vector into an evidence vector; the result derivation module uses evidence deep learning to quantify the classification uncertainty of the input evidence vector;

[0047] As an implementable method, the structure of the evidence generation module is as follows: Figure 2 As shown, from shallow to deep layers, it includes two convolutional layers, one pooling layer and two fully connected layers; the input feature vector passes through the convolutional layer, the pooling layer and the fully connected layer in turn, and is then converted into an evidence vector. Among them, the main function of the convolutional layer is to extract features from the input feature vector and save the global feature information in multiple local feature matrices. The pooling layer performs data dimensionality reduction and feature compression, and then the main function of the two fully connected layers is to map the feature space calculated by the previous layer to the sample label space, integrate the features into several values, and reduce the influence of the feature position on the classification results. In particular, in this embodiment, the pooling layer adopts maximum pooling, that is, a local maximum value is selected as the representative of the local data.

[0048] Specifically, given a sample x for K-class classification (i) , and its corresponding evidence e (i) Expressed as:

[0049] e (i) =g(f(x (i) ;θ))(1)

[0050] Among them, f(·) is the function with parameter θ learned by the convolutional neural network, and g(·) is the evidence e (i) Non-negative evidence function. Specifically, the evidence function can be implemented by an activation function (i.e., RELU, Sigmoid, etc.) to ensure that the network outputs a non-negative evidence vector e (i) .

[0051] It should be noted that evidence, as a measure of the amount of support collected from the data, is conducive to the classification of samples into a specific category. This embodiment uses a one-dimensional convolutional neural network to generate an evidence vector because a one-dimensional convolutional neural network can capture evidence-induced classification opinions from the input, and convolutional neural networks (CNN) are more suitable for processing data with higher feature dimensions, such as images, text, and the encrypted traffic in this article, compared to other deep learning models, due to their parameter sharing and sparse connection characteristics and their ability to extract local features of data. In addition, unlike 2D-CNN, 1D-CNN does not need to convert the input into a two-dimensional form, and can retain the original information of the data to the maximum extent, which is conducive to the classification and identification of encrypted traffic.

[0052] As an implementable method, Figure 2 As shown in FIG, the processing process of the result derivation module specifically includes:

[0053] Given a sample x for K-class classification (i) , according to its corresponding evidence vector Based on subjective logic theory, the evidence vector e (i) Conversion to Dirichlet distribution with concentration parameter α (i) , that is, α (i) =e (i) +1 and record it as and Then the corresponding Dirichlet distribution Dir(p (i) |α (i) ); and then calculate the evidence vector e according to formula (2) (i) The quality of belief b k and uncertainty u (i) ; and calculate the sample x according to formula (3) (i) The expected probability of belonging to the kth category

[0054]

[0055] Among them, S (i) is the total intensity of the Dirichlet distribution,

[0056] From formula (2), we can see that the greater the amount of evidence for a certain category, the higher the quality of its belief, while the uncertainty is inversely proportional to the total amount of evidence observed. The smaller the total amount of evidence, the greater the uncertainty. The output of the standard neural network classifier is the probability distribution of the possible categories to which a certain sample belongs, but the Dirichlet distribution parameterized on the evidence represents the density of each such probability distribution, so it models the second-order probability and uncertainty. Therefore, the sample x(i) The expected probability of belonging to the kth category is the mean of the corresponding Dirichlet distribution, and the calculation formula is shown in formula (3).

[0057] For example, let's further illustrate formulas (2) and (3) using a three-class classification task. Assuming evidence e = <30, 0, 0>, we obtain the Dirichlet concentration parameter α = <31, 1, 1>, and calculate its class probability as p = <0.94, 0.03, 0.03>, with a corresponding uncertainty u of 0.09, indicating that sufficient evidence is observed to obtain a confident prediction. Conversely, if the evidence e = <0.01, 0.01, 0.01>, we obtain the Dirichlet concentration parameter α = <1.01, 1.01, 1.01>, with uncertainty u close to 1. The evidence is highly insufficient, leading to a questionable classification result. When the evidence e = <1, 1, 1>, the uncertainty u is still high, but it is reduced compared to the second case.

[0058] It should be noted that subjective logic theory [2] Through the formula α (i) =e (i) +1 Evidence of learning (i) Converted into the concentration parameter of the Dirichlet distribution, and then constructing the Dirichlet distribution, the Dirichlet distribution is regarded as the conjugate prior of the category distribution, so that the deep neural network gives uncertainty while outputting the prediction result. The concentration parameter of the Dirichlet distribution needs to be determined, which is closely related to the uncertainty of the prediction result. Therefore, the result derivation module of this embodiment uses evidence deep learning to quantify classification uncertainty, which can simultaneously model classification probability and overall uncertainty. In addition, the subjective logic theory regards the multi-classification problem as a belief quality distribution problem. Assuming that the overall belief quality is certain, for the K classification problem, the belief quality is divided into K+1 parts, corresponding to the belief quality of the category and the confidence of the current prediction result. These K+1 values ​​are non-negative and sum to 1, that is, they should satisfy formula (4):

[0059]

[0060] Where u≥0 and b k ≥0 denote the overall uncertainty and the belief quality of the k-th category, respectively.

[0061] S102: Constructing a training set, using the training set to train the deep evidence malicious traffic open set identification model, and using the trained deep evidence malicious traffic open set identification model as a malicious traffic classifier or identifier;

[0062] Specifically, the original network traffic data packets are pre-processed to obtain feature vectors, and the feature vectors are input into the deep evidence malicious traffic open set identification model for training. The specific definition is as follows: given a training data set wherein x i is an instance sample composed of a session flow, y i is the label of x i , and N represents the total number of session flows in the training set.

[0063] The purpose of training is that, in the test phase, the test set D te is an open set containing attack types not appearing in the training set, and therefore the deep evidence malicious traffic open set identification model is trained to learn a model M: x→y, x∈D te , y∈Y os ={1, 2,..., k, unknown}, wherein the instance sample x labeled as unknown represents a new class sample not appearing in the training phase.

[0064] As an implementable manner, the data pre-processing includes three key steps, which are specifically described as follows:

[0065] (1) Session division: according to the quintuple information (i.e. source IP address, source port, destination IP address, destination port, network protocol), the original traffic file is divided into session flows.

[0066] In actual application, the original traffic file set is set as and a set S is initialized as a session flow set in advance, and then for any original traffic file x i , the following steps can be used for session division:

[0067] Step 1.1: Extract the packet set and the quintuple set

[0068] Step 1.2: Select packets with the same quintuple from P i to form a session flow

[0069] Step 1.3: Add s to S;

[0070] Step 1.4: Repeat steps 1.2 to 1.3 until all packets in P i are selected.

[0071] (2) Packet processing includes removing the Ethernet layer, performing IP anonymization and filling the UDP header;

[0072] Specifically, in the TCP / IP structure, each traffic data packet consists of the Ethernet layer, the network layer, the transport layer, and the application layer. For each data packet p in any session flow s in the session flow set, k Both perform packet processing, including:

[0073] Step 2.1: First, remove the Ethernet layer. This is because the three fields in the Ethernet layer are source MAC address, destination MAC address, and protocol version. These fields have little gain in the traffic classification task, so this embodiment does not use Ethernet layer data.

[0074] Step 2.2: Then perform IP anonymization (for example, set the source IP address and destination IP address fields in the network layer header to 0.0.0.0). This is to prevent the model from considering the network layer IP address information as a key feature for attack identification.

[0075] Step 2.3: Finally, fill in the packet header. This is because the packet header lengths of different protocol types used in the transport layer are inconsistent. To maintain the uniformity of the characteristic structure, the packet header lengths of different protocol types need to be adjusted to a uniform length. For example, the commonly used protocols in the transport layer are TCP and UDP. The TCP protocol header length is generally 20 bytes, and the UDP protocol header length is generally 8 bytes. To maintain the uniformity of the characteristic structure, for data packets whose transport layer protocol is UDP, 12 bytes of 0x00 are padded to the end of the UDP header of the data packet to make it 20 bytes in length.

[0076] (3) Feature vectorization: Because each conversation flow contains a different number of data packets and each data packet contains different byte lengths, the purpose of this step is to ensure that the data input to the model has the same dimension.

[0077] As an implementation method, the dimension of the feature vector that meets the model requirements is set to n1×n2, where n1 is the number of packets allowed to be included in each session flow, and n2 is the byte length allowed to be included in each packet. The feature vectorization specifically includes: for any session flow s in the session flow set, if the number of packets it contains is not less than n1, then extract the first n1 packets of s; otherwise, use 0 to fill it to n1 packets; for any packet p in the session flow s, k If its length is greater than n2, then intercept p k Otherwise, use 0 to replace p k The length is padded to n2; thus, the dimension of the conversation flow feature of the conversation flow s is finally obtained to be n1 × n2. It should be noted that in order to achieve the best attack identification performance, the specific values ​​of the number of packets n1 and the byte length n2 are determined according to needs.

[0078] In practical applications, the following process can be used to implement feature vectorization: The number of packets n1 allowed for each conversation flow that meets the model requirements and the byte length n2 allowed for each packet are pre-set. A set y is initialized as an intermediate quantity and a set Y is initialized as a feature vector set. Based on this, for any conversation flow s, feature vectorization specifically includes the following process:

[0079] Step 3.1: Take a packet p from the session stream s k If its length is greater than n2, then intercept p k Otherwise, the first n2 bytes of p k Fill the length to n2;

[0080] Step 3.2: Set the byte length of p to n2 k Add to y;

[0081] Step 3.3: Repeat steps 3.1 to 3.2 until the number of packets in y is no less than n1, then jump to step 3.4. If the number of packets in y is still less than n1 after all packets in session stream s are processed, fill n1 with zeros and then jump to step 3.4.

[0082] Step 3.4: Add y to Y.

[0083] Compared with the commonly used manual traffic data packet feature extraction method, the data preprocessing method adopted by the present invention does not require filtering or designing the traffic features to be extracted, and can retain all the feature information of each data packet.

[0084] The entire data preprocessing algorithm can be found in Algorithm 1 below.

[0085]

[0086]

[0087] The model is trained to obtain classification evidence for each sample, which is then used to calculate the corresponding classification probability and overall uncertainty. When a feature of a sample is found to be associated with a category within the K classes, the corresponding evidence is added, and the Dirichlet distribution is updated based on this evidence. For example, specific patterns in a traffic sample may help classify it as a specific class. Furthermore, taking the remote control malware njRAT as an example, the traffic generated by njRAT typically has characteristics such as greater upstream traffic than downstream traffic and an increased proportion of packets with PSH and SYN flags. If traffic with these characteristics is observed, the Dirichlet concentration parameter corresponding to the njRAT class is increased.

[0088] It should be noted that, in the process of training the deep evidence malicious traffic open set recognition model, the loss function shown in formula (5) used by the traditional neural network classifier can be used to guide the model to train in the right direction:

[0089]

[0090] in Represents sample x (i) The predicted probability of belonging to the kth class.

[0091] Preferably, in order to further calibrate the deep evidence malicious traffic open set identification model, the embodiment of the present invention adjusts the above cross entropy loss so that the model generates more evidence for the correct category of each sample, and the adjusted loss function is shown in formula (6). This loss function is used to guide the model training in the right direction:

[0092]

[0093] Where ψ(·) is the digamma function, y (i) is the sample x i Tags, is the sample x i is the label when it belongs to the kth category, B(·) is the multivariate beta function, and θ is the model parameter.

[0094] Preferably, use L mce The trained DEMTR model can give the classification probability and prediction uncertainty of the sample, but its uncertainty is not calibrated and can be unreliable when used directly for unknown recognition. A well-calibrated model should be certain when making accurate predictions and should give higher uncertainty when it may be inaccurate. Studies have shown that miscalibration of neural networks may be related to overfitting of the negative log-likelihood function. [3] . Since the DEMTR objective function in (6) is equivalent to minimizing the negative log-likelihood function, the trained model may overfit, resulting in poor generalization ability in the malicious traffic open set identification task. Minimizing the cross-entropy loss of the DEMTR model is equivalent to minimizing the negative log-likelihood, so it also needs to be further calibrated. In order to further calibrate the DEMTR model, the embodiment of the present invention adopts the method of maximizing the accuracy and uncertainty (Accuracy vs. Uncertainty, AvU) function.

[0095]

[0096] where n AC ,n AU ,n IC ,n IUThey represent the number of samples in the following four categories, namely accurate and certain (AC), accurate and uncertain (AU), inaccurate and certain (IC), and inaccurate and uncertain (IU). Figure 3 Figure 2 shows examples of DEMTR model outputs for these four cases. To calibrate the forecast uncertainty, we encourage the model to learn a skewed and sharp Dirichlet distribution to get accurate forecasts (e.g. Figure 3 (a) in ), and gives an unbiased and flat Dirichlet distribution simplex for incorrect predictions (e.g. Figure 3 To achieve this goal, the present invention proposes to standardize the model training process by maximizing the expectations of AC and IU. Specifically, according to formula (8), a logarithmic constraint relationship L between the maximum class probability and uncertainty is established. AvU To maximize the AvU function:

[0097]

[0098] in, is the input sample x (i) The maximum class probability; According to formula (8), when the model prediction is accurate, the class probability It should be close to 1, but not accurate approaches 0. Similarly, when the model prediction is confident, the uncertainty u (i) It should be close to 0, otherwise it should be close to 1. Only when all accurate predictions are certain and inaccurate predictions are uncertain, L AvU It is equal to 0.

[0099] Correspondingly, the loss function shown in formula (9) is used to guide the model to train in the right direction:

[0100]

[0101] Where N represents the total number of samples in the training set.

[0102] Testing phase:

[0103] S103: Preprocessing the original traffic file to be detected to obtain a feature vector;

[0104] S104: Input the feature vector into the trained deep evidence malicious traffic open set recognition model to obtain a recognition result.

[0105] Existing deep learning models usually use the softmax function for classification after the neural network and use the maximum value of the softmax output for the final prediction. However, using the softmax output as the confidence of the prediction result will result in a biased high confidence value, and the softmax output is often too "confident" for the wrong prediction result. In order to overcome the defects of the maximum softmax probability value being too high and the weak generalization ability of the distance metric, and to enable the model to better discover unknown class data in the open set malicious traffic identification task, the present invention transforms the original problem into an uncertainty estimation problem. DEMTR solves the problems of multi-class classification and uncertainty estimation at the same time, using the Dirichlet distribution of class probabilities predicted by deep neural networks, and regards the model prediction process as a process of evidence collection. The collected evidence provides information for quantifying the uncertainty of the model prediction. In the process of inference, unknown samples will generate high uncertainty, thereby enabling the model to have the ability to identify the unknown. The model is compared with existing methods on a real data set. The proposed method has improved in multiple indicators such as accuracy and F1 score, and has certain effectiveness and robustness.

[0106] Example 2

[0107] Corresponding to the above method, an embodiment of the present invention provides a malicious traffic open set identification device based on deep learning prediction uncertainty, comprising: a model building module, a data preprocessing module and an identification module;

[0108] The model building module is used to build and train a deep evidence malicious traffic identification model; this model includes an evidence generation module and a result derivation module. The evidence generation module uses a convolutional neural network to convert input feature vectors into evidence vectors, while the result derivation module uses evidence deep learning to quantify the classification uncertainty of the input evidence vectors. The data preprocessing module preprocesses the raw traffic files to be tested to obtain feature vectors. The recognition module inputs these feature vectors into the trained deep evidence malicious traffic identification model to obtain recognition results.

[0109] It should be noted that the embodiment of the present invention is to implement the above method. Its specific functions can be referred to the above method embodiment and will not be described in detail here.

[0110] In order to verify the effectiveness and robustness of the method and device of the present invention, relevant experiments are as follows.

[0111] (1) Experimental data and experimental environment

[0112] Experimental data: Using the MCFP dataset [4]The MCFP dataset consists of raw traffic data captured from a real network environment and is stored in the form of pcap files. The MCFP dataset covers a wide range of malware and contains a large amount of data. This experiment randomly selected 20 types of attack traffic, 10 of which constituted the known class dataset, and the remaining traffic constituted the unknown class dataset. Details of the known and unknown attacks used in the experiment are shown in Table 1. In addition, the known class dataset was divided into a training set, a validation set, and a known class test set in a ratio of 8:1:1. The unknown class dataset was used entirely for testing.

[0113] Experimental environment: The experimental equipment is a desktop computer with Ubuntu 20.04 operating system, Intel Xeon Gold 5218 CPU and GTX 3080 GPU. Pycharm 2021 is used as IDE, Python 3.8 is used as programming language, and Pytorch 1.10.2 is used as deep learning platform, supplemented by GPU acceleration. Data reading, session division and preprocessing operations involve Python's Scapy library and DPKT library. Data format conversion and matrix operations use the NumPy library.

[0114] Table 1 Detailed information of known and unknown attacks used in the experiment

[0115]

[0116] (2) Evaluation indicators

[0117] Given that the proposed DEMTR model is essentially a K+1 classification model, this experiment uses accuracy and F1-score as indicators to evaluate the effectiveness of the model in the open set setting. These indicators are defined as follows:

[0118] Accuracy: It defines the ratio of the number of network traffic samples correctly classified by the model to the total number of traffic samples in the test set.

[0119]

[0120] F1-Score: Defined as the harmonic mean of precision and recall, it is a more effective evaluation metric. It combines both precision and recall, reflecting the overall performance of the model.

[0121]

[0122] Here, assuming that C represents a class in the test set, TP (true positive) is the number of samples correctly classified as class C, TN (true negative) is the number of samples correctly classified as non-class C, FP (false positive) is the number of samples incorrectly classified as class C, and FN (false negative) is the number of samples incorrectly classified as non-class C. (III) Experimental Results and Analysis

[0123] This experiment is divided into three stages: Stage 1 aims to determine the specific values ​​of the hyperparameters (n1, n2) involved in data preprocessing; Stages 2 and 3, based on the work of Stage 1, use the MCFP dataset to compare the proposed DEMTR model with the work related to uncertainty estimation and malicious traffic open set identification, respectively.

[0124] (1) Comparative experiment on feature extraction hyperparameter combinations

[0125] The data preprocessing step involves a set of hyperparameters (n1, n2), which correspond to the spatial features of the attack traffic. Their product represents the length of the extracted spatial features of the session flow. To meet the input constraints of the 2D-CNN model, some related works set this value to a power-of-two form, such as 784 (28*28) or 1521 (39*39), to construct a symmetrical two-dimensional feature matrix. This paper uses a 1D-CNN to process spatial features, eliminating this consideration. The hyperparameter n1 represents the number of intercepted packets in each session flow. A statistical analysis of the traffic sessions in the dataset found that over 95% of sessions have more than 5 packets, while over 95% have fewer than 30 packets. Therefore, in the comparative experiments, n1 was set to a value range of [5, 30], with an interval of 5, for a total of six different value options: {5, 10, 15, 20, 25, 30}. The hyperparameter n2 represents the byte length of each packet intercepted. This is limited by the Ethernet protocol's maximum transmission unit (MTU), which limits packet length to 1500 bytes. Since both the network and transport layer protocol headers of encrypted traffic are particularly useful for attack type identification, the protocol headers are retained. To maximize coverage of the traffic protocol headers and the transport layer payload of specific data transmission links, the hyperparameter n2 in the comparative experiment ranges from [100, 1500], with an interval of 200, for a total of eight possible values: {100, 300, 500, 700, 900, 1100, 1300, 1500}. There are 48 possible combinations of n1 and n2. Traversing these combinations captures the dependencies between n1 and n2, resulting in superior identification performance.

[0126] Table 2 Experimental results of hyperparameter combination comparison

[0127]

[0128] Table 2 lists the F1 scores of the DEMTR model for different (n1, n2) combinations for the malware traffic identification dataset. The overall trend shows that the F1 score increases with the number of packets selected from the session flow and the length of the bytes intercepted from the packets. Intuitively, the more bytes and packets fed into the model, the better the results, but the longer the training time. Experimental results show that the model's F1 score is not simply linearly positively correlated with the value of (n1, n2). Blindly increasing the dimension of the extracted features may cause interference with effective features, thereby weakening the overall model's effectiveness in identifying malicious traffic. In all cases listed in Table 2, the model achieved the best F1 score with n1 and n2 of 30 and 1500, respectively, while the second-best results were with n1 and n2 of 20 and 500, respectively. Considering the feature extraction workload and the time and resource requirements for model training, we selected (n1, n2) of (20, 500). In the hyperparameter comparison experiments, only 800 samples were randomly selected from each class to conserve runtime resources and reduce runtime. Although the trained model is underfitting due to the small number of training samples, it can still reflect the impact of hyperparameters on the model recognition performance and provide the optimal hyperparameter combination.

[0129] (2) Uncertainty estimation comparison experiment

[0130] In order to illustrate that the prediction uncertainty obtained by the DEMTR model has the ability to distinguish between known and unknown classes, this experiment compares it with two representative uncertainty estimation methods BNN SVI [5] and MC Dropout [6] For comparison, BNN SVI expresses the posterior uncertainty of network parameters by approximating the posterior distribution of neural network weights, while MC Dropout uses dropout as a regular term to calculate the uncertainty of prediction. The performance of unknown class detection can be Figure 4 Obviously, the uncertainty ranges generated by BNN SVI and MC Dropout for known and unknown class samples are highly overlapping. However, the proposed method assigns smaller uncertainty to known class instances and larger uncertainty to unknown class instances, which can better distinguish known and unknown classes through uncertainty values.

[0131] (3) Comparative experiments on open-set recognition related work

[0132] The proposed method DEMTR is compared with the most advanced malicious traffic identification models to verify its effectiveness. The comparison models include: (i) CNN, a common one-dimensional convolutional neural network based on the softmax function, which directly uses the softmax output as the confidence to filter low-confidence samples to adapt to open set scenarios; (ii) CNN_LSTM[7] State-of-the-art deep learning based intrusion detection model, (iii) Open-CNN [1] A model that applies statistical extreme value theory and convolutional neural networks to unknown network attack detection.

[0133] During the training process, the model of the application uses L DEMTR The loss function replaces the traditional cross-entropy loss, and a total of 20 iterations of training are performed, with a batch size of 256. The proposed model uses the Adam optimizer, with an initial learning rate of 0.0001, and is decayed once every 7 iterations. We identify 95% of the training data as known classes as the criterion for selecting the threshold of the CNN model and the proposed DEMTR model. The implementation of Open-CNN and CNN_LSTM is based on the corresponding literature. All models are trained on the training set and are subjected to single-class unknown attack testing and multi-class unknown attack testing.

[0134] (3-1) Single-class unknown attack testing

[0135] In order to evaluate the performance of the proposed DEMTR model in detecting a single type of unknown attack, one type of unknown attack is added to the known class test set in each round of experiment, and the accuracy and F1 score are calculated to reflect the unknown class recognition performance. Table 3 lists the experimental results of the proposed DEMTR model and three comparison models under a single type of unknown attack

[0136] Table 3 Comparison of the performance of the DEMTR model of the application and the baseline model in detecting a single type of unknown attack

[0137]

[0138]

[0139] As shown in Table 3, the proposed DEMTR model achieves the best accuracy and F1 score for each unknown attack class. Compared to the state-of-the-art malicious traffic identification method, CNN_LSTM, the proposed DEMTR model significantly improves unknown class detection performance, with the highest accuracy improvement reaching as high as 70%. However, accuracy is susceptible to class imbalance, while the F1 score better reflects overall recognition performance. The DEMTR model achieves a maximum F1 score improvement of 21%, demonstrating its effectiveness in unknown attack identification. A comparison of three models with open set operations—the CNN model, the Open-CNN model, and the DEMTR model—shows that the DEMTR model achieves the best recognition performance, followed by the Open-CNN model, and the CNN model performs the worst, demonstrating the superiority of the DEMTR model. The CNN model's lowest accuracy and F1 score indicate that directly using predicted probabilities as criteria for determining unknown classes is inappropriate. This is because the SoftMax-based neural network gives high confidence even for misclassifications, making its predictions overconfident. The Open-CNN model is able to identify unknown classes because it uses an OpenMax layer and outputs predicted probabilities for the unknown classes. In addition, it is worth mentioning that the comparison method CNN_LSTM is a closed-set classification model, which misclassifies all unknown class instances appearing in the test phase as known classes, which is why its performance is far worse than the proposed DEMTR model.

[0140] (3-2) Multiple unknown attack tests

[0141] To explore the impact of different numbers of unknown attack classes on the proposed algorithm, we plotted the F1 score versus openness curve to show how the F1 score changes as openness increases. Openness is a key concept in open set recognition problems, indicating how open the problem is. In this experiment, N and K represent the number of known and unknown classes, respectively. Openness can be more accurately expressed as:

[0142]

[0143] In the experiment, the recognition classifier was trained using the known class training set, and then unknown classes were gradually added to the known class test set for testing. The unknown class test set contains 10 classes, so the K value increases from 0 to 10. A larger K means a greater openness. For each open point, K new classes are randomly selected from the unknown class test set and the F1 score is calculated. After repeating the random selection 10 times, the average is calculated to obtain the final F1 score. The F1 score and openness curve are shown in Figure 2. Figure 5 shown.

[0144] Figure 5 It clearly shows that the proposed DEMTR model achieves the best performance. Figure 5As can be seen, when the openness is 0, meaning no unknown classes are added, the F1 scores of the CNN model and the CNN_LSTM model exceed 95%. This demonstrates that traditional deep learning-based malicious traffic identification models can achieve excellent performance when the test data does not contain unknown classes. However, once unknown classes are added to the test set, the recognition performance of the CNN and LSTM models rapidly declines. As the openness increases, the proportion of unknown classes in the test set gradually increases, and the F1 score curves of the four models all show a downward trend. However, the proposed method shows the smallest decline and the gap with the other comparison methods continues to widen, demonstrating its robustness in identifying unknown classes. It is important to note that the closed set accuracy of Open-CNN is significantly lower than that of the other comparison methods. This is because Open-CNN directly modifies the activation layer vector and outputs the prediction probability of the unknown class, which may negatively impact the accurate prediction of known class instances.

[0145] The related prior art documents mentioned in this invention include:

[0146] [1]Zhang Y,Niu J,Guo D,et al.Unknown network attack detection based on open set recognition[J].Procedia Computer Science,2020,174:387-392.

[0147] [2]Jsang A.Subjective Logic:Aformalism for reasoning underuncertainty[M].Springer,2018.

[0148] [3]Mukhoti J,Kulharia V,Sanyal A,et al.Calibrating Deep NeuralNetworks using Focal Loss[C].arXiv,2020.

[0149] [4]Malware Capture Facility Project[EB / OL].https: / / www.stratosphereips.org / datasets-malware.

[0150] [5]Blundell C, Cornebise J, Kavukcuoglu K, et al. Weight Uncertainty in Neural Networks[J]. Computer Science, 2015.

[0151] [6] Gal Y, Ghahramani Z. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning [C]. International Conference on Machine Learning, 2016.

[0152] [7] Zhang Y, Chen X, Jin L, et al. Network intrusion detection: Based on deep hierarchical network and original flow data [J]. IEEE Access, 2019, 7: 37004-37016.

[0153] Finally, it should be noted that: the above examples are used to illustrate the technical solutions of the present application, but not limited to them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An open set identification method for malicious traffic based on deep learning prediction uncertainty, characterized by: include: Step 1: Build and train a deep evidence malicious traffic open set identification model; The deep evidence malicious traffic open set identification model includes an evidence generation module and a result derivation module. The evidence generation module uses a convolutional neural network to convert an input feature vector into an evidence vector. The result derivation module uses evidence deep learning to quantify the classification uncertainty of the input evidence vector. The processing of the result derivation module specifically includes: Given a sample x for K-class classification (i) , according to its corresponding evidence vector Based on subjective logic theory, the evidence vector e (i) Conversion to Dirichlet distribution with concentration parameter α (i) , that is, α (i) =e (i) +1 and record it as and Then the corresponding Dirichlet distribution Dir(p (i) |α (i) ); and then calculate the evidence vector e according to formula (2) (i) The quality of belief and uncertainty u (i) ; and calculate the sample x according to formula (3) (i) The expected probability of belonging to the kth category Among them, S (i) is the total intensity of the Dirichlet distribution, Step 2: Preprocess the original traffic file to be detected to obtain a feature vector; Step 3: Input the feature vector into the trained deep evidence malicious traffic open set recognition model to obtain the recognition result.

2. The method for identifying malicious traffic open sets based on deep learning prediction uncertainty according to claim 1 is characterized in that: Data preprocessing includes session segmentation, packet processing, and feature vectorization; The session division refers to dividing the original traffic file into session flows according to the five-tuple information; The packet processing includes removing the Ethernet layer, performing IP anonymization and filling the packet header; The feature vectorization refers to processing all conversation streams into feature vectors with the same dimension.

3. The method for identifying malicious traffic open sets based on deep learning prediction uncertainty according to claim 2 is characterized in that: The feature vectorization process specifically includes: setting the feature vector dimension that meets the model requirements to n1×n2, where n1 is the number of packets allowed to be included in each session flow, and n2 is the byte length allowed to be included in each packet, and then performing the following operations on any session flow s in the session flow set: if the number of packets it contains is not less than n1, extract the first n1 packets of s; otherwise, fill it with 0 to n1 packets; for any packet p in the session flow s k Perform the following operation: If its length is greater than n2, then intercept p k Otherwise, use 0 to replace p k The length is padded to n2.

4. The method for identifying malicious traffic open sets based on deep learning prediction uncertainty according to claim 1 is characterized in that: The evidence generation module includes two convolutional layers, one pooling layer and two fully connected layers from shallow to deep layers; the input feature vector passes through the convolutional layer, pooling layer and fully connected layer in sequence, and is then converted into an evidence vector.

5. The method for identifying malicious traffic open sets based on deep learning prediction uncertainty according to claim 1 is characterized in that: In the process of training the deep evidence malicious traffic open set recognition model, the loss function shown in formula (6) is used to guide the model to train in the right direction: Where ψ(·) is the double gamma function, y (i) is the sample x i Tags, is the sample x i is the label when it belongs to the kth category, B(·) is the multivariate beta function, and θ is the model parameter.

6. The method for identifying malicious traffic open sets based on deep learning prediction uncertainty according to claim 5 is characterized in that: In the process of training the deep evidence malicious traffic open set identification model, it also includes: using the maximum accuracy and uncertainty AvU function to calibrate the model, specifically including: establishing the logarithmic constraint relationship L between the maximum class probability and uncertainty according to formula (8) AvU To maximize the AvU function: in, is the input sample x (i) The maximum class probability of Correspondingly, the loss function shown in formula (9) is used to guide the model to train in the right direction: Where N represents the total number of samples in the training set.

7. Malicious traffic open set identification device based on deep learning prediction uncertainty, characterized by: The method for identifying malicious traffic open sets based on deep learning prediction uncertainty as claimed in claim 1 comprises: A model building module is used to build and train a deep evidence malicious traffic identification model; the deep evidence malicious traffic identification model includes an evidence generation module and a result derivation module; wherein the evidence generation module uses a convolutional neural network to convert the input feature vector into an evidence vector; the result derivation module uses evidence deep learning to quantify the classification uncertainty of the input evidence vector; The data preprocessing module is used to preprocess the original traffic file to be detected to obtain a feature vector; The recognition module is used to input the feature vector into the trained deep evidence malicious traffic recognition model to obtain a recognition result.

Images