A method, apparatus and storage medium for detecting malicious traffic in small samples

Abstract

This invention relates to a method, apparatus, and storage medium for detecting small-sample malicious traffic in edge intelligent networks. The method includes: collecting and labeling network traffic from known IoT network activities; preprocessing and extracting features from the network traffic, and dividing it into an election set and an auxiliary set; constructing a metric learning model based on a triple network, where the input of the model is a triple and the output is the Euclidean distance between feature vectors in the embedding layer; generating training triples based on the election set and the auxiliary set; using the training triples, distinguishing similar and dissimilar sample pairs based on a contrastive distance loss metric, and training the metric learning model; combining real-time network traffic samples with supporting sample pairs of malicious node network traffic to generate detection triples; and using the metric learning model to detect unknown malicious activity network traffic. Compared with existing technologies, this invention has the advantages of high detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of encrypted malicious traffic detection, and in particular to a method, apparatus and storage medium for detecting small sample malicious traffic for edge intelligent networks. Background Technology

[0002] With the development of edge intelligent networks, future networks such as the Internet of Things (IoT) and massive machine-type communication (mMTC) can achieve low latency and high reliability. However, malware targeting these networks is rapidly emerging. It has been reported that IoT nodes can be controlled by attackers to carry out malicious activities such as DDoS attacks and flood attacks. These malicious nodes degrade the QoS of IoT networks. In recent years, artificial intelligence (AI) has developed rapidly, enabling machines to learn from historical samples and make decisions on new samples. Recently, many AI methods have been proposed to monitor IoT node activity and detect anomalous IoT nodes. These AI methods are mainly based on machine learning and deep learning algorithms, which rely on sufficient network traffic samples generated from IoT networks. A significant drawback of these models is that once the network environment changes, such as the entry of new nodes or the emergence of new malicious activities, the system needs to be updated based on collecting enough fresh training data. When malicious nodes with unknown malicious activities appear, these classification models become ineffective, requiring sufficient samples to be collected for retraining before classification can be achieved.

[0003] The rapid development of future networks such as the Internet of Things (IoT) and autonomous vehicles has led to a massive increase in network traffic. Consequently, intelligent backhaul and forward provisioning have emerged in scientific research. These intelligent backhaul and forward provisioning networks can meet the high reliability and low latency requirements of future networks. Many papers have proposed new solutions to make backhaul and forward provisioning networks more intelligent by using technologies such as artificial intelligence. Timely detection of malicious nodes in future networks is crucial for low-latency backhaul / forward provisioning. However, traditional supervised learning-based detection models require a large number of malicious training samples, and capturing enough malicious samples cannot meet the timely detection requirements of intelligent backhaul / forward provisioning networks to improve QoS. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, and storage medium for detecting malicious traffic in small samples for edge intelligent networks, thereby improving the QoS of IoT backhaul / fronthaul networks by detecting malicious nodes with unknown malicious activities using a limited number of network traffic samples.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A method for detecting small-sample malicious traffic in edge intelligent networks includes the following steps:

[0007] Network traffic is collected and labeled from known IoT network activity, including normal activity network traffic from benign nodes and malicious activity network traffic from malicious nodes. The labeled network traffic is categorized into normal categories and various malicious categories.

[0008] Network traffic is preprocessed and features are extracted, and then divided into an election set and an auxiliary set;

[0009] A metric learning model based on a triple network is constructed. The input of the metric learning model is a triple, and the output is the Euclidean distance of the feature vectors in the embedding layer, which is used to realize malicious traffic detection.

[0010] Training triples are generated based on the election set and auxiliary set;

[0011] By using training triples, similar and dissimilar sample pairs are distinguished based on the contrastive distance loss metric, and the metric learning model is trained.

[0012] Real-time network traffic samples are combined with supporting samples of malicious node network traffic to generate detection triples. The trained metric learning model is then used to detect network traffic of unknown malicious activities.

[0013] The election set consists of network traffic samples from each node, selected in equal numbers at random, while the auxiliary set consists of the remaining network traffic samples excluding those from the election set.

[0014] The specific steps for generating training triples based on the election set and auxiliary set are as follows: select an election sample from the election set, randomly select a positive sample and a negative sample from the auxiliary set, and generate training triples, wherein the positive sample is labeled the same as the election sample, and the negative sample is labeled differently from the election sample.

[0015] The method for constructing the support sample pairs is as follows: a support set is constructed by K-shot network traffic samples of unknown malicious nodes and K-shot network traffic samples of known malicious nodes. Each sample in the support set is combined with a traffic sample of the normal category to construct a support sample pair. The value of K is preset and represents the number of samples in the support set.

[0016] The triple network consists of three identical artificial neural networks that share weights with each other. Each artificial neural network consists of a three-layer network structure, and each layer consists of a fully connected layer connected to a Dropout layer.

[0017] The contrast distance loss is calculated based on the positive distance between the elected sample and the positive sample, and the negative distance between the elected sample and the negative sample.

[0018]

[0019] Where, d + Let x be the election sample and x be the positive sample. + The positive distance between them, d - For the election sample x and the negative sample x - The negative distance between them, ||·||2 represents the Euclidean distance.

[0020] The positive distance d between the election sample and the positive sample + for:

[0021]

[0022] The negative distance d between the election sample and the negative sample - for:

[0023]

[0024] In this context, Net(·) represents a triple network.

[0025] When detecting unknown malicious network traffic, the positive and negative distances between real-time network traffic samples and malicious node network traffic support sample pairs are calculated to obtain similarity scores. A confidence set is generated, and the confidence set is voted on to obtain the maximum similarity score. It is then determined whether the maximum similarity score is greater than a pre-configured threshold. If it is, the category of the sample corresponding to the maximum value is the detection result of the real-time network traffic sample. Otherwise, the real-time network traffic sample is marked as an unknown malicious traffic category, and the sample and its category are assigned to the support set. Based on this, the detection of real-time network traffic is achieved.

[0026] A small-sample malicious traffic detection device for edge intelligent networks includes a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements the method described above.

[0027] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) For edge intelligent networks, this invention proposes a novel method for detecting small amounts of malicious traffic, which can detect and prevent malicious nodes from accessing the core network in a timely manner, thereby reducing the traffic pressure and congestion possibility of the backhaul / fronthaul network of the edge intelligent network and improving the QoS and security of the backhaul / fronthaul network.

[0030] (2) The present invention can detect malicious traffic when there is a lack of malicious traffic sample data, and the detection accuracy is high.

[0031] (3) This invention proposes a model training method based on metric learning, which uses sample pairs generated from benign nodes and known malicious nodes for model training, while a small number of snapshot traffic samples of unknown malicious nodes are used as a support set for detecting malicious nodes. It can learn to distinguish whether two traffic samples are of the same class or different classes, and realize model training and detection when there is a lack of malicious traffic sample data.

[0032] (4) When a malicious node with unknown activity appears, the present invention can accurately detect it and classify it into a support set as a support sample pair for subsequent detection. It can achieve accurate detection when the same type of malicious traffic appears in the future without retraining the model. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the system structure corresponding to the malicious traffic detection method of the present invention;

[0034] Figure 2 This is a comparative diagram of the malicious traffic detection method of the present invention and the traditional detection method, wherein the dashed box on the left represents the traditional detection method and the dashed box on the right represents the method described in the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of a metric learning model based on a triple network;

[0036] Figure 4 A schematic diagram showing the convergence results of models in different training categories;

[0037] Figure 5 A schematic diagram showing the convergence results of models for different IoT devices;

[0038] Figure 6 A schematic diagram comparing the detection results of models with different numbers of support set samples;

[0039] Figure 7 A schematic diagram showing the detection results of different IoT devices;

[0040] Figure 8 This diagram illustrates the detection results of malicious traffic from different unknown attack categories. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0042] This embodiment provides a method for detecting small-sample malicious traffic in edge intelligent networks, and the corresponding system is as follows: Figure 1 As shown, the IoT traffic processing and detection method proposed in this invention can detect malicious traffic, enabling IoT devices to block the access of malicious nodes and improve the QoS of the IoT backhaul / fronthaul network. This system can detect malicious nodes with unknown malicious activity using a limited number of network traffic samples, and can promptly detect and block malicious nodes from accessing the core network, thereby reducing backhaul / fronthaul traffic pressure and the possibility of congestion, and improving the QoS and security of the backhaul / fronthaul network.

[0043] The method for detecting small-sample malicious traffic in edge intelligent networks according to the present invention includes the following steps:

[0044] Step 1) Collect and label network traffic from known IoT network activities, including normal activity network traffic of benign nodes and malicious activity network traffic of malicious nodes. The labeled network traffic is categorized into normal categories and various malicious categories.

[0045] Step 2) Preprocess the network traffic and extract its temporal features, and divide it into an election set and an auxiliary set.

[0046] The election set C consists of network traffic samples from each node selected in equal numbers at random, while the auxiliary set A consists of the remaining network traffic samples excluding those from the election set.

[0047] Step 3) Construct a metric learning model based on a triple network, such as... Figure 3 As shown, the input of the metric learning model is a triple, and the output is the Euclidean distance of the feature vectors in the embedding layer, which is used to realize malicious traffic detection.

[0048] Distance metric learning has been proposed for many years and has subsequently been applied to many fields such as computer vision and natural language processing. This invention uses a distance metric learning model to detect a small number of malicious activities in the Internet of Things (IoT). Existing technologies employ a Siamese network-based metric learning model to learn to distinguish between similar and dissimilar sample pairs. Formally, given a pair of networks (X, Y) sharing weights and a pair of samples (x1, x2) as input, the model calculates similarity using Euclidean distance:

[0049] d = ||f1(x1) - f2(x2)||2

[0050] The training objective is to minimize the contrast loss, as shown in the following equation:

[0051] l(x1,x2)=y(x1,x2) i lgd i+(1-y(x1,x2) i )lg(1-d i )+γ|w|*2

[0052] Where γ is the L2 regularization parameter. Furthermore, another loss function for Siamese networks is shown in the following formula, which includes a specified floating value m:

[0053]

[0054] This invention uses a metric learning model based on a triple network to detect the similarity between different traffic instances. The triple network consists of three identical artificial neural networks that share weights. Each artificial neural network has a three-layer structure, and each layer consists of a fully connected layer connected to a Dropout layer.

[0055] Step 4) Generate training triples based on the election set and auxiliary set, specifically:

[0056] Select an election sample x from the election set C, and randomly select a positive sample x from the auxiliary set A. + and a negative sample x - The process involves generating training triples, where the positive samples are labeled the same as the election samples, and the negative samples are labeled differently; that is, the positive samples and election samples belong to the same category, and the negative samples and election samples belong to different categories. This process is repeated to generate enough training triples to train the metric learning model.

[0057] Step 5) Use training triples to distinguish between similar and dissimilar sample pairs based on the contrastive distance loss metric, and train the metric learning model.

[0058] The contrast distance loss is calculated based on the positive distance between the elected sample and the positive sample, and the negative distance between the elected sample and the negative sample.

[0059]

[0060] Where, d + Let x be the election sample and x be the positive sample. + The positive distance between them, d - For the election sample x and the negative sample x - The negative distance between them, ||·||2 represents the Euclidean distance.

[0061] The positive distance d between the election sample and the positive sample + for:

[0062]

[0063] The negative distance d between the election sample and the negative sample - for:

[0064]

[0065] In this context, Net(·) represents a triple network.

[0066] Step 6) Combine the real-time network traffic samples with the malicious node network traffic support samples to generate detection triples. Use the trained metric learning model to detect unknown malicious network traffic.

[0067] The method for constructing the support sample pairs is as follows: a support set S is constructed by K-shot network traffic samples of unknown malicious nodes and K-shot network traffic samples of known malicious nodes. Each sample in the support set S is combined with a traffic sample of the normal category to construct a support sample pair. The value of K is preset based on empirical values ​​and represents the number of samples in the support set.

[0068] When detecting unknown malicious network traffic, the positive and negative distances between real-time network traffic samples and malicious node network traffic support sample pairs are calculated to obtain similarity scores. A confidence set is generated, and the confidence set is voted on to obtain the maximum similarity score. It is then determined whether the maximum similarity score is greater than a pre-configured threshold. If it is, the category of the sample corresponding to the maximum value is the detection result of the real-time network traffic sample. Otherwise, the real-time network traffic sample is marked as an unknown malicious traffic category, and the sample and its category are assigned to the support set. Based on this, the detection of real-time network traffic is achieved.

[0069] Comparison of the method described in this invention with traditional methods Figure 2 As shown, traditional detection methods require acquiring a large amount of malicious traffic to train the model. Furthermore, once a malicious node with unknown malicious activity appears, historical and new traffic samples are needed to construct a retraining set to retrain the detection model. Since collecting a large number of IoT traffic samples from unknown malicious nodes is impractical, using traditional machine learning methods to detect these emerging malicious nodes is difficult and inefficient. The method described in this invention effectively overcomes this deficiency.

[0070] A) Experimental Dataset

[0071] The evaluation dataset in this embodiment uses the NB-aIoT dataset, which consists of preprocessed network traffic from eight different IoT devices. Each device includes normal activity and ten different malicious activities caused by Mirai and Gafgyt malware. In this embodiment, IoT devices with malicious activity are considered malicious nodes. The preprocessing method extracts temporal features from the raw traffic and directly applies this preprocessed traffic as training and testing samples, generating training triples, support sets, and unknown test samples. This embodiment uses data from five IoT devices and five types of network activity as training data. The testing phase uses data from the remaining devices and malicious activities. Specifically, this embodiment evaluates the ability of the metric learning model to detect unknown malicious nodes, including multiple sets of experimental results. In the testing phase of each set of experiments, a device with one specific activity is selected as a malicious node.

[0072] To demonstrate the detailed performance of the proposed low-probability malicious node detection system, this embodiment conducted three types of experiments. These experiments evaluated the model convergence capability, detection accuracy with small support sets of different sizes, and detection performance against different types of IoT devices and malicious active nodes.

[0073] B) Evaluation of the convergence ability of the metric learning model.

[0074] B1) This section aims to demonstrate the convergence capability of the metric learning model for malicious node detection. Since the main advantage of the metric learning model proposed in this invention is its ability to detect unknown malicious nodes not present during training, a set of experiments was first conducted to test the model's convergence capability by using different numbers of activity classes for learning. The models used in this set of experiments are all metric learning models based on a triple network, and the model loss is as follows: Figure 4 As shown, the metric learning model of this invention converges better when trained on 4 classes and 8 classes. Since there is almost no difference in convergence ability between the models trained on 4 classes and 8 classes, 4 classes were used as the number of training activity classes in the remaining experiments, considering that training on fewer classes has better efficiency and usability.

[0075] B2) This section primarily tests the model's convergence capability across different IoT devices. A metric learning model based on a triple network is trained on three different IoT devices, and for each device, samples from four malicious activity classes are used to train the model. The evaluation accuracy during training is as follows: Figure 5 As shown, it can be seen that the model converges well on IoT devices.

[0076] C) Evaluation of K-shot support set detection accuracy.

[0077] The purpose of this section is to explore the performance of the metric learning detection model of this invention when tested with different K values ​​on the K-shot support set. In this embodiment, the impact of different cases, K=1, 5, and 10, on the detection results was studied. All experiments in this section were conducted on Baby Monitor traffic samples. Each experiment was trained against four types of malicious activity and tested against the remaining malicious activity. The detection accuracy of these experiments is as follows: Figure 6 As shown, better performance was obtained with 5 and 10 tests. Considering detection efficiency, this embodiment uses a support set size of 5-shot, i.e., K=5.

[0078] D) Performance evaluation of malicious node detection.

[0079] This section of experiments evaluates the detection accuracy of metric learning models based on triple networks, Siamese networks, and state-of-the-art (SOTA) supervised learning models such as CNNs.

[0080] D1) This section trains and tests different types of IoT devices. Each experiment is trained on normal activity samples of four device types and tested on samples of the remaining device types. Detection results are as follows: Figure 7 As shown, this demonstrates that the proposed metric learning model based on a triple network outperforms the model based on a Siamese network, and its detection accuracy is comparable to that of a CNN model, achieving an average accuracy of approximately 97.67%. Furthermore, the model of this invention only requires 5 samples from malicious nodes to achieve this accuracy. These detection results demonstrate that the proposed detection method can detect malicious nodes in unknown IoT devices, even though these nodes did not appear during the training phase of the metric learning model, and maintains high detection accuracy.

[0081] D2) This section trains and tests against different types of malicious activity. Each experiment is conducted on samples from the same IoT device (i.e., a baby monitor), but while training on samples from four types of malicious activity, samples from the remaining types of malicious activity are tested. Detection accuracy is as follows: Figure 8 As shown, the performance of the metric learning model based on the triple network proposed in this invention is comparable to that of the CNN model, while the system of this invention can detect malicious nodes with unknown malicious activities in only 5 samples.

[0082] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for detecting small-sample malicious traffic in edge intelligent networks, characterized in that, Includes the following steps: Network traffic is collected and labeled from known IoT network activity, including normal activity network traffic from benign nodes and malicious activity network traffic from malicious nodes. The labeled network traffic is categorized into normal categories and various malicious categories. Network traffic is preprocessed and features are extracted, and then divided into an election set and an auxiliary set; A metric learning model based on a triple network is constructed. The input of the metric learning model is a triple, and the output is the Euclidean distance of the feature vectors in the embedding layer, which is used to realize malicious traffic detection. Training triples are generated based on the election set and auxiliary set; By using training triples, similar and dissimilar sample pairs are distinguished based on the contrastive distance loss metric, and the metric learning model is trained. Real-time network traffic samples are combined with malicious node network traffic support samples to generate detection triples. The trained metric learning model is then used to detect unknown malicious network traffic. The method for constructing the support sample pairs is as follows: a support set is constructed by K-shot network traffic samples of unknown malicious nodes and K-shot network traffic samples of known malicious nodes. Each sample in the support set is combined with a traffic sample of the normal category to construct a support sample pair. The value of K is preset and represents the number of samples in the support set.

2. The method for detecting small-sample malicious traffic in edge intelligent networks according to claim 1, characterized in that, The election set consists of network traffic samples from each node, selected in equal numbers at random, while the auxiliary set consists of the remaining network traffic samples excluding those from the election set.

3. The method for detecting small-sample malicious traffic in edge intelligent networks according to claim 1, characterized in that, The specific steps for generating training triples based on the election set and auxiliary set are as follows: select an election sample from the election set, randomly select a positive sample and a negative sample from the auxiliary set, and generate training triples, wherein the positive sample is labeled the same as the election sample, and the negative sample is labeled differently from the election sample.

4. The method for detecting small-sample malicious traffic in edge intelligent networks according to claim 1, characterized in that, The triple network consists of three identical artificial neural networks that share weights with each other. Each artificial neural network consists of a three-layer network structure, and each layer consists of a fully connected layer connected to a Dropout layer.

5. The method for detecting small-sample malicious traffic in edge intelligent networks according to claim 1, characterized in that, The contrast distance loss is calculated based on the positive distance between the elected sample and the positive sample, and the negative distance between the elected sample and the negative sample. in, Election Sample x Compared with positive samples The positive distance between them Election Sample x With negative samples Negative distance between them This indicates Euclidean distance.

6. The method for detecting small-sample malicious traffic in edge intelligent networks according to claim 5, characterized in that, The positive distance between the election sample and the positive sample for: The negative distance between the election sample and the negative sample for: in, This represents a triple network.

7. The method for detecting small-sample malicious traffic in edge intelligent networks according to claim 6, characterized in that, When detecting unknown malicious network traffic, the positive and negative distances between real-time network traffic samples and malicious node network traffic support sample pairs are calculated to obtain similarity scores. A confidence set is generated, and the confidence set is voted on to obtain the maximum similarity score. It is then determined whether the maximum similarity score is greater than a pre-configured threshold. If it is, the category of the sample corresponding to the maximum value is the detection result of the real-time network traffic sample. Otherwise, the real-time network traffic sample is marked as an unknown malicious traffic category, and the sample and its category are assigned to the support set. Based on this, the detection of real-time network traffic is achieved.

8. A small-sample malicious traffic detection device for edge intelligent networks, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

9. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-7.

Images