Lightweight IoT malicious behavior identification method and system

Abstract

The invention discloses a lightweight IoT malicious behavior identification method and system, and belongs to the technical field of network security. According to the method, the flow data in the monitoring range of the IoT gateway is acquired, the acquired flow data is preprocessed into the one-way flow metadata based on the equipment, then the host-level features are generated based on the one-way flow metadata, the high-quality host-level features are reserved, and the anomaly detection model and the malicious family detection model are constructed based on the high-quality host-level features; ioT malicious traffic is identified based on the high-quality host-level features using an anomaly detection model, and a malicious family from which the traffic originates is identified based on the high-quality host-level features of the IoT malicious traffic using an identification malicious family detection model. According to the invention, the infected IoT device and the infected malicious family thereof in the monitoring network segment can be identified.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology, and specifically relates to a method and system for identifying lightweight IoT malicious behavior. Background Art

[0002] IoT malware refers to programs or codes that target IoT devices and are intended to manipulate IoT devices to conduct illegal activities. IoT malware locates target devices through IP scanning and spreads by brute-forcing remote login services or exploiting vulnerabilities in other services. Infected devices usually communicate with command and control servers through DGA domain names and execute malicious instructions. IoT malware usually targets distributed denial of service attacks, confidential information theft, virtual currency mining, malicious email propagation, etc.

[0003] Identifying IoT malware is a key step in preventing IoT threats. Existing research focuses on identifying IoT malware from the perspective of network behavior. Traditional malicious traffic identification usually uses deep packet inspection (DPI) or deep flow inspection (DFI) technology, such as the common network intrusion detection system (NIDS) Snort, Suricata, etc., to identify known malicious traffic by detecting JA3, JA3S fingerprints of packet headers, packet payloads or TLS handshakes. With the development of AI technology, machine learning (ML)-based methods have become a research hotspot. Fingerprint-based detection methods are difficult to provide sufficient detection levels for unknown malware variants, while ML methods can identify unknown malicious traffic with similar behavior patterns after being trained in a supervised or unsupervised manner. Existing IoT malicious traffic identification methods are mainly machine learning or deep learning identification methods based on packet or flow granularity. However, due to the large amount of computation required for fine-grained traffic processing such as packets or flows and the limited computing resources of IoT devices, existing methods always face the problems of low efficiency and high cost, making the protection of IoT devices challenging. Since IoT devices are generally low-power, limited in computing power and memory, traditional recognition methods are no longer applicable to IoT scenarios due to their reliance on expensive computing resources.

[0004] Therefore, in order to enhance the security of IoT networks, an accurate and efficient IoT malicious traffic identification method is needed, which can identify infected devices in the monitored network segment with low computing cost, high computing efficiency and high identification accuracy in resource-constrained scenarios such as IoT gateways. Summary of the invention

[0005] The purpose of the present invention is to propose a lightweight IoT malicious behavior identification method and system based on host-level efficient traffic processing, which can identify infected IoT devices and the malicious families they infect within the monitoring network segment, and exchange coarser traffic processing granularity for higher computing efficiency. The present invention aggregates traffic based on devices, and characterizes the connection behavior characteristics, network activity characteristics, and request service type characteristics of IoT malware from its network behavior, combined with a lightweight machine learning algorithm, to solve the problems of low recognition efficiency and high computing cost in the prior art in malicious traffic identification.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0007] A lightweight IoT malicious behavior identification method includes the following steps:

[0008] S1: Obtain traffic data within the monitoring range of the IoT gateway;

[0009] S2: preprocess the acquired traffic data into device-based unidirectional flow metadata;

[0010] S3: Generate host-level features based on unidirectional streaming metadata;

[0011] S4: retaining high-quality host-level features through feature selection;

[0012] S5: Build anomaly detection models and malicious family detection models based on high-quality host-level features;

[0013] S6: Use anomaly detection models to identify malicious IoT traffic based on high-quality host-level features;

[0014] S7: Use the Identify Malicious Families detection model to identify the malicious families from which the traffic originates based on the high-quality host-level features of IoT malicious traffic.

[0015] Furthermore, the method of obtaining flow data in step S1 includes two methods:

[0016] Capture traffic online: Input traffic data by data block based on queue mode to capture traffic;

[0017] Offline data reading: Traffic data is input by data blocks based on the sliding window method to read data.

[0018] Furthermore, the data preprocessing step in step S2 includes:

[0019] Keep the one-way traffic sent from the host in the monitoring area to the gateway, and filter the traffic from outside to inside;

[0020] For the reserved unidirectional traffic, the traffic data packets with the same source IP are aggregated into host-level flows;

[0021] Extract metadata from host-level flows, including source port, destination IP, destination port, packet length, and timestamp gap information, and build a metadata dictionary.

[0022] Furthermore, the host-level features in step S3 include:

[0023] Connection behavior characteristics, including the destination IP, source port, and non-repetition rate of the packet block originating from an IoT device, as well as the mean, variance, skewness, and kurtosis of the frequency list of the destination IP, source port, and destination port;

[0024] Network activity characteristics, including the byte sequence of packets from an IoT device, the mean, variance, skewness, kurtosis, and packet rate of the sequence between timestamps;

[0025] The request service type characteristics include the access service type of the packet block originating from an IoT device and the distribution of the access destination port number. The access service type includes HTTP service, HTTPS service, SSH service, Telnet service, Mail service, DNS service, NTP service, MQTT service, UPnP service, MySQL service, IRC service and BitTorrent service. The distribution of the access destination port number, that is, whether the recognized port, registered port and dynamic port are accessed by the device.

[0026] Furthermore, the feature selection step in step S4 includes: calculating the contribution of each host-level feature to the decision criterion in the process of constructing a decision tree through a random forest algorithm, sorting the importance according to the contribution, and selecting the first several host-level features with the greatest importance as high-quality host-level features.

[0027] Furthermore, the decision criterion is selected from one of the Gini coefficient, information gain, and mean square error.

[0028] Furthermore, the method for constructing an anomaly detection model based on high-quality host-level features in step S5 is: using high-quality host-level features to construct a binary classification data set containing benign traffic and malicious traffic, using a random forest algorithm to perform anomaly detection training on the binary classification data set, and obtaining an anomaly detection model.

[0029] Furthermore, the method for constructing a malicious family detection model based on high-quality host-level features in step S5 is: using high-quality host-level features to construct a multi-classification data set containing several malicious family samples, using a random forest algorithm to perform malicious family detection training on the multi-classification data set, and obtaining a malicious family detection model.

[0030] Furthermore, the several malicious families are specifically 11 malicious families: Mirai, Gafgyt, IRCBot, Kenjiro, Okiru, Muhstik, Hideandseek, Trojan, Troii, Hakai, and Hajime.

[0031] A lightweight IoT malicious behavior identification system, including:

[0032] IoT gateway traffic input module, used to obtain traffic data within the monitoring range of IoT gateway;

[0033] A host-level metadata extraction module, used to pre-process the acquired traffic data into device-based unidirectional flow metadata;

[0034] A host-level feature vector building module, used to generate host-level features based on unidirectional stream metadata;

[0035] A feature selection module, used to retain high-quality host-level features through feature selection;

[0036] A training and evaluation module for building anomaly detection models and malicious family detection models based on high-quality host-level features;

[0037] Anomaly detection module: used to identify IoT malicious traffic based on high-quality host-level features using anomaly detection models; and to identify the malicious family from which the traffic originates based on high-quality host-level features of IoT malicious traffic using malicious family detection models;

[0038] The log output module is used to output logs of anomaly detection and malicious family detection.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention uses host-level traffic granularity as the recognition granularity, exchanging coarser granularity for higher computing efficiency. Without losing the recognition effect, it reduces the computing load brought by traffic processing and greatly improves the system throughput.

[0041] 2. The present invention designs host-level features to describe the network behavior of IoT devices from a higher-dimensional perspective. Such features are closely integrated with the life cycle of IoT malware and avoid the impact of mixing benign and malicious traffic from the same device.

[0042] 3. Based on host-level features, the present invention further accurately identifies the malicious family from which the malicious traffic originates, ultimately achieving the purpose of accurately locating infected IoT devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1This is a flow chart of a method for identifying lightweight IoT malicious behavior in an embodiment.

[0044] Figure 2 This is a module composition diagram of a lightweight IoT malicious behavior identification system in an embodiment. DETAILED DESCRIPTION

[0045] In order to make the various technical features and advantages or technical effects in the above technical solutions of the present invention more obvious and easy to understand, they are described in detail below with reference to the accompanying drawings.

[0046] The embodiment of the present invention specifically discloses a lightweight IoT malicious behavior identification method. The method obtains the traffic in the monitoring network segment from the IoT gateway. The key point is to obtain it in data blocks, and the data block is set to 1000 data packets. The monitoring network segment contains benign and infected IoT devices. The traffic generated by benign devices is marked as benign, and the traffic generated by infected devices is a mixed traffic that contains both benign traffic and malicious traffic. The mixed traffic is marked as malicious. The infected devices involve multiple malware families. The process of this method is as follows Figure 1 As shown, the specific steps include:

[0047] S1: Obtain traffic data within the monitoring range of the IoT gateway

[0048] It can be obtained by capturing traffic online and reading data offline. When capturing online, a queue-based method is used to input data in blocks. When reading offline, a sliding window-based method is used to input data in blocks. A data block is a set of data packets obtained in each data collection.

[0049] S2: Traffic preprocessing into device-based unidirectional flow metadata

[0050] Traffic preprocessing includes the following steps:

[0051] S2-1: Perform unidirectional flow filtering. Only the unidirectional flow sent from the host in the monitoring area to the gateway is retained, and the flow from outside to inside is filtered out.

[0052] S2-2: Aggregate by source IP. Obtain several host-level flows. Each host-level flow is defined as the set of all data packets sent from the same source IP.

[0053] S2-3: Metadata extraction. For each host-level flow, extract the source port, destination IP, destination port, packet length, and timestamp interval of each data packet to prepare for the next step of extracting host-level features. This step forms a metadata dictionary with the device IP as the key and the metadata of its host-level flow as the value.

[0054] S3: Building host-level features

[0055] Based on the metadata dictionary formed in step S2, host-level features are generated. Host-level features are mainly aimed at the stage where IoT malicious processes have significant active network behaviors, including connection behavior features, network activity features, and request service type features.

[0056] The feature construction process includes the following steps:

[0057] S3-1: Connection behavior feature construction. There are 15 features of this type, namely the destination IP, source port, and non-repetition rate of the packet block originating from an IoT device, and the mean, variance, skewness, and kurtosis of the frequency list of the destination IP, source port, and destination port. The non-repetition rate is defined as the ratio of the number of non-repetitive elements in a set to the total number. The frequency list is defined as a set consisting of the number of times each element appears in a set.

[0058] This type of feature is mainly used to identify malware scanning behavior. A large number of malware have horizontal scanning or vertical scanning behaviors. Horizontal scanning refers to a zombie host scanning multiple hosts for the same port number to search for attack targets that have a certain service enabled. Vertical scanning refers to scanning a large number of port numbers for the same target to detect services enabled by the target host. Scanning behavior will produce connection behaviors that are abnormal from benign traffic.

[0059] S3-2: Network activity feature construction. There are 9 features in this category, namely the byte sequence of the packet block from an IoT device, the mean, variance, skewness, kurtosis of the sequence between timestamps, and the packet sending rate (packets / second).

[0060] The network activity feature can describe the amount of data sent and the frequency of packets sent by a host in a collection cycle. The network activity of a normal host should be within an appropriate range. When a device is infected, IoT malware may perform scanning or DOS attacks, thereby generating a large number of packets, resulting in a significant increase in network load.

[0061] S3-3: Construction of request service type features. There are 15 features of this type, namely the access service types of packets from an IoT device, including HTTP service, HTTPS service, SSH service, Telnet service, Mail service, DNS service, NTP service, MQTT service, UPnP service, MySQL service, IRC service, BitTorrent service, corresponding to 23 ports in total, and the distribution of the destination port number accessed, that is, whether the recognized ports (0 to 1023), registered ports (1024 to 49151) and dynamic ports (49151 to 65535) are accessed by the device.

[0062] The basis for the request service type feature is that a large number of malware tend to use different services when conducting CnC communication. This type of feature is designed to effectively distinguish malicious activity traffic, including CnC communication, from benign traffic at the host level, and can distinguish the network behavior of different malware families.

[0063] S4: Host-level feature selection

[0064] Perform feature selection on the host-level traffic in step S3. All features will be selected, and only a small number of high-quality features will be retained. High-quality features are selected by ranking the feature importance of random forest (RF). During the construction of the decision tree in RF, the reduction of the criterion of each feature is calculated to determine the construction decision of the current branch. The criterion is calculated by the Gini coefficient by default.

[0065] S5: Establish anomaly detection model and malicious family detection model respectively

[0066] Two data sets can be constructed using the high-quality host-level features obtained in step S4. Data set A is a binary classification data set of benign and malicious, which is directly input into anomaly detection for training. Data set B is a multi-classification data set containing 11 malicious family samples, which is input into malicious family detection for training. Among them, both the anomaly detection model and the malicious family detection model are based on the random forest algorithm. At the same time, a test set is constructed to test the two models.

[0067] The 11 malicious families include:

[0068] 1. Mirai: A family of botnets that primarily infect IoT devices by scanning for open ports and exploiting weak passwords, and is often used in DDoS attacks.

[0069] 2. Gafgyt: A botnet family that targets IoT devices, spreads by exploiting default or weak passwords, and is also commonly used in DDoS attacks.

[0070] 3. IRC Bot: A malware family that communicates and is controlled via the IRC protocol. It is often used to create botnets. Attackers can control infected devices by sending commands through IRC channels.

[0071] 4. Kenjiro: A botnet family that also typically has the ability to scan and exploit vulnerabilities to infect devices.

[0072] 5. Okiru: A market-specific IoT malware family that spreads by scanning and exploiting weak passwords.

[0073] 6. Muhstik: A botnet malware family that targets IoT devices, exploiting known vulnerabilities to infect devices and incorporate them into the botnet.

[0074] 7. Hideandseek: A sophisticated P2P botnet malware family capable of establishing direct communication links between infected devices, bypassing traditional command and control servers.

[0075] 8.Trojan: includes various Trojan programs targeting IoT devices, which are usually used to gain device control permissions, steal information, or be used in botnets.

[0076] 9. Troii: A malware family commonly associated with Trojan programs, usually disguised as legitimate software to trick users into downloading and installing it.

[0077] 10. Hakai: A malware family that specifically targets routers, spreading by exploiting router vulnerabilities and incorporating infected devices into a botnet.

[0078] 11.Hajime: A sophisticated IoT botnet malware family with no centralized command and control server and communicates through a P2P network.

[0079] S6: Abnormal traffic identification

[0080] Use anomaly detection models to identify traffic generated by IoT malware on a test set.

[0081] S7: Malicious family traffic identification

[0082] For the malicious traffic identified in step S6, its host-level features are input into the malicious family detection model to identify the malicious family from which the traffic originates.

[0083] The embodiment of the present invention also discloses a lightweight IoT malicious behavior identification system for executing the above identification method. The module composition of the system is as follows: Figure 2 As shown, specifically including:

[0084] The IoT gateway traffic input module is used to obtain traffic data within the monitoring range of the IoT gateway. Specifically, the network traffic is obtained online in the form of a queue, or the network traffic is read offline in the form of a sliding window.

[0085] The host-level metadata extraction module is used to pre-process the acquired traffic data into device-based unidirectional flow metadata. Specifically, the acquired original traffic is subjected to unidirectional flow filtering, host-level traffic aggregation, and metadata collection and retention.

[0086] Host-level feature vector construction module, used to generate host-level features based on unidirectional flow metadata, including connection behavior features, network activity features, and request service type features;

[0087] Feature selection module, used to select the best host-level features and filter out high-quality features;

[0088] The training and evaluation module is used to build anomaly detection models and malicious family detection models based on high-quality host-level features. Specifically, it constructs training sets and test sets, trains and tests recognition models and association models, and evaluates their overall accuracy, F1 score, specified type recall, and specified type precision.

[0089] Anomaly detection module: used to identify IoT malicious traffic based on high-quality host-level features using anomaly detection models; and to identify the malicious family from which the traffic originates based on high-quality host-level features of IoT malicious traffic using malicious family detection models;

[0090] The log output module is used to output the logs of anomaly detection and malicious family detection during the deployment phase.

[0091] Experimental test:

[0092] The experimental test of the present invention is based on the extended dataset IoT-23-Gate of the open source dataset IoT-23. IoT-23-Gate contains 12.6GB of benign traffic and 16.2GB of mixed traffic, of which the benign traffic includes 9 IoT devices and 16 PC devices, and the mixed traffic includes 20 IoT devices. The mixed traffic is a mixture of benign traffic and malicious traffic to simulate the real network behavior of infected devices in real scenarios. In the anomaly detection task, the IoT-23-Gate dataset is processed into 52,583 benign samples and 463,568 malicious samples, and the model classification index is obtained through ten-fold cross validation.

[0093] In the malicious family classification task, mixed traffic is divided into 11 categories according to the malware family. Due to the extreme imbalance of data from different families, the experiment oversampled and undersampled the data set. Among them, according to the number of samples of each family, the Mirai, Gafgyt, IRCBot, and Kenjiro families are downsampled to 8,000 samples, the Okiru family is downsampled to 2,000 samples, and the Muhstik, Hideandseek, Trojan, Troii, Hakai, and Hajime families are upsampled to 200 samples. The model classification indicators are also obtained through ten-fold cross validation.

[0094] Experiments have shown that the recognition accuracy and F1 score of the anomaly detection model are both above 0.99, and the recognition accuracy and F1 score of the malicious family detection model are above 0.99 and 0.98 respectively. The system can reach more than 60,000 PPS (packets per second) when processing offline traffic data on a Raspberry Pi 4B. It can be seen that the method and system provided by the present invention can efficiently and accurately identify IoT malware traffic.

[0095] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should all be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.

Claims

1. A lightweight IoT malicious behavior identification method, characterized in that: The following steps are involved: S1: Obtain traffic data within the monitoring range of the IoT gateway; S2: preprocess the acquired traffic data into device-based unidirectional flow metadata; S3: Generate host-level features based on unidirectional streaming metadata; S4: retaining high-quality host-level features through feature selection; S5: Build anomaly detection models and malicious family detection models based on high-quality host-level features; S6: Use anomaly detection models to identify malicious IoT traffic based on high-quality host-level features; S7: Use the Identify Malicious Families detection model to identify the malicious families from which the traffic originates based on the high-quality host-level features of IoT malicious traffic.

2. The identification method according to claim 1, characterized in that: There are two methods for obtaining flow data in step S1: Capture traffic online: Input traffic data by data block based on queue mode to capture traffic; Offline data reading: Traffic data is input by data blocks based on the sliding window method to read data.

3. The identification method according to claim 1, characterized in that: The data preprocessing steps in step S2 include: Keep the one-way traffic sent from the host in the monitoring area to the gateway, and filter the traffic from outside to inside; For the reserved unidirectional traffic, the traffic data packets with the same source IP are aggregated into host-level flows; Extract metadata from host-level flows, including source port, destination IP, destination port, packet length, and timestamp gap information, and build a metadata dictionary.

4. The identification method according to claim 1, characterized in that: The host-level features in step S3 include: Connection behavior characteristics, including the destination IP, source port, and non-repetition rate of the packet block originating from an IoT device, as well as the mean, variance, skewness, and kurtosis of the frequency list of the destination IP, source port, and destination port; Network activity characteristics, including the byte sequence of packets from an IoT device, the mean, variance, skewness, kurtosis, and packet rate of the sequence between timestamps; The request service type characteristics include the access service type of the packet block originating from an IoT device and the distribution of the access destination port number. The access service type includes HTTP service, HTTPS service, SSH service, Telnet service, Mail service, DNS service, NTP service, MQTT service, UPnP service, MySQL service, IRC service and BitTorrent service. The distribution of the access destination port number, that is, whether the recognized port, registered port and dynamic port are accessed by the device.

5. The identification method according to claim 1, characterized in that: The feature selection step in step S4 includes: calculating the contribution of each host-level feature to the decision criterion in the process of building a decision tree through the random forest algorithm, sorting the importance according to the contribution, and selecting the first several host-level features with the greatest importance as high-quality host-level features.

6. The identification method according to claim 5, characterized in that: The decision criterion is one of the Gini coefficient, information gain, and mean square error.

7. The identification method according to claim 1, characterized in that: The method for constructing an anomaly detection model based on high-quality host-level features in step S5 is: using high-quality host-level features to construct a binary classification data set containing benign traffic and malicious traffic, using a random forest algorithm to perform anomaly detection training on the binary classification data set, and obtaining an anomaly detection model.

8. The identification method according to claim 1, characterized in that: The method for constructing a malicious family detection model based on high-quality host-level features in step S5 is: using high-quality host-level features to construct a multi-classification data set containing several malicious family samples, using a random forest algorithm to perform malicious family detection training on the multi-classification data set, and obtaining a malicious family detection model.

9. The identification method according to claim 8, characterized in that: There are 11 malicious families: Mirai, Gafgyt, IRCBot, Kenjiro, Okiru, Muhstik, Hideandseek, Trojan, Troii, Hakai, and Hajime.

10. A lightweight IoT malicious behavior identification system, used to implement the identification method according to any one of claims 1 to 9, characterized in that: include: IoT gateway traffic input module, used to obtain traffic data within the monitoring range of IoT gateway; A host-level metadata extraction module, used to pre-process the acquired traffic data into device-based unidirectional flow metadata; A host-level feature vector building module, used to generate host-level features based on unidirectional stream metadata; A feature selection module, used to retain high-quality host-level features through feature selection; A training and evaluation module for building anomaly detection models and malicious family detection models based on high-quality host-level features; Anomaly detection module: used to identify IoT malicious traffic based on high-quality host-level features using anomaly detection models; Use the malicious family detection model to identify the malicious family from which the traffic originates based on the high-quality host-level features of IoT malicious traffic; The log output module is used to output logs of anomaly detection and malicious family detection.

Images