A fine-grained internet of things device automatic identification method based on firmware emulation

Abstract

This invention provides a fine-grained automatic identification method for IoT devices based on firmware simulation, relating to the field of IoT technology. The method includes: automatically collecting IoT device tags from aggregation websites using web crawling technology and constructing an IoT device tag library; automatically collecting IoT device firmware using XPath expressions combined with an ASCII-based firmware search algorithm; simulating the obtained device firmware, accessing the simulated device IP address and extracting the device fingerprint; automatically generating and matching regular expressions for positive samples based on the device fingerprint; using a binary tree-based model regular expression matching strategy for efficient and accurate model identification; extracting features from web files in the IoT device firmware using BinWalk; training the extracted features using a decision tree model; designing a tree-based rule-based scanning and identification strategy; converting the decision tree model into a rule tree; and identifying the version number of the IoT device, thereby improving the accuracy of identification.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a fine-grained automatic identification method for IoT devices based on firmware emulation. Background Technology

[0002] The continuous advancements in computer, internet, and mobile communication technologies have driven the rapid development of the Internet of Things (IoT). The number of IoT devices has also increased dramatically, projected to reach 27 billion by 2025. Simultaneously, security issues surrounding IoT devices have become increasingly prominent. In 2021, IoT devices suffered 60.1 million attacks. Particularly in IoT application environments, the harmful effects of known vulnerabilities on IoT devices can persist for a long time. Due to the inherent limitations of IoT devices, including difficulties in firmware upgrades, limited performance, simple protocol design, and a lack of effective security authentication mechanisms, it is difficult to promptly patch these devices with known vulnerabilities. This leads to the long-term persistence of vulnerabilities, allowing attackers to exploit them. Therefore, accurately identifying and assessing unpatched and vulnerable IoT devices is of paramount importance for IoT security.

[0003] Currently, IoT device identification methods can be mainly divided into two categories: coarse-grained IoT device identification and fine-grained IoT device identification. Coarse-grained IoT device identification methods can identify partial information about IoT devices, including type, brand, and model. Fine-grained IoT device identification methods can identify more detailed information about IoT devices, including type, brand, model, and version number. Among the existing coarse-grained identification methods, a multi-classifier IoT device type identification technology has been proposed. This technology inputs the feature information of the device to be identified into multiple classifiers for identification and uses an edit distance algorithm to combine the classification results to determine the final identification result. However, due to its high complexity, this identification technology has relatively low efficiency and accuracy in identifying new devices. Among existing fine-grained identification methods, a fine-grained IoT device identification scheme based on identification feature words is proposed. This scheme mines potential identification feature words in the protocol signature information of IoT devices using the TF-IDF algorithm and designs an IoT device identification result matching algorithm based on inverted indexing. The Simhash algorithm is used to approximate the fine-grained attribute identification results of IoT devices. Although this scheme improves the accuracy of device identification, the TF-IDF algorithm may filter out some identification feature words, resulting in low device identification accuracy. In addition, there is a device identification method based on embedded web file fingerprints for identifying model and version numbers. This method hashes the web files embedded in the firmware and uses pruning techniques to extract features during the matching process between the hash and the fingerprint database to identify the device model and version number. However, this method relies on a fixed feature set and has poor scalability. Research shows that in the field of IoT device identification, fine-grained IoT identification requires more information and the identification process is more complex. Therefore, most research on fine-grained IoT identification methods focuses on non-automation. However, non-automation methods for fine-grained IoT device identification can no longer meet the security needs of the rapidly growing number of IoT devices.

[0004] In summary, while coarse-grained IoT device identification can obtain type, brand, and model information, enabling the identification of a large number of IoT devices, it lacks comprehensive feature extraction and may overlook some information during the identification process, resulting in insufficient detail and an inability to accurately assess the security of IoT devices. It also suffers from low accuracy in identifying device models, reliance on manual intervention in non-automated methods, and high technical complexity in automated methods. Fine-grained IoT device identification, in addition to obtaining type, brand, and model information, can also identify the device version number, which is crucial for security assessment and can provide more accurate results. However, most existing methods are non-automated, requiring significant manual operation, resulting in low efficiency and unsuitability for handling large-scale data, failing to meet the security needs of the rapidly growing number of IoT devices. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, this invention proposes a fine-grained automatic identification method for IoT devices based on firmware simulation, aiming to solve the problems of low accuracy, low efficiency, and poor scalability in IoT device identification.

[0006] To address the aforementioned technical problems, this invention proposes a fine-grained automatic identification method for IoT devices based on firmware simulation, comprising the following steps:

[0007] Step 1: Collect IoT device tags and build an IoT device tag library based on the collected IoT device tags;

[0008] Step 2: Use web crawler technology to design a firmware identification algorithm to automatically collect the firmware of IoT devices, obtain the firmware of IoT devices, and save the obtained firmware to the IoT device tag library;

[0009] Step 3: Simulate the firmware of the IoT device, access the simulated device and extract the device fingerprint, generate a regular expression based on the device fingerprint, and save the generated regular expression using a binary tree-based storage structure, thereby enabling automatic identification of the IoT device model.

[0010] Step 4: Use BinWalk to extract features from the web files in the firmware of IoT devices to obtain status code features, image features, and non-image features; use a decision tree model to train the above features and optimize the decision tree model by pruning; design a scanning and recognition strategy based on tree rules; convert the decision tree model into a rule tree; and use the rule tree to identify the version number of IoT devices.

[0011] Furthermore, the IoT device tags in step 1 include: the type of IoT device, the brand of IoT device, and the model of IoT device; the IoT device tag library includes: an IoT device type tag library, an IoT device brand tag library, and an IoT device model tag library.

[0012] Furthermore, step 2 includes:

[0013] Step 2.1: Use a web crawler in a search engine to automatically retrieve the official website address of publicly available firmware for IoT devices;

[0014] Step 2.2: Based on the official website address, use a web crawler combined with a breadth-first search algorithm to identify and collect all device firmware files of IoT devices from the official website, and use the encoding characteristics of ASCII code to distinguish the collected device firmware files to determine the firmware of IoT devices.

[0015] The method for distinguishing the collected device firmware files by utilizing the encoding characteristics of ASCII code in step 2.2 is as follows: the byte content is divided into a whitelist of visible encoded characters and a blacklist of invisible characters. If the obtained device firmware file contains at least one byte from the whitelist and does not contain any bytes from the blacklist, it is a text file; otherwise, it is a binary file, i.e., firmware.

[0016] Step 2.3: Match the device model corresponding to the firmware link determined in Step 2.2 from the IoT device tag library. After successful matching, remove the device model information from the firmware file name and delete the space symbol to obtain the firmware version number information; save the firmware to the IoT device tag library in the form of a quadruple of "device type, device brand, device model, version number".

[0017] Furthermore, step 3 includes:

[0018] Step 3.1: Use the FirmAE simulation tool to simulate the firmware of the IoT device, obtain the virtual IP address of the IoT device firmware, access the virtual IP address using the HTTP protocol and collect the response data packets, separate the Header and Body in the response data packets, extract the fingerprint feature information in the response data packets as the device fingerprint, and build a device fingerprint database.

[0019] Step 3.2: Design an automatic regular expression generation algorithm to automatically generate regular expressions based on the device fingerprint database;

[0020] Step 3.2 includes:

[0021] Step 3.2.1: Divide the device fingerprints in the fingerprint database into positive samples and negative samples. Positive samples refer to fingerprint data that includes the device model, and negative samples refer to fingerprint data that does not include the device model.

[0022] Step 3.2.2: Generate subsets of positive samples by field, and use the Product function in Python to perform a Cartesian product to obtain a set of regular expressions containing many subsets;

[0023] Step 3.2.3: Use the regular expression library in Python to perform regular expression matching and filtering on the regular expression set, filter out the regular expressions that do not match the negative samples, and calculate the score of each regular expression according to the weighted matching ratio and length ratio using formula (1):

[0024]

[0025] Where score is the score for each regular expression; Len(Matches(part,ps)) represents the number of positive samples that the current regular expression can match; Len(ps) is the total number of positive samples; the ratio of Len(Matches(part,ps)) to Len(ps) represents the proportion of positive samples that the current regular expression can match; Len(part) represents the length of the regular expression; max_length represents the total length of the field; the ratio of Len(part) to max_length represents the proportion of the regular expression length.

[0026] Step 3.2.4: Delete the positive samples that match the regular expression with the highest score, generate new regular expressions using the unmatched positive samples, and iterate until all positive samples are matched;

[0027] Step 3.3: Use a binary tree storage structure to store the generated regular expression. When the device fingerprint is obtained, automatically input the device fingerprint into the root node of the binary tree, and select the left and right nodes in turn for regular expression matching until the leaf node is successfully reached to determine the model of the IoT device.

[0028] Furthermore, step 4 includes:

[0029] Step 4.1: Use the BinWalk tool to decompile the IoT device firmware and extract the file system of the IoT device firmware. Use the OS.Walk method to traverse the file system to locate the Web module, then traverse all Web files in the Web module to obtain the URL access results, and extract features from the Web files in the IoT device firmware based on the URL access results to obtain status code features, image features and non-image features.

[0030] The status code feature mentioned in step 4.1 indicates the status of the webpage request; the image feature indicates the characteristics of the image or icon displayed when accessing the webpage through an IoT device; the non-image feature indicates the frequency of the string appearing in the webpage.

[0031] Step 4.2: Train the three types of features obtained in Step 4.1 using a decision tree model and optimize the decision tree model through pruning. Design a scanning and recognition strategy based on tree rules. Use the export_graphviz function to convert the path from each leaf node to the root node into a rule, and process it using the pydotplus library to form a rule tree. Input the extracted status code features, image features, and non-image features from the web file into the root node of the rule tree, perform feature decomposition, and sequentially identify the status code features, image features, and non-image features. Compare the feature values ​​of the status code features, image features, and non-image features respectively. Select the left and right nodes according to the comparison results and traverse until a leaf node is successfully reached. Complete the traversal and determine the model of the IoT device.

[0032] The beneficial effects of adopting the above technical solution are as follows:

[0033] The method of this invention uses web crawling technology to automatically collect device firmware information from aggregated websites, and compared with the existing technology, it constructs a more complete device tag library. It designs an automatic regular expression generation algorithm to generate unique regular expressions, and also designs a regular expression matching algorithm based on binary trees, which improves matching efficiency and stability while reducing the number of matching times, and achieves accurate model identification.

[0034] The method of this invention solves the dataset scalability problem by using a fingerprint extraction strategy based on firmware simulation. It extracts device fingerprint information by generating virtual IPs through simulation, which improves the quality and coverage of the training dataset and greatly reduces data collection time and labor costs.

[0035] The method of this invention uses firmware simulation technology to construct a fingerprint database to build a rule base, and adopts a binary tree storage structure to improve matching efficiency. Different features of the device firmware Web file are extracted separately, which significantly improves the richness of the feature set. The method uses a decision tree model and pruning optimization to train the feature set and extract tree rules to solve the problem of low version number recognition efficiency.

[0036] The method of this invention enables fine-grained automated identification of IoT devices, improving identification accuracy and efficiency, and ensuring the security of IoT devices. Attached Figure Description

[0037] Figure 1This is a flowchart of a fine-grained automatic identification method for IoT devices based on firmware simulation in this embodiment;

[0038] Figure 2 This is a flowchart of the tag library construction process in this embodiment;

[0039] Figure 3 The flowchart for automatic regular expression generation in this implementation method is as follows;

[0040] Figure 4 This is a flowchart of the version number identification process in this embodiment. Detailed Implementation

[0041] To facilitate understanding of this application, specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following embodiments are illustrative of the invention but are not intended to limit its scope. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0042] The core idea of ​​this invention includes: automating the collection of IoT device tags using web crawling technology on aggregated websites, constructing an IoT device tag library based on the collected tags, and automatically collecting IoT device firmware using XPath expressions combined with an ASCII-based firmware search algorithm; simulating the obtained device firmware, accessing the simulated device IP and extracting its HTTP fingerprint, using an automatic regular expression generation algorithm combined with the RE library to automatically generate regular expressions that can match positive samples, and using a binary tree-based model regular expression matching strategy for efficient and accurate model identification; extracting features from web files in the IoT device firmware using BinWalk to obtain status code features, image features, and non-image features; training the above features using a decision tree model and optimizing the decision tree model through pruning, designing a tree-based rule-based scanning and recognition strategy, converting the decision tree model into a rule tree, and using the rule tree to identify the version number of the IoT device.

[0043] This embodiment provides a fine-grained automatic identification method for IoT devices based on firmware emulation, such as... Figure 1 As shown, it includes the following steps:

[0044] Step 1: Collect IoT device tags and build an IoT device tag library based on the collected IoT device tags;

[0045] Furthermore, the IoT device label includes: the type of IoT device, the brand of IoT device, and the model of IoT device;

[0046] In this embodiment, such as Figure 2As shown, web crawling technology is used to collect IoT device tags. First, the DOM tree structure information of the aggregation website is crawled. Then, different XPath expressions are used to obtain the type, brand, and model of IoT devices to build an IoT device tag library. For the type of IoT devices, since all type tags are distributed under the unordered list UL in the HTML with the class attribute "Clearfix", each specific type tag in the UL is denoted as LI. The XPath expression " / / *[@class="Clearfix"]" is used to obtain the type of IoT devices in the aggregation website. Based on the obtained IoT device types, an IoT device type tag library is built.

[0047] For IoT device brands, brand tag information is distributed under the Class attribute "BrandAll". The XPath expression " / / *[@id="BrandAll"]" is used to obtain all brand tags of IoT devices. The IoT device brand tag library is built based on the obtained brand tag strings and links.

[0048] For IoT device models, the model is obtained by accessing the brand link information of each device. The model is stored in the "list-box" with the Class attribute. The XPath expression " / / *[@class="list-box"]" is used to traverse and retain the block tags DIV in HTML with the Class "list-item". The model information is obtained from the third-level heading tags in HTML, i.e., H3 tags, and deduplicated. The IoT device model tag library is built based on the deduplicated IoT device model information.

[0049] Step 2: Use web crawler technology to design a firmware identification algorithm to automatically collect the firmware of IoT devices, obtain the firmware of IoT devices, and save the obtained firmware to the IoT device tag library;

[0050] In this embodiment, firmware information for IoT devices is typically published on official websites. By using web crawling technology and leveraging the relevance characteristics of search engines, the official website addresses for publicly available firmware are retrieved. This is combined with XPath expressions to search for publicly available official websites for IoT device firmware. Furthermore, by analyzing the HTML source code of these websites, all firmware links for IoT devices are searched. A firmware identification algorithm is designed to automatically collect IoT device firmware, ensuring the completeness of the collected firmware.

[0051] Step 2.1: Use a web crawler in a search engine to automatically retrieve the official website address of publicly available firmware for IoT devices;

[0052] In this implementation, the Selenium automation tool is used to define the search box using XPath=" / / *[@class="gLFyf"]", and then to enter the next search page using XPath=" / / *[@class="gNO89b"]". Finally, " / / *[@class="yuRUbf"]" is used to automatically obtain the target address of the official website for the publicly available firmware of the IoT device.

[0053] Step 2.2: Based on the official website address, use a web crawler combined with a breadth-first search algorithm to identify and collect all device firmware files of IoT devices from the official website, and use the encoding characteristics of ASCII code to distinguish the collected device firmware files to determine the firmware of IoT devices.

[0054] Furthermore, the method for distinguishing the collected device firmware files using the encoding characteristics of ASCII code is as follows: the byte content is divided into a whitelist of visible encoding and a blacklist of invisible characters. If the obtained device firmware file contains at least one byte from the whitelist and does not contain any bytes from the blacklist, it is a text file; otherwise, it is a binary file, i.e., firmware.

[0055] In this implementation, a breadth-first search algorithm is used to identify and collect all firmware links. The Selenium auto-tool controls the browser driver, and the find_elements function is used to traverse all HTML elements in the official website of the IoT device. The links contained in the official website are used as the first layer, and then each link in the first layer is accessed in turn to obtain all device firmware files of the IoT device. Based on the characteristics of ASCII encoding, the collected device firmware files are distinguished as firmware by searching for visual encoding.

[0056] Step 2.3: Match the device model corresponding to the firmware link determined in Step 2.2 from the IoT device tag library. After successful matching, remove the device model information from the firmware file name and delete the space symbol to obtain the firmware version number information. Save the firmware to the IoT device tag library in the form of a quadruple of "device type, device brand, device model, version number".

[0057] Step 3: Simulate the firmware of the IoT device, access the simulated device and extract the device fingerprint, generate a regular expression based on the device fingerprint, and save the generated regular expression using a binary tree-based storage structure, thereby enabling automatic identification of the IoT device model.

[0058] In this embodiment, device fingerprint refers to the characteristic information used to identify and describe network devices. Device model identification is performed on the fingerprint information. To improve identification accuracy, a high-quality device fingerprint dataset needs to be constructed for the generation of regular expressions.

[0059] Step 3.1: Use the FirmAE simulation tool to simulate the firmware of the IoT device, obtain the virtual IP address of the IoT device firmware, access the virtual IP address using the HTTP protocol and collect the response data packets, separate the Header and Body in the response data packets, extract the fingerprint feature information in the response data packets as the device fingerprint, and build a device fingerprint database.

[0060] Step 3.2: Design an automatic regular expression generation algorithm to automatically generate regular expressions based on the device fingerprint database;

[0061] In this embodiment, IoT device model identification employs both machine learning and regular expression methods. Using regular expressions for device model identification offers higher accuracy and stability; however, constructing regular expressions is typically difficult, time-consuming, and lacks scalability. To address these issues, as... Figure 3 As shown, this embodiment designs an automatic regular expression generation algorithm. First, the regular expression format is processed, and the device fingerprints in the fingerprint database are divided into positive samples and negative samples. These samples are matched and filtered to finally obtain the shortest regular expression that matches the positive samples.

[0062] Step 3.2.1: Divide the device fingerprints in the fingerprint database into positive samples and negative samples. Positive samples refer to fingerprint data that includes the device model, and negative samples refer to fingerprint data that does not include the device model.

[0063] Step 3.2.2: Generate subsets of positive samples by field, and use the Product function in Python to perform a Cartesian product to obtain a set of regular expressions containing many subsets;

[0064] Step 3.2.3: Use the regular expression library in Python to perform regular expression matching and filtering on the regular expression set, filter out the regular expressions that do not match the negative samples, and calculate the score of each regular expression according to the weighted matching ratio and length ratio using formula (1):

[0065]

[0066] Where score is the score for each regular expression; Len(Matches(part,ps)) represents the number of positive samples that the current regular expression can match; Len(ps) is the total number of positive samples; the ratio of Len(Matches(part,ps)) to Len(ps) represents the proportion of positive samples that the current regular expression can match; Len(part) represents the length of the regular expression; max_length represents the total length of the field; the ratio of Len(part) to max_length represents the proportion of the regular expression length.

[0067] In this embodiment, the regular expression library in Python, namely the RE library, is used to perform regular expression matching and filtering on the regular expressions in the regular expression set, and to filter out the regular expressions that do not match the negative samples. The matching ratio weight is set to 2 and the length ratio weight is set to 1. The score of each regular expression is calculated using formula (1).

[0068] Step 3.2.4: Delete the positive samples that match the regular expression with the highest score, generate new regular expressions using the unmatched positive samples, and iterate until all positive samples are matched;

[0069] In this implementation, positive samples that match the highest-scoring regular expression are deleted, and new regular expressions are generated using the unmatched positive samples. To ensure matching efficiency, the new regular expressions cannot match positive samples that have already been matched. This process is repeated until all positive samples can be matched.

[0070] Step 3.3: Use a binary tree storage structure to store the generated regular expression. When the device fingerprint is obtained, automatically input the device fingerprint into the root node of the binary tree, and select the left and right nodes in turn for regular expression matching until the leaf node is successfully reached to determine the model of the IoT device.

[0071] In this embodiment, a large number of regular expressions are obtained from step 3.2. To achieve efficient and accurate model identification, a binary tree storage structure is introduced to store all regular expressions. When a device fingerprint is obtained, the fingerprint is automatically input into the root node of the binary tree, and the left and right nodes are selected sequentially for regular expression matching until a leaf node is successfully reached, thus determining the model of the IoT device.

[0072] Step 4: Use BinWalk to extract features from the web files in the firmware of IoT devices to obtain status code features, image features, and non-image features; use a decision tree model to train the above features and optimize the decision tree model by pruning; design a scanning and recognition strategy based on tree rules; convert the decision tree model into a rule tree; and use the rule tree to identify the version number of IoT devices.

[0073] In this embodiment, there is a close correlation between the version number of an IoT device and vulnerabilities. Timely upgrade to the latest version can effectively fix vulnerabilities and improve the security and performance of IoT devices. Since many IoT devices use mixed firmware versions, where lower versions may have security vulnerabilities, automatic version number identification can help with the security analysis of IoT devices. Because the web files in the firmware of IoT devices of the same model but different version numbers differ, the version number can be identified through differences in web file features. However, this requires scanning hundreds of web files. To improve identification efficiency, this embodiment proposes a new feature extraction strategy, such as... Figure 4 As shown, Web files are divided into three categories of features for extraction. Since existing identification methods are inefficient and feature extraction by sending data packets to IoT devices causes significant damage to IoT devices, this implementation proposes a tree-based scanning and identification strategy based on decision tree models. A URL request is sent to the IoT device and the system automatically traverses the tree according to the tree rules to finally identify the device version number.

[0074] Step 4.1: Use the BinWalk tool to decompile the IoT device firmware and extract the file system of the IoT device firmware. Use the OS.Walk method to traverse the file system to locate the Web module, and then traverse all Web files in the Web module to obtain the URL access results. Based on the URL access results, perform feature extraction on the Web files in the IoT device firmware to obtain status code features, image features, and non-image features. The status code features indicate the status of the web page request, the image features indicate the features of the images or icons displayed when accessing the Web page through the IoT device, and the non-image features indicate the frequency of the strings appearing in the web page.

[0075] Step 4.2: Train the three types of features obtained in Step 4.1 using a decision tree model and optimize the decision tree model through pruning. Design a scanning and recognition strategy based on tree rules. Use the export_graphviz function to convert the path from each leaf node to the root node into a rule, and process it using the pydotplus library to form a rule tree. Input the extracted status code features, image features, and non-image features from the web file into the root node of the rule tree, perform feature decomposition, and sequentially identify the status code features, image features, and non-image features. Compare the feature values ​​of the status code features, image features, and non-image features respectively. Select the left and right nodes according to the comparison results and traverse until a leaf node is successfully reached. Complete the traversal and determine the model of the IoT device.

[0076] In this embodiment, the DecisionTreeClassifier method from Scikit-learn is used to train all features. To improve efficiency, this embodiment proposes a scanning and recognition strategy based on tree rules. First, the trained decision tree model is converted into rule form. The export_graphviz function is used to convert the path from each leaf node to the root node into a rule, and the pydotplus library is used to process and form a rule tree. The extracted status code features, image features, and non-image features of the web file are input into the root node of the rule tree. Feature decomposition is performed to identify status code features, image features, and non-image features in turn. The feature values ​​of status code features, image features, and non-image features are compared. First, the status codes are compared. Since the feature values ​​of firmware for different IoT devices are not the same, in this embodiment, / style / advanced_home.css = homes ≤ 3.0, meaning that the number of times 'homes' appears in / style / advanced_home.css is less than or equal to 3. The status code feature is... The URL ` / style / advanced_home.css=homes` has a status code feature value of 3.0. Then, ` / style / advanced_home.css=homes` is decomposed using the "=" sign, resulting in ` / style / advanced_home.css` and `homes`. Next, ` / style / advanced_home.css` is identified to determine if it's an image or not. If it's an image, the response to that URL is MD5 hashed and compared. If it's not an image, the URL is accessed, and the number of occurrences of `homes` in the response is counted. For example, accessing `ip:port / style / advanced_home.css` and counting the number of occurrences of `homes` in the response. If the count is less than or equal to 3, it's correct, and the nodes are traversed to the left; otherwise, it's incorrect, and the nodes are traversed to the right until a leaf node is reached. The `class` value in the leaf node is the firmware version number of the IoT device, such as `class=V1.0.0.38`. Traversing the entire rule tree completes the identification of the IoT device's firmware version number.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A fine-grained automatic identification method for IoT devices based on firmware emulation, characterized in that, The method includes the following steps: Step 1: Collect IoT device tags and build an IoT device tag library based on the collected IoT device tags; Step 2: Use web crawler technology to design a firmware identification algorithm to automatically collect the firmware of IoT devices, obtain the firmware of IoT devices, and save the obtained firmware to the IoT device tag library; Step 3: Simulate the firmware of the IoT device, access the simulated device and extract the device fingerprint, generate a regular expression based on the device fingerprint, and save the generated regular expression using a binary tree-based storage structure, thereby enabling automatic identification of the IoT device model. Step 4: Use BinWalk to extract features from the web files in the firmware of IoT devices to obtain status code features, image features, and non-image features; use a decision tree model to train the above features and optimize the decision tree model by pruning; design a scanning and recognition strategy based on tree rules; convert the decision tree model into a rule tree; and use the rule tree to identify the version number of IoT devices. Step 4.1: Use the BinWalk tool to decompile the IoT device firmware and extract the file system of the IoT device firmware. Use the OS.Walk method to traverse the file system to locate the Web module, then traverse all Web files in the Web module to obtain the URL access results, and extract features from the Web files in the IoT device firmware based on the URL access results to obtain status code features, image features and non-image features. Step 4.2: Train the three types of features obtained in Step 4.1 using a decision tree model and optimize the decision tree model through pruning. Design a scanning and recognition strategy based on tree rules. Use the export_graphviz function to convert the path from each leaf node to the root node into a rule, and process it using the pydotplus library to form a rule tree. Input the extracted status code features, image features, and non-image features from the web file into the root node of the rule tree, perform feature decomposition, and sequentially identify the status code features, image features, and non-image features. Compare the feature values ​​of the status code features, image features, and non-image features respectively. Select the left and right nodes according to the comparison results and traverse until a leaf node is successfully reached. Complete the traversal and determine the model of the IoT device.

2. The method for fine-grained automatic identification of IoT devices based on firmware emulation as described in claim 1, characterized in that, The IoT device tags in step 1 include: the type of IoT device, the brand of IoT device, and the model of IoT device; the IoT device tag library includes: the IoT device type tag library, the IoT device brand tag library, and the IoT device model tag library.

3. The method for fine-grained automatic identification of IoT devices based on firmware emulation as described in claim 1, characterized in that, Step 2 includes: Step 2.1: Use a web crawler in a search engine to automatically retrieve the official website address of publicly available firmware for IoT devices; Step 2.2: Based on the official website address, use a web crawler combined with a breadth-first search algorithm to identify and collect all device firmware files of IoT devices from the official website, and use the encoding characteristics of ASCII code to distinguish the collected device firmware files to determine the firmware of IoT devices. Step 2.3: Match the device model corresponding to the firmware link determined in Step 2.2 from the IoT device tag library. After successful matching, remove the device model information from the firmware file name and delete the space symbol to obtain the firmware version number information. Save the firmware to the IoT device tag library in the form of a quadruple of "device type, device brand, device model, version number".

4. The method for fine-grained automatic identification of IoT devices based on firmware emulation as described in claim 3, characterized in that, The method for distinguishing the collected device firmware files using the encoding characteristics of ASCII code in step 2.2 is as follows: the byte content is divided into a whitelist of visible encoded characters and a blacklist of invisible characters. If the obtained device firmware file contains at least one byte from the whitelist and does not contain any bytes from the blacklist, it is a text file; otherwise, it is a binary file, i.e., firmware.

5. The method for fine-grained automatic identification of IoT devices based on firmware emulation as described in claim 1, characterized in that, Step 3 includes: Step 3.1: Use the FirmAE simulation tool to simulate the firmware of the IoT device, obtain the virtual IP address of the IoT device firmware, access the virtual IP address using the HTTP protocol and collect the response data packets, separate the Header and Body in the response data packets, extract the fingerprint feature information in the response data packets as the device fingerprint, and build a device fingerprint database. Step 3.2: Design an automatic regular expression generation algorithm to automatically generate regular expressions based on the device fingerprint database; Step 3.3: Use a binary tree storage structure to store the generated regular expression. When the device fingerprint is obtained, automatically input the device fingerprint into the root node of the binary tree, and select the left and right nodes in turn for regular expression matching until the leaf node is successfully reached to determine the model of the IoT device.

6. The method for fine-grained automatic identification of IoT devices based on firmware emulation as described in claim 5, characterized in that, Step 3.2 includes: Step 3.2.1: Divide the device fingerprints in the fingerprint database into positive samples and negative samples. Positive samples refer to fingerprint data that includes the device model, and negative samples refer to fingerprint data that does not include the device model. Step 3.2.2: Generate subsets of positive samples by field, and use the Product function in Python to perform a Cartesian product to obtain a set of regular expressions containing many subsets; Step 3.2.3: Use the regular expression library in Python to perform regular expression matching and filtering on the regular expression set, filter out the regular expressions that do not match the negative samples, and calculate the score of each regular expression according to the weighted matching ratio and length ratio using formula (1): （1） in The score for each regular expression; This indicates the number of positive samples that the current regular expression can match; It is the total number of positive samples; and The ratio represents the proportion of positive samples that the current regular expression can match; Indicates the length of the regular expression; Indicates the total length of the field; and The ratio represents the proportion of the regular expression length; Step 3.2.4: Delete the positive samples that match the regular expression with the highest score, generate new regular expressions using the unmatched positive samples, and iterate until all positive samples are matched.

7. The method for fine-grained automatic identification of IoT devices based on firmware emulation as described in claim 6, characterized in that, The status code feature mentioned in step 4.1 indicates the status of the URL access result; the image feature indicates the characteristics of the image or icon displayed when accessing the web page through an IoT device; and the non-image feature indicates the frequency of the string appearing in the web page.

Images