Internet of Things equipment fingerprint generation method and system

Abstract

The invention relates to an Internet of Things equipment fingerprint generation method and system, and the method comprises the steps: collecting the protocol slogan data, with a label, of Internet of Things equipment through a network scanning tool, and decomposing the protocol slogan data into a byte-level sub-word data set through a word segmentation device; pre-training a mask language model through a byte-level sub-word data set, learning a contextual relationship between word meanings and sub-words, optimizing the stability of embedding to dynamic contents through a supervision and comparison loss fine tuning method, and generating slogan-level embedding suitable for fingerprint identification of downstream task equipment; performing dimension reduction clustering on the slogan-level embedding to obtain device class clusters, extracting a common sub-character string from each device class cluster, and converting the common sub-character string into a regular expression device fingerprint; and comparing the regular expression device fingerprint with the existing fingerprint database, and updating the fingerprint database. By adopting the technical scheme of the invention, automatic generation of the active detection device fingerprint can be realized through active detection and technical optimization, and the manual maintenance cost of the device fingerprint is reduced.

Description

Technical Field

[0001] This application relates to the field of Internet of Things security technologies, and particularly to a method and system for generating fingerprints of Internet of Things devices. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology, intelligent devices have been widely used in fields such as smart home, industrial control, and medical health. However, the proliferation of IoT devices has also brought serious security risks. Due to problems such as default configuration vulnerabilities and unauthorized access, a large number of IoT devices have become entry points for network attacks. Therefore, a covert and efficient IoT device detection technology has become one of the keys to network security protection, asset management, and vulnerability detection. By identifying the types of IoT devices and their service statuses in the network, data support can be provided for risk warning, access control, and threat tracing.

[0003] In the prior art, the mainstream IoT device detection technologies mainly rely on two methods: active scanning and passive traffic analysis. Among them, the passive traffic analysis technology extracts device traffic characteristics and then uses technologies such as machine learning and deep learning to identify the device types. However, this method depends on the active communication behavior of the device, with low detection efficiency and poor real-time performance, and cannot identify silent devices. The active scanning technology sends standardized detection requests to the target network and identifies the device according to the returned information. It mainly relies on a manually generated regular expression fingerprint library and matches protocol banner data to identify the device types. However, since the protocol banners often contain dynamic content such as timestamps, random IDs, or minor version numbers, it will lead to unstable fingerprint matching, and the manual maintenance of the fingerprint database is very difficult. It requires experts to update to adapt to new devices. With the continuous emergence of new devices and software, a large amount of human resources and time need to be invested.

[0004] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0005] In view of the above problems, this application provides a method and system for generating fingerprints of Internet of Things devices, which can actively detect and optimize technologies to achieve the automatic generation of active detection device fingerprints and reduce the manual maintenance cost of device fingerprints.

[0006] To achieve the objectives of this application, the following technical solutions are provided in this application:

[0007] In a first aspect, this application provides a method for generating fingerprints of Internet of Things devices, including:

[0008] Collect the protocol slogan data with tags of IoT devices through a network scanning tool, and decompose the protocol slogan data into a byte-level sub-word dataset through a tokenizer;

[0009] Pre-train a masked language model with the byte-level sub-word dataset to learn the semantic relationship between words and sub-word contexts, and optimize the stability of the embedding for dynamic content through the supervised contrastive loss fine-tuning method to generate slogan-level embeddings suitable for downstream task device fingerprint recognition;

[0010] Perform dimensionality reduction clustering on the slogan-level embeddings to obtain device clusters, and extract common substrings from each device cluster and convert them into regular expression device fingerprints;

[0011] Compare the regular expression device fingerprints with the existing fingerprint database and update the fingerprint database.

[0012] In a possible implementation, the step of collecting the protocol slogan data with tags of IoT devices through a network scanning tool and decomposing the protocol slogan data into a byte-level sub-word dataset through a tokenizer includes:

[0013] Use a port scanning tool to scan the available ports of the IP address, combine a protocol scanning tool to capture the protocol slogan data, and obtain the tags corresponding to the protocol slogan data through a network mapping tool to obtain the protocol slogan data with tags;

[0014] Preprocess the collected slogan data, remove special symbols and HTML structure symbols in it, and convert uppercase characters to lowercase;

[0015] Train a byte-level tokenizer to decompose the protocol slogan data with tags into byte-level sub-words.

[0016] In a possible implementation, the step of training a byte-level tokenizer to decompose the protocol slogan data with tags into byte-level sub-words includes:

[0017] Decompose each slogan into a single-character sequence and map it to an ASCII byte sequence;

[0018] Initialize a vocabulary containing all single characters, and count the frequencies of characters and adjacent byte pairs;

[0019] Merge the highest-frequency byte pairs in the current dataset into new sub-words and update the slogan dataset;

[0020] Repeat the merging process until the vocabulary size reaches the first preset threshold or the number of merges reaches the second preset threshold.

[0021] In a possible implementation, the steps of pre-training a masked language model through a byte-level sub-word dataset, learning the relationship between word semantics and sub-word context, and optimizing the stability of embedding to dynamic content through a supervised contrast loss fine-tuning method to generate a slogan-level embedding suitable for downstream task device fingerprint recognition include:

[0022] Pre-train the masked language model on a byte-level subword dataset to learn the semantics and contextual relationships of subwords;

[0023] Conduct multiple detections on the same IP address and port at different times, extract protocol slogan data and record them as slogan pairs, filter the dynamic content in the slogan pairs through regular expression rules, and generate static slogan data;

[0024] The static slogan pairs generated by the same slogan pair are taken as positive sample pairs, slogans are extracted from different slogan pairs to generate negative sample pairs, and cosine similarity is used for pre-screening to construct positive and negative sample pairs;

[0025] The subword embeddings of each slogan are generated through the pre-trained masked language model, and the embeddings of all subwords in the sequence are averaged to generate slogan-level embeddings suitable for downstream task device fingerprint recognition. The first loss function is constructed to bring the cosine similarity of the positive sample embeddings closer and push the similarity of the negative sample embeddings further away through the supervised contrast loss.

[0026] In a possible implementation, the first loss function is:

[0027] ;

[0028] in, is the first loss function, For slogan The embedding vector of For slogan The positive sample embedding of For slogan The embedding vector of the slogan Negative sample embedding, Calculate the cosine similarity, is the number of positive sample pairs, is the temperature parameter.

[0029] In a possible implementation, the masked language model is pre-trained using a cross entropy loss function; the cross entropy loss function is:

[0030] ;

[0031] in, is the cross entropy loss function, M represents the set of masked subwords, is the true value of subword i, The probability predicted by the model based on the context.

[0032] In a possible implementation, the step of performing dimensionality reduction clustering on the slogan-level embedding to obtain device clusters and extracting common substrings from each device cluster and converting them into regular expression device fingerprints includes:

[0033] Perform linear dimensionality reduction on the slogan-level embedding through principal component analysis, and conduct comparative verification in combination with uniform manifold approximation and projection non-linear dimensionality reduction;

[0034] Use hierarchical density clustering to cluster the dimensionality-reduced slogan-level embedding to generate device clusters;

[0035] Statistically analyze the distribution of metadata labels of the slogans within each cluster, and calculate the label consistency score; if the consistency score is higher than the fourth preset threshold, assign the most common label classification to the cluster as the device information label of the cluster;

[0036] Select samples from each cluster to extract common substrings, generate device fingerprints in the form of regular expressions, and combine with the device information labels of the clusters to generate structured fingerprint rules.

[0037] In a possible implementation, after the step of selecting samples from each cluster to extract common substrings and generating device fingerprints in the form of regular expressions, it further includes:

[0038] Apply the regular expression to all the slogans within the cluster, calculate the coverage rate, and if the coverage rate is less than the fifth preset threshold, reselect samples or adjust the regular expression rules until the coverage rate is greater than or equal to the fifth preset threshold.

[0039] In a possible implementation, the step of comparing the regular expression device fingerprints with the existing fingerprint database and updating the fingerprint database includes:

[0040] Match the newly generated regular expression fingerprints one by one with the rules in the original fingerprint database for semantic equivalence;

[0041] The fingerprints that match successfully inherit the corresponding device information in the database;

[0042] The fingerprint rules that do not match successfully are used as new rules to update the fingerprint rule library.

[0043] In a second aspect, the present application further provides an Internet of Things device fingerprint generation system for executing the above-mentioned Internet of Things device fingerprint generation method. The system includes:

[0044] A data collection module, configured to collect protocol slogan data with tags of Internet of Things devices through a network scanning tool, and decompose the protocol slogan data into a byte-level sub-word data set through a tokenizer;

[0045] An embedding generation module, configured to pre-train a masked language model through the byte-level sub-word data set, learn the semantic relationship between words and sub-word context, and optimize the stability of the embedding for dynamic content through the supervised contrastive loss fine-tuning method, and generate slogan-level embeddings suitable for downstream task device fingerprint recognition;

[0046] A fingerprint generation module, configured to perform dimensionality reduction clustering on the slogan-level embeddings to obtain device clusters, and extract common substrings from each device cluster and convert them into regular expression device fingerprints;

[0047] A fingerprint database update module, configured to compare the regular expression device fingerprints with the existing fingerprint database and update the fingerprint database.

[0048] The technical solution provided by this application may include the following beneficial effects:

[0049] Through an Internet of Things device fingerprint generation method and system provided by this application, it is possible to actively detect and obtain protocol slogan data, use the supervised contrastive loss to fine-tune the masked language model to generate stable semantic embeddings to handle dynamic content, and combine dimensionality reduction and HDBSCAN clustering to automatically generate and update regular expression-based device fingerprints, realizing automatic generation of Internet of Things device fingerprints based on active detection, so as to be able to identify silent devices, achieve stable fingerprint matching, and reduce the cost of manual fingerprint maintenance.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0052] Figure 1 It is a schematic flowchart of an Internet of Things device fingerprint generation method provided by an embodiment of this application;

[0053] Figure 2 It is a schematic flowchart of step S100 of an Internet of Things device fingerprint generation method provided by an embodiment of this application;

[0054] Figure 3It is a schematic flowchart of step S200 of an Internet of Things device fingerprint generation method provided by an embodiment of the present application;

[0055] Figure 4 It is a schematic flowchart of step S300 of an Internet of Things device fingerprint generation method provided by an embodiment of the present application;

[0056] Figure 5 It is a schematic flowchart of step S400 of an Internet of Things device fingerprint generation method provided by an embodiment of the present application;

[0057] Figure 6 It is a schematic structural diagram of an Internet of Things device fingerprint generation system provided by an embodiment of the present application. Detailed implementation manners

[0058] Now, example embodiments will be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The features, structures, or characteristics described can be combined in any suitable manner in one or more embodiments.

[0059] In this example embodiment, a dynamic interference power management Internet of Things device fingerprint generation method under feedback sparsity and observation loss conditions is first provided. Referring to Figure 1 as shown in, this Internet of Things device fingerprint generation method may include the following steps:

[0060] Step S100: Collect protocol slogan data with tags of the Internet of Things device through a network scanning tool, and decompose the protocol slogan data into a byte-level sub-word data set through a word segmenter.

[0061] Step S200: Pre-train a masked language model through the byte-level sub-word data set, learn the semantic relationship between words and sub-word context, and optimize the stability of the embedding for dynamic content through the supervised contrastive loss fine-tuning method to generate a slogan-level embedding suitable for downstream task device fingerprint recognition.

[0062] Step S300: Perform dimensionality reduction clustering on the slogan-level embedding to obtain device clusters, and extract common substrings from each device cluster and convert them into regular expression device fingerprints.

[0063] Step S400: Compare the regular expression device fingerprint with the existing fingerprint database and update the fingerprint database.

[0064] Through the above-mentioned method for generating Internet of Things device fingerprints, it is possible to distinguish the dynamic and static parts in the protocol slogan, propose a fine-tuning method based on supervised contrastive loss, optimize the stability of the embedding, enhance the robustness to dynamic content, and at the same time automatically generate fingerprint rules through dimensionality reduction and clustering, supplement and update the fingerprint rule library, adapt to new devices and software, so as to obtain protocol slogan data through active detection, supervise the contrastive loss fine-tuning masked language model to generate stable semantic embeddings to handle dynamic content, and combine dimensionality reduction and HDBSCAN clustering to automatically generate and update device fingerprints in the form of regular expressions, realizing the identification of silent devices and improving the stability of fingerprint matching at the same time.

[0065] Next, reference will be made to Figures 2 to 5 to describe each step of the above-mentioned method for generating Internet of Things device fingerprints in the exemplary embodiment in more detail.

[0066] In step S100, protocol slogan data with tags of Internet of Things devices is collected through a network scanning tool, and the protocol slogan data is decomposed into a byte-level sub-word data set by a tokenizer.

[0067] It can be understood that the protocol slogan data includes the header information of HTTP, FTP, and SSH protocols; the tags include device type, device manufacturer, and model data.

[0068] In a possible implementation manner, step S100 may further include the following sub-steps:

[0069] In step S110, the available ports of the IP address are scanned using a port scanning tool, the protocol slogan data is captured in combination with a protocol scanning tool, and the tags corresponding to the protocol slogan data are obtained through a network mapping tool to obtain protocol slogan data with tags.

[0070] It should be noted that data collection is divided into two parts. First, the collection of protocol slogan data. Zmap is used as the port scanning tool to scan the common ports of the target IP address in the network space, including ports 80, 21, and 22, to obtain a list of open ports. Secondly, the protocol scanning tool ZGrab is used to capture the slogan data of protocols such as HTTP, FTP, and SSH. The slogan data includes protocol header information, such as the Server field of HTTP and the welcome message of FTP. Finally, device metadata is obtained from the existing network mapping engine to provide preliminary tags for some protocol slogans, including device type, manufacturer, and model tags.

[0071] In step S120, the collected slogan data is preprocessed to remove special symbols and HTML structure symbols therein, and convert uppercase characters to lowercase.

[0072] It is understandable that the collected slogan data is processed by removing special symbols and line breaks, retaining letters, numbers, and necessary separators, and converting uppercase characters in the slogans to lowercase to ensure consistent data formats.

[0073] In step S130, a byte-level tokenizer is trained to break the protocol slogan data with labels into byte-level subwords.

[0074] It is understandable that the tokenizer training part uses the BPE tokenizer to tokenize the slogans, generating a sequence of byte-level subwords.

[0075] Furthermore, step 130 may further include the following substeps:

[0076] In step S131, each slogan is broken into a single-character sequence and mapped to an ASCII byte sequence.

[0077] In step S132, a vocabulary containing all single characters is initialized, and the frequencies of characters and adjacent byte pairs are counted.

[0078] It is understandable that after initialization, the vocabulary consists of single characters (such as the ASCII character set), and each character in the slogan dataset exists independently.

[0079] In step S133, the highest-frequency byte pair in the current dataset is merged into a new subword, and the slogan dataset is updated.

[0080] It is understandable that the occurrence frequencies of all adjacent character pairs are counted, and the highest-frequency pair (such as "ab") is selected and merged into a new subword (such as " <ab>"), update the data set (replace "ab" in the original slogan with " <ab>")

[0081] In step S134, the merging process is repeated until the vocabulary size reaches the first preset threshold or the number of merges reaches the second preset threshold.

[0082] In step S200, a masked language model is pre-trained on the byte-level sub-word dataset to learn the semantics of words and the relationship between sub-words and contexts, and through the supervised contrastive loss fine-tuning method, the stability of the embedding for dynamic content is optimized to generate slogan-level embeddings suitable for downstream task device fingerprint recognition.

[0083] It can be understood that a pre-trained masked language model is used to generate semantic embeddings of protocol slogans, capture the context relationship of sub-words, and through supervised contrastive loss fine-tuning, the stability of the embedding for dynamic content is optimized to generate high-quality embedding representations suitable for downstream task device fingerprint recognition.

[0084] In a possible implementation manner, step S200 may further include the following sub-steps:

[0085] In step S210, the masked language model is pre-trained on the byte-level sub-word dataset to learn the semantics and context relationship of sub-words.

[0086] It should be noted that the pre-trained BERT model is used as the masked language model and fine-tuned on the tokenized slogan dataset; the fine-tuning dataset is obtained by converting the protocol slogan dataset into a sub-word sequence using a trained BPE tokenizer, and the maximum sub-word sequence length of each slogan is limited; a part of the sub-words in each slogan sub-word sequence is randomly selected for masking operation, and the selected sub-words are replaced with the special token [MASK]; with the goal of predicting the masked sub-words, the model's semantic understanding of the protocol slogans is optimized.

[0087] Optionally, the cross-entropy loss function is used for model training:

[0088] ;

[0089] where is the cross-entropy loss function, M represents the set of masked sub-words, is the true value of sub-word i, is the probability predicted by the model based on the context.

[0090] In step S220, the same IP address and port are probed multiple times at different times, and the protocol slogan data is extracted and recorded as slogan pairs, and the dynamic content in the slogan pairs is filtered through regular expression rules to generate static slogan data.

[0091] It is understandable that each pair of banners reflects the response of the same device at different times, and dynamic content may cause differences in banners; the regular expressions used include, timestamp regular expression [0-9]{4}-[0-9]{2}-[0-9]{2}, random ID regular expression [0-9a-f]{8}, dynamic version number regular expression [0-9]+\.[0-9]+\.[0-9]+; dynamic content includes timestamp and random ID.

[0092] In step S230, static slogan pairs generated from the same slogan pair are taken as positive sample pairs, slogans are extracted from different slogan pairs to generate negative sample pairs, and cosine similarity is used for pre-screening to construct positive and negative sample pairs.

[0093] It can be understood that static slogans of the same slogan pair are positive samples, and slogans randomly selected from slogans of different devices are negative samples. To ensure the semantic difference of negative samples, pre-calculated cosine similarity is used to filter negative samples, and slogans with similarity below a third preset threshold are retained.

[0094] In step S240, a subword embedding of each slogan is generated through a pre-trained masked language model, the embeddings of all subwords in the sequence are averaged to generate a slogan-level embedding suitable for downstream task device fingerprint recognition, and a first loss function is constructed to bring the cosine similarity of the positive sample embedding closer and push the similarity of the negative sample embedding further away through the supervised contrast loss.

[0095] It should be noted that the subword embedding of each slogan is generated through the fine-tuned BERT model, and the embeddings of all subwords in the sequence are averaged (Mean Pooling) to generate slogan-level embeddings to reduce the local impact of dynamic content on the embedding.

[0096] Furthermore, the first loss function is:

[0097] ;

[0098] in, is the first loss function, For slogan The embedding vector of For slogan The positive sample embedding of For slogan The embedding vector of the slogan Negative sample embedding, Calculate the cosine similarity, is the number of positive sample pairs, is the temperature parameter.

[0099] It can be understood that the first loss function is used to bring the positive samples closer and push the negative samples further away. In the case of bringing the positive samples closer, the slogan embeddings of the same device at different times after mean pooling are made highly similar in the semantic space, thereby strengthening the consistency of the static core semantics and ignoring the interference of dynamic content; in the case of pushing the negative samples further away, the slogan embeddings of different devices with large semantic differences after cosine filtering are significantly separated in the semantic space, thereby enhancing the semantic recognition between devices.

[0100] In step S300, the slogan-level embedding is subjected to dimensionality reduction clustering to obtain device clusters, and common substrings are extracted from each device cluster and converted into regular expression device fingerprints.

[0101] It should be noted that the dimensionality reduction adopts a dimensionality reduction strategy that combines principal component analysis (PCA) and unified manifold approximation and projection (UMAP) to ensure the retention of semantic information; the clustering adopts the HDBSCAN algorithm to cluster the embeddings after dimensionality reduction, generate device clusters, and mine slogan groups with similar semantics.

[0102] In a possible implementation, step S300 may include the following sub-steps:

[0103] In step S310, linear dimensionality reduction is performed on the slogan-level embedding through principal component analysis, and a comparison and verification is performed by combining unified manifold approximation with projected nonlinear dimensionality reduction.

[0104] It should be noted that PCA linear dimensionality reduction: by calculating the covariance matrix of the embedding matrix, extracting the eigenvector with the largest eigenvalue, and constructing a low-dimensional projection matrix. The embedding vector is projected to the low-dimensional space through matrix multiplication to generate the embedded representation after dimensionality reduction. UMAP nonlinear dimensionality reduction verification: In order to verify the semantic preservation effect of PCA dimensionality reduction, UMAP is used for nonlinear dimensionality reduction comparison; based on manifold learning, a topological representation of high-dimensional data is constructed, and the distance relationship between data points is retained by optimizing the cross entropy loss of low-dimensional embedding; the number of neighbors is set to control the degree of retention of local structure, the minimum distance is set to avoid too dense embedding points, and the cosine distance is used as a metric to adapt to the semantic characteristics of the embedding vector.

[0105] In step S320, hierarchical density clustering is used to cluster the reduced-dimensional slogan-level embeddings to generate device clusters.

[0106] It should be noted that the clustering uses the HDBSCAN algorithm to cluster the embeddings after dimensionality reduction, generate device clusters, and mine slogan groups with similar semantics.

[0107] In step S330, the metadata tag distribution of the slogans in each cluster is counted, and the tag consistency score is calculated; if the consistency score is higher than the fourth preset threshold, the most common tag classification is assigned to the cluster as the device information tag of the cluster.

[0108] It should be noted that for each cluster, the distribution of metadata tags (device type, manufacturer, model) of the slogans within the cluster is statistically analyzed, and the tag consistency score is calculated. If the consistency score is higher than the threshold, the most common tag is assigned to the cluster as the device information tag of the cluster.

[0109] In step S340, samples are selected from each cluster to extract common substrings, generate a device fingerprint in the form of a regular expression, and combine it with the device information tag of the cluster to generate a structured fingerprint rule.

[0110] It can be understood that several slogan samples that are semantically most similar to the cluster center are selected from each cluster as representative slogans. First, the mean vector of all embedding vectors within the cluster is calculated as the cluster center, representing the semantic features of the cluster. The cosine similarity between each slogan within the cluster and the cluster center is calculated, and several slogans with the highest similarity are selected after sorting to ensure the representativeness of the samples. Second, multiple sequence alignment is performed on the selected slogan samples to extract common substrings. A multiple sequence alignment algorithm based on dynamic programming is used to perform character-level alignment on the slogans to identify common substrings. Finally, according to the semantic structure of the slogans, a regular expression is designed to cover variable parts such as version numbers and secondary fields, resulting in a device fingerprint in the form of a regular expression.

[0111] The structured fingerprint rule is a complete fingerprint containing device information and is used for library management. The device fingerprint in the form of a regular expression is the core part responsible for matching and is used for real-time identification. The combination of the two, the structured rule solves the "dynamic update of the fingerprint library", and the regular expression solves the "stable and efficient matching". They are stored and executed in a hierarchical decoupled manner, which not only adapts to the diversity of IoT devices but also ensures the identification performance.

[0112] Optionally, the format of the structured fingerprint rule is as follows:

[0113] 。

[0114] Optionally, after step S340, it further includes:

[0115] In step S341, the regular expression is applied to all slogans within the cluster, and the coverage rate is calculated. If the coverage rate is less than the fifth preset threshold, the samples are reselected or the regular expression rule is adjusted until the coverage rate is greater than or equal to the fifth preset threshold.

[0116] It can be understood that the regular expression is applied to all slogans within the cluster, and the proportion of slogans that match successfully is statistically analyzed to ensure that the coverage rate reaches a relatively high level. If the coverage rate is insufficient, the samples are reselected or the regular expression rule is adjusted until the requirements are met.

[0117] In step S400, the regular expression device fingerprint is compared with the existing fingerprint database, and the fingerprint database is updated.

[0118] It should be noted that the comparison between the regular expression device fingerprint and the existing fingerprint database is performed by semantic equivalence matching.

[0119] Furthermore, step S400 may include the following sub-steps:

[0120] In step S410, the newly generated regular expression fingerprints are semantically equivalent matched with the rules in the original fingerprint database one by one.

[0121] It can be understood that for each new regular fingerprint, it is compared with the rules in the library one by one: converted into a finite automaton to check whether the languages are equivalent.

[0122] In step S420, the fingerprints that match successfully inherit the corresponding device information in the database.

[0123] It can be understood that if they are equivalent, the device information is inherited, such as the manufacturer and model. There is no need to add new rules, avoiding duplicate storage and keeping the library concise.

[0124] In step S430, the fingerprint rules that do not match successfully are used as new rules to update the fingerprint rule library.

[0125] It can be understood that if the new regular expression is semantically non-equivalent to all the rules in the library, such as matching the sensor slogan of VendorY and there is no corresponding rule in the library, it is used as a new rule to be put into the library and its device information is bound. The device information is automatically obtained through the class cluster annotation in step 300.

[0126] Furthermore, in the present exemplary embodiment, an Internet of Things device fingerprint generation system is also provided for executing the above-mentioned Internet of Things device fingerprint generation method. Refer to Figure 6 As shown in, the system may include: a data collection module, an embedding generation module, a fingerprint generation module, and a fingerprint library update module.

[0127] The data collection module is used to collect the protocol slogan data with tags of the Internet of Things devices through a network scanning tool, and decompose the protocol slogan data into a byte-level sub-word data set through a word segmenter.

[0128] The embedding generation module is used to pre-train a masked language model through the byte-level sub-word data set, learn the semantic relationship between words and sub-word context, and optimize the stability of the embedding for dynamic content through the supervised contrastive loss fine-tuning method, and generate slogan-level embeddings suitable for downstream task device fingerprint recognition.

[0129] A fingerprint generation module, which is used to perform dimensionality reduction clustering on slogan-level embeddings to obtain device clusters, extract common substrings from each device cluster, and convert them into regular expression device fingerprints.

[0130] A fingerprint database update module, which is used to compare the regular expression device fingerprints with the existing fingerprint database and update the fingerprint database.

[0131] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only illustrative, and the true scope and spirit of the present disclosure are pointed out by the appended claims.

[0132] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. The present application is not limited to the exact structures already described and illustrated in the drawings, and it cannot be considered that the specific implementation of the present application is only limited to these descriptions. For those of ordinary skill in the technical field to which the present application belongs, various changes and deformations made without departing from the concept of the present application should be regarded as falling within the protection scope of the present application.< / ab> < / ab>

Claims

1. A method for generating fingerprints of Internet of Things devices, characterized in that, include: Collect the protocol slogan data with labels of IoT devices through network scanning tools, and decompose the protocol slogan data into byte-level subword data sets through word segmentation; The masked language model is pre-trained on a byte-level subword dataset to learn the relationship between word semantics and subword context, and the stability of embeddings to dynamic content is optimized through supervised contrastive loss fine-tuning to generate slogan-level embeddings suitable for downstream task device fingerprint recognition. Perform dimensionality reduction clustering on slogan-level embeddings to obtain device clusters, extract common substrings from each device cluster, and convert them into regular expression device fingerprints; Compare the regular expression device fingerprint with the existing fingerprint database and update the fingerprint database.

2. The method for generating the fingerprint of the Internet of Things device according to claim 1, wherein The step of collecting the protocol slogan data with labels of the IoT devices through the network scanning tool and decomposing the protocol slogan data into byte-level subword data sets through a word segmenter includes: Use a port scanning tool to scan the available ports of the IP address, combine with a protocol scanning tool to capture the protocol banner data, and obtain the labels corresponding to the protocol banner data through a network mapping tool to obtain the protocol banner data with labels; Preprocess the collected slogan data to remove special symbols and HTML structure symbols, and convert uppercase characters to lowercase; Train a byte-level tokenizer to decompose the labeled protocol slogan data into byte-level sub-words.

3. The method for generating the fingerprint of the Internet of Things device according to claim 2, wherein The step of training a byte-level word segmenter to decompose the protocol slogan data with labels into byte-level subwords includes: Decompose each slogan into a sequence of single characters and map them into a sequence of ASCII bytes; Initialize a vocabulary containing all single characters and count the frequencies of characters and adjacent byte pairs; Merge the most frequent byte pairs in the current dataset into new subwords and update the slogan dataset; The merging process is repeated until the vocabulary size reaches a first preset threshold or the number of merging times reaches a second preset threshold.

4. The method for generating the fingerprint of the Internet of Things device according to claim 1, wherein The steps of pre-training a masked language model through a byte-level sub-word dataset, learning the relationship between word semantics and sub-word context, and optimizing the stability of embedding to dynamic content through a supervised contrast loss fine-tuning method to generate a slogan-level embedding suitable for downstream task device fingerprint recognition include: Pre-train the masked language model on a byte-level subword dataset to learn the semantics and contextual relationships of subwords; Conduct multiple detections on the same IP address and port at different times, extract protocol slogan data and record them as slogan pairs, filter the dynamic content in the slogan pairs through regular expression rules, and generate static slogan data; The static slogan pairs generated by the same slogan pair are taken as positive sample pairs, slogans are extracted from different slogan pairs to generate negative sample pairs, and cosine similarity is used for pre-screening to construct positive and negative sample pairs; The subword embeddings of each slogan are generated through the pre-trained masked language model, and the embeddings of all subwords in the sequence are averaged to generate slogan-level embeddings suitable for downstream task device fingerprint recognition. The first loss function is constructed to bring the cosine similarity of the positive sample embeddings closer and push the similarity of the negative sample embeddings further away through the supervised contrast loss.

5. The method for generating the fingerprint of the Internet of Things device according to claim 4, wherein, The first loss function is: ； Among them, is the first loss function, is the embedding vector of the slogan , is the positive sample embedding of the slogan , is the embedding vector of the slogan , that is, the negative sample embedding of the slogan , Calculate the cosine similarity, is the number of positive sample pairs, is the temperature parameter.

6. The method for generating the fingerprint of the Internet of Things device according to claim 4, wherein, The masked language model is pre-trained using a cross entropy loss function; the cross entropy loss function is: ； Among them, is the cross-entropy loss function, M represents the set of masked subwords, is the true value of subword i, is the probability predicted by the model based on the context.

7. The method for generating an Internet of Things device fingerprint according to claim 1, wherein, The step of performing dimensionality reduction clustering on the slogan-level embedding to obtain device clusters and extracting common substrings from each device cluster and converting them into regular expression device fingerprints includes: Performing linear dimensionality reduction on the slogan-level embedding through principal component analysis and verifying by comparison in combination with uniform manifold approximation and projection non-linear dimensionality reduction; Using hierarchical density clustering to cluster the dimensionality-reduced slogan-level embedding to generate device clusters; Statistically analyzing the metadata label distribution of the slogans within each cluster and calculating the label consistency score; if the consistency score is higher than the fourth preset threshold, classifying and assigning the most common label to the cluster as the device information label of the cluster; Selecting samples from each cluster to extract common substrings, generating device fingerprints in the form of regular expressions, and combining with the device information labels of the clusters to generate structured fingerprint rules.

8. The method for generating the fingerprint of the Internet of Things device according to claim 7, wherein After the step of selecting samples from each cluster to extract common substrings and generating device fingerprints in the form of regular expressions, it further includes: Applying the regular expression to all the slogans within the cluster and calculating the coverage rate. If the coverage rate is less than the fifth preset threshold, reselecting samples or adjusting the regular expression rules until the coverage rate is greater than or equal to the fifth preset threshold.

9. The method for generating the fingerprint of the Internet of Things device according to claim 1, wherein The step of comparing the regular expression device fingerprints with the existing fingerprint database and updating the fingerprint database includes: Performing semantic equivalence matching on the newly generated regular expression fingerprints one by one with the rules in the original fingerprint database; The fingerprints that match successfully inherit the corresponding device information in the database; The fingerprint rules that do not match successfully are used as new rules to update the fingerprint rule library.

10. An Internet of Things device fingerprint generation system, characterized in that, The system is used to execute the Internet of Things device fingerprint generation method according to any one of claims 1 to 9, and the system includes: A data collection module for collecting protocol slogan data with labels of Internet of Things devices through a network scanning tool and decomposing the protocol slogan data into a byte-level sub-word data set through a tokenizer; An embedding generation module for pre-training a masked language model through the byte-level sub-word data set, learning the semantic relationship between words and sub-word context, and optimizing the stability of the embedding for dynamic content through the supervised contrast loss fine-tuning method to generate slogan-level embeddings suitable for downstream task device fingerprint recognition; A fingerprint generation module for performing dimensionality reduction clustering on the slogan-level embedding to obtain device clusters and extracting common substrings from each device cluster and converting them into regular expression device fingerprints; A fingerprint database update module for comparing the regular expression device fingerprints with the existing fingerprint database and updating the fingerprint database.

Images