Cluster-based network entity fingerprinting and network vulnerability cyber situational awareness

Abstract

This invention belongs to the field of cyberspace mapping, specifically involving a cluster-based method for network entity fingerprinting and network vulnerability situational awareness. First, it proposes a novel clustering-based network application fingerprinting method, constructing a database of known network node application type points. Through clustering algorithms, it efficiently classifies and labels collected network assets. Based on this, it automatically updates clustering features to discover new network entities and applications. Finally, it proposes a knowledge graph-based network vulnerability verification method, constructing a network entity information database, a vulnerability information database, and vulnerability verification knowledge atomic units. It designs and implements a vulnerability verification path generation algorithm to perform vulnerability verification on collected network assets, achieving automated vulnerability verification based on the knowledge graph. Ultimately, this results in a high-concurrency, multi-protocol, fast, and easily maintainable cyberspace asset detection system.

Description

Technical Field

[0001] This invention belongs to the field of cyberspace mapping, and more specifically, relates to a clustering-based method for network entity fingerprinting and network vulnerability situational awareness. Background Technology

[0002] Cyberspace mapping involves actively or passively probing network entities that provide open services on the internet, storing and analyzing the results to enable rapid retrieval of cyberspace assets. This helps researchers and enterprises quickly perform tasks such as network asset matching (including protocols, services, versions, and even device types), application distribution statistics, and management of critical information infrastructure. Furthermore, when vulnerabilities exist in specific versions of network services, cyberspace mapping technology can rapidly assess threat dynamics, determine the scope of vulnerability impact, and promptly patch vulnerable applications and services.

[0003] However, current cyberspace mapping technology still suffers from several critical shortcomings in its implementation: First, there's the efficiency issue. Current cyberspace mapping systems typically require deploying numerous probe nodes for large-scale scanning and analysis to complete the entire network mapping, incurring enormous operational and maintenance costs. Second, scanned and analyzed traffic is frequently mistaken for malicious attack traffic by firewalls and other protective measures, leading to scanning and detection failures. Third, cyberspace mapping systems face difficulties in identifying and verifying vulnerabilities across the entire network, and the accuracy is generally not high, requiring further manual verification, which is inefficient and lacks accuracy. Finally, new types of entities emerging in the network are difficult to detect in a timely manner, and their types and services are hard to accurately identify, resulting in low efficiency in detecting and identifying new types of network nodes through cyberspace mapping. These shortcomings limit the speed and accuracy of network entity fingerprinting. Summary of the Invention

[0004] To address the shortcomings and improvement needs of existing technologies, this invention provides a clustering-based method for network entity fingerprinting and network vulnerability situational awareness, with the aim of improving the efficiency of network entity fingerprinting.

[0005] To achieve the above objectives, according to one aspect of the present invention, a network entity fingerprinting method based on clustering is provided, comprising:

[0006] The network entity under test is probed and parsed to obtain and store its response information;

[0007] When the number of detected network entities reaches a preset number, density clustering is performed on the network entity response information sample library of known fingerprint tag clusters and the stored network entity response information of the network entities to be tested. Fingerprint tags are then applied to the network entities to be tested that fall into the known fingerprint tag clusters to achieve fingerprint recognition. The fingerprint includes application type information. The sample library is pre-constructed in the following manner:

[0008] The target network entity is probed and parsed to obtain its response information;

[0009] Based on the fingerprint rule base, the response information of the target network entity is subjected to rule-matching fingerprint recognition, and the target network entity is labeled with a fingerprint tag. When the number of target network entities with known fingerprint tags reaches a preset number, the response information of the target network entities with fingerprint tags is subjected to density clustering to generate a sample library of network entity response information with known fingerprint tag clusters.

[0010] Furthermore, the fingerprint also includes application version information.

[0011] Furthermore, when performing density clustering on the network entity response information sample library of known fingerprint tag clusters and the stored network entity response information to be tested, the method further includes:

[0012] If a new cluster without fingerprint tags is formed, the cluster is stored in the unlabeled network entity library. When the number of network entities in the unlabeled network entity library reaches a preset number, density clustering is performed on the response information of all network entities in the unlabeled network entity library. The fingerprints of each new entity are obtained by consulting the fingerprint information of the network entities, and fingerprint tags are assigned to each cluster. The clusters are then updated to the network entity response information sample library of known fingerprint tag clusters.

[0013] Furthermore, the detection is implemented as follows:

[0014] For a single IP address or a range of IP addresses, probe packets are sent according to various protocols, and the probe traffic is distributed to each network entity for synchronization information probe. The response results of each network entity are obtained and parsed to obtain the response information of each network entity.

[0015] Furthermore, the various protocols include HTTP, HTTPS, FTP, and SSH.

[0016] The present invention also provides a cyberspace mapping system, comprising:

[0017] The detection and parsing module is used to perform the detection and parsing operations in the clustering-based network entity fingerprinting method described above.

[0018] The first fingerprint recognition module based on clustering is used to perform fingerprint recognition operations on the network entities under test in a network entity fingerprint recognition method based on clustering as described above.

[0019] The second fingerprint recognition module, based on a fingerprint rule base, is used to construct a sample library of network entity response information of known fingerprint tag clusters in a clustering-based network entity fingerprint recognition method as described above.

[0020] This invention also provides a network vulnerability situation awareness method, comprising:

[0021] The above-described clustering-based network entity fingerprinting method is used to perform fingerprinting on each network entity to be perceived.

[0022] By combining the vulnerability database and the detection results of each network entity to be detected, an attack semantic path is constructed, and based on the attack semantic path, vulnerability verification is automatically performed on each network entity to be detected.

[0023] Furthermore, the attack semantic path is constructed as follows:

[0024] Based on the fingerprint information of each network node to be detected, and combined with the vulnerability database, a vulnerability number applicable to that network node is obtained.

[0025] Network entities are extracted from the response information and vulnerability numbers of the network nodes to be detected, vulnerability entities are extracted from the vulnerability database, and vulnerability attack relationships are constructed to associate network entities and vulnerability entities, wherein the network entity includes the vulnerability number;

[0026] For the extracted network entities, vulnerable entities, and attack relationships, an attack semantic path of {network entity--use--vulnerability--attack--network entity} is constructed by associating vulnerability numbers between entity attributes.

[0027] The network entity points, vulnerability entity elements, and use and attack behavior elements in the attack semantic path are represented by symbolic language to form a data description structure of network entities, vulnerability entities, use and attack behavior elements;

[0028] Based on the data description structure, vulnerability exploitation tools are selected from the vulnerable entities to launch vulnerability verification attacks on the network entities, thereby achieving automated verification of network entity vulnerabilities.

[0029] The present invention also provides a cyberspace mapping system, which is obtained by configuring a network entity vulnerability verification module on the cyberspace mapping system described above;

[0030] The network entity vulnerability verification module is used to execute a network vulnerability situational awareness method as described above.

[0031] The present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed by a processor, it controls the device where the storage medium is located to execute a clustering-based network entity fingerprinting method and / or a network vulnerability situational awareness method as described above.

[0032] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0033] (1) The method of the present invention pre-constructs a network entity response information sample library based on the fingerprint rule library and performs density clustering on it to obtain a network entity response information sample library with known fingerprint tag clusters. The sample library and the network entity response information to be fingerprinted are then used to perform density clustering again to perform fingerprint recognition on network entities in batches, which is fast.

[0034] (2) When some entities cannot be classified into existing clusters and the preset number is reached, density clustering can be performed independently on these network entities. New entities can be discovered by checking fingerprint tags and can be used to update the sample library, thereby improving the efficiency and accuracy of density clustering.

[0035] (3) This invention proposes an automated vulnerability verification method based on knowledge graphs. By constructing language attack paths and performing symbolic language representation on network entity points, vulnerability entity elements and use and attack behavior elements in semantic attack paths, a data description structure of network entities, vulnerability entities, use and attack behavior elements is formed. Based on the data description structure, vulnerability exploitation tools in vulnerability entities are selected to launch vulnerability verification attacks on network entities, thereby realizing automated verification of network entity vulnerabilities.

[0036] (4) In response to the above fingerprint recognition method and vulnerability verification method, the present invention designs a highly intelligent and concurrent network space mapping system. The task execution process basically does not require manual intervention. It supports scanning and analysis of multiple protocols such as HTTP, HTTPS, SSH, and FTP. It can be deployed in a distributed manner, and there is huge room for efficiency improvement. Attached Figure Description

[0037] Figure 1 A flowchart illustrating a cluster-based fingerprint recognition method provided in an embodiment of the present invention;

[0038] Figure 2 This is an overall flowchart of the network space mapping system performing mapping according to an embodiment of the present invention;

[0039] Figure 3 Example diagram of the data structure of network entities and vulnerability entities provided in the embodiments of the present invention;

[0040] Figure 4 Example diagram of entity construction attack path provided in embodiments of the present invention;

[0041] Figure 5 A block diagram of a cyberspace mapping system provided in an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of a vulnerability verification knowledge unit provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0044] Example 1

[0045] A clustering-based network entity fingerprinting method includes:

[0046] The network entity under test is probed and parsed to obtain and store its response information;

[0047] When the number of detected network entities reaches a preset number, density clustering is performed on the network entity response information sample library of known fingerprint tag clusters and the aforementioned stored network entity response information of the network entities to be tested. Fingerprint tags are then assigned to the network entities to be tested that fall into the known fingerprint tag clusters to achieve fingerprint recognition. The fingerprint includes application type information. The aforementioned sample library is pre-constructed using the following method:

[0048] The target network entity is probed and parsed to obtain its response information;

[0049] Based on the fingerprint rule base, the response information of the target network entity is subjected to rule-matching fingerprint recognition, and the target network entity is labeled with a fingerprint tag. When the number of target network entities with known fingerprint tags reaches a preset number, the response information of the target network entities with fingerprint tags is subjected to density clustering to generate a sample library of network entity response information with known fingerprint tag clusters.

[0050] This embodiment can be implemented using the Rust language with event-driven asynchronous runtime scheduling technology.

[0051] like Figure 1As shown, this embodiment proposes a novel network application fingerprinting method based on clustering. It constructs a sample library of network entity response information with known fingerprint tag clusters, and can achieve classification and tagging of collected network entities through the DBSCAN equal-density clustering algorithm.

[0052] Preferably, the fingerprint also includes application version information for more accurate identification.

[0053] Preferably, the above detection is implemented as follows:

[0054] For a single IP address or a range of IP network entities, probe data packets are sent according to various protocols, and the probe traffic is distributed to each network entity for synchronous information probe. The response results of each network entity are obtained and the response information of each network entity is parsed, achieving high concurrency, multi-protocol, and fast probe.

[0055] Preferably, when performing density clustering on the network entity response information sample library of known fingerprint tag clusters and the stored network entity response information to be tested, the method further includes:

[0056] If a new cluster without fingerprint tags is formed, the cluster is stored in the unlabeled network entity library. When the number of network entities in the unlabeled network entity library reaches a preset number, density clustering is performed on the response information of all network entities in the unlabeled network entity library. The fingerprints of each new entity are obtained by consulting the fingerprint information of the network entities, and fingerprint tags are assigned to each cluster. The clusters are then updated to the network entity response information sample library of known fingerprint tag clusters.

[0057] This embodiment also discovers new network entities by collecting network entity response information, and realizes the discovery of new network entities and new applications by automatically updating clustering features.

[0058] Example 2

[0059] A cyberspace mapping system, comprising:

[0060] The detection and parsing module is used to perform the detection and parsing operations in the clustering-based network entity fingerprinting method as described in Embodiment 1.

[0061] The first fingerprint recognition module based on clustering is used to perform fingerprint recognition operations on the network entity to be tested in a network entity fingerprint recognition method based on clustering as described in Embodiment 1.

[0062] The second fingerprint recognition module, based on a fingerprint rule base, is used to perform the construction operation of a network entity response information sample library with known fingerprint tag clusters in a clustering-based network entity fingerprint recognition method as described in Embodiment 1.

[0063] Specifically, this system can be divided into interconnected master nodes, slave nodes, a transaction cache database, a result storage database, and an ES big data search engine. In particular, the master node includes an application interface module and two task scheduling modules, while the slave nodes include a task scheduling unit, a detection and scanning unit, a service analysis unit, and a vulnerability verification unit.

[0064] like Figure 2 As shown, the system first performs task allocation and scheduling: mainly managing and scheduling scanning and detection tasks as well as analysis, identification, and vulnerability verification tasks. The system administrator issues specific scanning and analysis tasks by calling the application interface module of the master node. After the slave nodes retrieve the tasks through the transaction cache database, the task scheduling unit distributes the corresponding tasks to the specific functional units for execution.

[0065] The system then performs high-concurrency scanning and probing: primarily implementing high-concurrency scanning and probing of target network nodes. After a node obtains a scanning task from the scanning task queue, it uses the probing module to send probe data packets to network nodes with single IP addresses or IP ranges according to various protocol specifications. The probing module distributes the probe traffic to remote network nodes for information probing, and the probe response results are sent to the result parsing module for analysis.

[0066] The system further performs detection result parsing: analyzing the network response results collected during the detection. The result parsing module parses the detection results according to the scanning protocol format, extracts the corresponding content of the protocol fields in the detection results, and saves the extracted content to the result storage database.

[0067] The system further performs a novel cluster-based fingerprint recognition: it realizes fingerprint recognition of the network node detection and parsing results, and achieves fingerprint recognition of the application type of the network node through a novel cluster-based fingerprint recognition method as described in Example 1.

[0068] It should be noted that, prior to this, a sample library of network entity response information with known fingerprint tag clusters needs to be constructed in advance: the service analysis unit obtains the parsing results of the network nodes to be analyzed from the result storage database, constructs a dataset of network node information types through fingerprint preprocessing, and then uses density clustering to realize fingerprint recognition of the application type and version of the network node detection data, thereby obtaining a sample library of network entity response information with known fingerprint tag clusters.

[0069] The system further performs cluster-based discovery of new network entities and automatic updating of entity type fingerprints: For network node result data in the service analysis unit that failed to identify fingerprints and perform fingerprint labeling through cluster-based fingerprint recognition, clustering is used to realize the discovery of new network entities and automatic updating of network node application type fingerprints.

[0070] This embodiment designs and implements a cyberspace asset detection engine, which uses the Rust language and event-driven asynchronous runtime scheduling technology to form a high-concurrency, multi-protocol, fast, and easy-to-maintain cyberspace asset detection system.

[0071] Example 3

[0072] A network vulnerability situational awareness method, comprising:

[0073] A clustering-based network entity fingerprinting method, as described in Example 1, is used to perform fingerprinting on each network entity to be sensed.

[0074] By combining the vulnerability database and the detection results of each network entity to be detected, an attack semantic path is constructed, and based on the attack semantic path, vulnerability verification is automatically performed on each network entity to be detected.

[0075] Preferably, the above-mentioned attack semantic path is constructed as follows:

[0076] Based on the fingerprint information of each network node to be detected, and combined with the vulnerability database, a vulnerability number applicable to that network node is obtained.

[0077] Network entities are extracted from the response information and vulnerability numbers of the network nodes to be detected, vulnerability entities are extracted from the vulnerability database, and vulnerability attack relationships are constructed to associate network entities and vulnerability entities, wherein the network entity includes the vulnerability number;

[0078] For the extracted network entities, vulnerable entities, and attack relationships, an attack semantic path of {network entity--use--vulnerability--attack--network entity} is constructed by associating vulnerability numbers between entity attributes, thus forming the attack path, such as... Figure 3 As shown;

[0079] Symbolic language representations are used to represent network entity points, vulnerable entity elements, and use / attack behavior elements in semantic attack paths, forming a data description structure for network entities, vulnerable entities, use / attack behavior elements, such as... Figure 4 As shown;

[0080] Based on the data description structure, vulnerability exploitation tools are selected from the vulnerable entities to launch vulnerability verification attacks on the network entities, thereby achieving automated verification of network entity vulnerabilities.

[0081] This embodiment proposes a network vulnerability verification method based on knowledge graphs. It constructs a network entity information database, a vulnerability information database, and vulnerability verification knowledge atomic units, designs and implements a vulnerability verification path generation algorithm, performs vulnerability verification on collected network assets, and realizes automated vulnerability verification based on knowledge graphs.

[0082] Example 4

[0083] A cyberspace mapping system is obtained by configuring a network entity vulnerability verification module on the cyberspace mapping system as described in Embodiment 2.

[0084] The network entity vulnerability verification module is used to execute a network vulnerability situational awareness method as described in Embodiment 3.

[0085] In other words, based on Example 2, the system further performs automated vulnerability verification based on knowledge graphs: for network nodes that have been tagged by fingerprint recognition, the system combines the vulnerability database and the network node detection result database with knowledge graph entity extraction and relationship construction methods to construct attack semantic paths, thereby achieving automated vulnerability verification of network target nodes.

[0086] Finally, the results can be displayed based on a big data search engine: use a big data search engine to query massive amounts of data in the results storage database, and display the task execution results to the administrator through a visual interface.

[0087] To better illustrate the cyberspace mapping system and its operation as described in this invention, the following examples are provided:

[0088] System as a whole Figure 5 The system includes an interactive master node, a transaction cache database 3, slave nodes, a result storage database, and an ES big data search engine.

[0089] Its working mechanism is:

[0090] The administrator issues scanning or analysis commands through the application interface module of the master node. Then, the master node 200 interacts with the transaction cache database to manage the scanning task queue 310 and the analysis task queue within it. At the same time, the slave nodes continuously read tasks from the transaction cache database 300. According to the task type, the task scheduling unit in the slave node dispatches the task to the corresponding execution unit. After the task is executed, the slave node saves the results to the result storage database. Finally, the administrator accesses the application interface of the master node to query and obtain the command execution results from the ES big data search engine.

[0091] The following describes each functional unit:

[0092] (1) The main node includes the application interface module, the scanning task scheduling module, and the analysis task scheduling module.

[0093] The application interface module refers to a collection of application interfaces registered by the master node that combine internal meta-functions to implement complex operations. Its function is to provide an operation interface to the administrator.

[0094] The scan task scheduling module is a program that interacts with the transaction cache database and its function is to manage the scan task queue.

[0095] The analysis task scheduling module is a program that interacts with the transaction cache database and its function is to manage the analysis task queue.

[0096] Working mechanism:

[0097] Administrators issue commands through the operation interface provided by the application interface module. These commands mainly fall into two categories: task management (add or remove) commands and query commands. The former will cause the master node to call the scanning task scheduling module and the analysis task scheduling module to manage the corresponding type of task queue, while the latter will use the ES big data search engine to query the results, store them in the database, and display the results to the administrator.

[0098] (2) Transaction Cache Database

[0099] Transactional cache databases refer to non-relational databases such as Redis, which internally maintain two task queues: a scan task queue and an analysis task queue.

[0100] (3) The nodes include a task scheduling unit, a detection and scanning unit, a service analysis unit, and a vulnerability verification unit.

[0101] The task scheduling unit is a program that retrieves a list of tasks and assigns tasks. Its function is to continuously retrieve tasks from the transaction cache database and then assign them to the corresponding task execution units according to the task type.

[0102] The probe scanning unit comprises a probe module and a result parsing module that interact sequentially. The probe module is a program that sends data packets to the target network node using various protocols (such as HTTP, HTTPS, FTP, SSH, etc.). Its function is to obtain the target node's network response results for subsequent parsing. The result parsing module is a program that parses the target network node's response according to protocol standards. Its purpose is to confirm the target network node's operating status and the specific protocol running on the node's port, and extract data from the response results according to the protocol format, outputting it to the scan dataset. Working mechanism: The probe module is responsible for sending data packets conforming to protocol standards to the target node. After obtaining the target node's response, the result parsing module parses it to obtain the target network node's status and the protocol running on its port, and outputs the parsing results to the scan dataset.

[0103] The service analysis unit includes a fingerprint rule base, a fingerprint recognition module, a data training set, a classifier, and new entity discovery. The fingerprint rule base is a database storing regular expression rules for common application services under various protocols. Its function is to be used by the fingerprint recognition module to perform regular expression-based fingerprint recognition of application services running on target network nodes. The fingerprint recognition module is a program that reads the scan results of successfully detected network nodes from the results storage database 500 and performs fingerprint recognition on them. Its function is to determine the application services and versions running on the target network nodes for further vulnerability analysis. The data training set refers to a dataset composed of the results successfully recognized by the fingerprint recognition module. Its function is to train the classifier to perform cluster-based fingerprint recognition on the network node data to be tested. The classifier is a classification model formed by a clustering algorithm based on the data training set. Its function is to cluster the fingerprint label dataset identified by the fingerprint recognition module with the data to be tested, and achieve fingerprint recognition by labeling the clusters. New entity discovery refers to the clustering and labeling of new entities from the classification results of nodes not identified by the fingerprint recognition module obtained from the classifier.

[0104] The vulnerability verification unit is a program that automates vulnerability verification of service information obtained by the service analysis unit. Its function is to identify and verify vulnerabilities present in services running on a target address. This includes vulnerability entity extraction, asset entity extraction, vulnerability verification path generation, and a vulnerability verification module. Its working mechanism is as follows: the vulnerability verification unit extracts vulnerability entities based on knowledge graph technology, extracts asset entities from the assets obtained by the service analysis unit, constructs a vulnerability verification semantic path, and uses the vulnerability verification module to automate vulnerability verification of network assets.

[0105] Vulnerability entity extraction refers to extracting attribute entities from vulnerability information in the vulnerability database to construct vulnerability entity units for vulnerability tool recommendation during vulnerability verification. Asset entity extraction refers to extracting attribute entities from tagged asset nodes in the service analysis unit to construct asset entity units for building vulnerability verification semantic paths. Vulnerability verification path generation refers to combining the vulnerability entities and asset entities extracted by vulnerability entity extraction and asset entity extraction to generate vulnerability verification semantic paths using attack semantics, which guide the vulnerability verification module in performing vulnerability verification. The vulnerability verification module uses the vulnerability verification semantic paths generated in vulnerability verification path generation to select vulnerability entities and asset entities for vulnerability verification.

[0106] (4) Result storage database refers to a non-relational database such as MongoDB, whose function is to store the results of task execution in the scan dataset and the analysis dataset.

[0107] (5) ES Big Data Engine is a program built on the ElasticSearch framework. Its function is to perform real-time search and analysis on massive amounts of data in the result storage database.

[0108] The following is combined with, for example Figure 1 , Figure 2 , Figure 6 The specific implementation steps of the method of the present invention will be described.

[0109] First, the system administrator issues specific scanning and analysis tasks by calling the application interface module of the master node. After receiving the scanning and analysis tasks, the master node interacts with the transaction cache database to save the tasks in the scanning and analysis task queue. The slave nodes continuously read the tasks from the scanning and analysis task queue of the transaction cache database and dispatch the tasks to the detection scanning unit, service analysis unit, and vulnerability verification unit through their own task scheduling unit to execute specific functions.

[0110] Secondly, after the node obtains a scanning task from the scanning task queue, it uses the probe module to send probe data packets to network nodes with a single IP address or IP range using various protocols (HTTP, HTTPS, FTP, SSH). The probe module distributes the probe traffic to remote network nodes for information probing, and the response results obtained from the probe are sent to the result parsing module for parsing.

[0111] Furthermore, the detection and scanning unit is implemented using Rust with event-driven asynchronous runtime scheduling technology, enabling high-concurrency port information collection for common protocols such as HTTP, HTTPS, FTP, and SSH. A single-core network node can scan up to 65,535 network connections at a scanning rate of up to 200 IPs / s, achieving high-speed network asset detection. The information collected mainly includes IP liveness status, domain names, port openness, port service information, port banners, and port protocol interaction information.

[0112] Furthermore, the result parsing module parses the detection results according to the scanning protocol format, extracts the corresponding content of the protocol fields in the detection results, and saves the extracted content to the result storage database.

[0113] Further, refer to Figure 1First, the fingerprint recognition module uses the system's built-in fingerprint rule base to perform rule-based fingerprint recognition on the scanned and parsed results of the target nodes, obtaining the specific application type and application version information of the target nodes, and saving the fingerprint recognition results to the corresponding network node information, thus labeling the nodes with fingerprint tags. Second, a network node information type dataset is constructed. When the fingerprint recognition results reach 10,000, the labeled network node parsing results are subjected to density clustering using DBSCAN to generate a network node information type dataset with known labels. Finally, intelligent fingerprint recognition is achieved using the network node information type dataset. When 10,000 new probe analysis results are generated, these 10,000 new data are clustered again with the network node information type dataset with known labels using DBSCAN. The new data falling into the known label clusters are labeled with the application fingerprint tag corresponding to the cluster, realizing rapid fingerprint recognition of the probe parsing results.

[0114] Furthermore, the novel clustering-based fingerprinting method fails to identify network nodes that will form new unlabeled clusters after fingerprinting clustering. The detection results data in these clusters cannot be fingerprinted using known type network node information datasets and rule-based matching methods. Such data is considered a new application type not in the fingerprint rule base. When the number of nodes in a single unknown cluster reaches 200, the node application type in that cluster is considered a new network entity type, thus achieving the discovery of new network entities. For the discovered new entity types, the clusters are labeled by manually checking the application types. The clusters are then updated to the known labeled network node information dataset for continued novel clustering-based fingerprinting, thus achieving automated updating of cluster fingerprints and effectively solving the problem of difficult fingerprint rule extraction in rule-based fingerprinting methods.

[0115] Furthermore, vulnerability information is identified and populated into the network node detection results that have already undergone fingerprinting. By comparing the identified node service application type and version fingerprint information with the applicable application and service version information in the vulnerability database, the vulnerability status of the network node is determined. The vulnerability numbers applicable to the network node's applications and services are then populated into the network node detection result database, labeling the network node with vulnerability information. Entity extraction and attack relationship construction are performed on the vulnerability database and the network node detection result database, extracting network node entities and vulnerability entities. These entities are then linked through vulnerability attack relationships to form knowledge units (see reference). Figure 4The corresponding expression is ({network node, vulnerability}, vulnerability attack). The network node entity attributes are defined as node type, IP address, operating system, port, service and version, and vulnerability number. The vulnerability entity attributes are defined as vulnerability benefit, vulnerability number, vulnerability exploitation tool, and vulnerability score. Network node entities and vulnerability entities are associated through vulnerability attack relationships. A vulnerability attack semantic path is constructed: for the extracted network node entities, vulnerability entities, and attack relationships, an attack semantic path of {network node--use--vulnerability--attack--network node} is constructed through the association of vulnerability numbers between entity attributes, forming an attack path. Based on the attack semantic path, the vulnerability exploitation tool in the vulnerability entity is selected to launch a vulnerability verification attack on the network node entity, achieving automated verification of network node vulnerabilities.

[0116] Furthermore, the network node detection and scanning results, fingerprint recognition results, and vulnerability verification results are indexed and managed using the big data search engine ElasticSearch, enabling the querying of massive amounts of data in the results storage database. The Vue framework is used for page development, and the task execution results are displayed to the administrator through a visual interface.

[0117] Example 5

[0118] A computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, it controls the device where the storage medium is located to perform a clustering-based network entity fingerprinting method as described in Embodiment 1 and / or a network vulnerability situational awareness method as described in Embodiment 3.

[0119] The relevant technical solutions are the same as those in Embodiment 1 and Embodiment 3, and will not be repeated here.

[0120] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cluster-based network entity fingerprinting method, characterized in that, The method comprises the following steps: probing and analyzing the network entities to be tested, obtaining and storing the response information of the network entities to be tested; when the number of the network entities to be tested reaches a preset number, performing density clustering on the network entity response information sample library of the known fingerprint label cluster and the stored response information of the network entities to be tested, and labeling the network entities to be tested in the known fingerprint label cluster to realize fingerprint identification, wherein the fingerprint comprises application type information, and the sample library is pre-constructed in the following manner; probing and analyzing the target network entities to obtain the response information of the target network entities; performing rule-based fingerprint identification on the response information of the target network entities according to a fingerprint rule library, and labeling the target network entities with a fingerprint label; when the number of the target network entities with the known fingerprint label reaches a preset number, performing density clustering on the response information of the target network entities with the known fingerprint label to generate a network entity response information sample library of the known fingerprint label cluster; wherein, when the network entity response information sample library of the known fingerprint label cluster and the stored response information of the network entities to be tested are subjected to density clustering, the method further comprises the following steps: if a new cluster without a fingerprint label is formed, the cluster is stored in a label-free network entity library, and when the number of the network entities in the label-free network entity library reaches a preset number, the response information of all the network entities in the label-free network entity library is subjected to density clustering, and the fingerprints of the new entities are obtained by referring to the network entity fingerprint information, and the clusters are labeled with a fingerprint label, and the clusters are updated to the network entity response information sample library of the known fingerprint label cluster.

2. The network entity fingerprinting method of claim 1, wherein, The fingerprint further comprises application version information.

3. The network entity fingerprinting method of claim 1, wherein, The implementation of the probing is as follows: for a single IP address or IP range network entity, a plurality of protocol-specified probe data packets are sent respectively, and the probe traffic is distributed to each network entity for synchronous information probing, the response results of each network entity are obtained by probing, and the response information of each network entity is obtained by analyzing.

4. The network entity fingerprinting method of claim 3, wherein, The plurality of protocols comprise HTTP, HTTPS, FTP and SSH.

5. A clustering-based network entity fingerprinting system, characterized in that, The method comprises the following steps: a first probing and analyzing module for probing and analyzing the network entities to be tested, obtaining and storing the response information of the network entities to be tested; a first fingerprint identification module based on clustering for performing density clustering on the network entity response information sample library of the known fingerprint label cluster and the stored response information of the network entities to be tested when the number of the network entities to be tested reaches a preset number, and labeling the network entities to be tested in the known fingerprint label cluster to realize fingerprint identification, wherein the fingerprint comprises application type information, and the sample library is pre-constructed in the following manner; a second probing and analyzing module for probing and analyzing the target network entities to obtain the response information of the target network entities; a fingerprint rule library for performing rule-based fingerprint identification on the response information of the target network entities, and labeling the target network entities with a fingerprint label; when the number of the target network entities with the known fingerprint label reaches a preset number, performing density clustering on the response information of the target network entities with the known fingerprint label to generate a network entity response information sample library of the known fingerprint label cluster; The second fingerprint identification module based on the fingerprint rule base is configured to perform rule matching-based fingerprint identification on response information of the target network entity according to the fingerprint rule base, and label the target network entity with a fingerprint tag; when a preset number of target network entities with known fingerprint tags are reached, the response information of the target network entities with the fingerprint tags is subjected to density clustering to generate a network entity response information sample library of known fingerprint tag clusters.

6. A cyber vulnerability posture awareness method, comprising: The method comprises the following steps: The fingerprint identification module is configured to perform fingerprint identification on each network entity to be perceived by using the clustering-based network entity fingerprint identification method according to any one of claims 1 to 4. The vulnerability verification module is configured to construct an attack semantic path based on a vulnerability library and detection results of each network entity to be perceived, and automatically perform vulnerability verification on each network entity to be perceived based on the attack semantic path.

7. The cyber vulnerability situation awareness method of claim 6, wherein, The attack semantic path is constructed in the following manner: According to the fingerprint information of each network node to be perceived, the vulnerability numbers suitable for the network node to be perceived are obtained in combination with the vulnerability library. Network entities are extracted from the response information of the network node to be perceived and the vulnerability numbers thereof, vulnerability entities are extracted from the vulnerability library, and a vulnerability attack relationship is constructed to associate the network entities and the vulnerability entities, wherein the network entities include the vulnerability numbers. The extracted network entities, vulnerability entities and attack relationships are associated through the vulnerability numbers between entity attributes to construct an attack semantic path of {network entity--use--vulnerability--attack--network entity}. The network entity, vulnerability entity, use and attack behavior elements in the attack semantic path are symbolically represented to form a data description structure of the network entity, vulnerability entity, use and attack behavior elements. Based on the data description structure, a vulnerability exploitation tool in the vulnerability entity is selected to initiate a vulnerability verification attack on the network entity, thereby realizing automatic verification of the vulnerability of the network entity.

8. A cyber vulnerability posture awareness system, comprising: The fingerprint identification module is configured to perform fingerprint identification on each network entity to be perceived by using the clustering-based network entity fingerprint identification method according to any one of claims 1 to 4. The vulnerability verification module is configured to construct an attack semantic path based on a vulnerability library and detection results of each network entity to be perceived, and automatically perform vulnerability verification on each network entity to be perceived based on the attack semantic path. The attack semantic path is constructed in the following manner:

9. The cyber vulnerability situation awareness system of claim 8, wherein, According to the fingerprint information of each network node to be perceived, the vulnerability numbers suitable for the network node to be perceived are obtained in combination with the vulnerability library. Network entities are extracted from the response information of the network node to be perceived and the vulnerability numbers thereof, vulnerability entities are extracted from the vulnerability library, and a vulnerability attack relationship is constructed to associate the network entities and the vulnerability entities, wherein the network entities include the vulnerability numbers. The extracted network entities, vulnerability entities and attack relationships are associated through the vulnerability numbers between entity attributes to construct an attack semantic path of {network entity--use--vulnerability--attack--network entity}. The network entity, vulnerability entity, use and attack behavior elements in the attack semantic path are symbolically represented to form a data description structure of the network entity, vulnerability entity, use and attack behavior elements. The network entity point, the vulnerability entity element and the use and attack behavior element in the attack semantic path are symbolically characterized by a language, and a data description structure of the network entity, the vulnerability entity, the use and the attack behavior element is formed; Based on the data description structure, a vulnerability exploitation tool in the vulnerability entity is selected, and a vulnerability verification attack is initiated on the network entity, so as to realize automatic verification of the network entity vulnerability.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein when the computer program is run by a processor, the device where the storage medium is located is controlled to execute the cluster-based network entity fingerprint identification method according to any one of claims 1 to 4 and / or the network vulnerability situation awareness method according to claim 6 or 7.

Images