Improved research of network asset surveying and mapping technology based on flow analysis

Abstract

The invention provides an asset surveying and mapping technology improvement method based on combined scanning and mirror image traffic analysis. According to the method, the port in the traffic is detected and analyzed by using the python-scapy in the combined scanning, and the survival port is deeply detected by using the python-nmap, so that a multi-thread effect is realized, the port detection time is shortened, and more port information is obtained. Tests show that compared with TCP scanning, the method has the advantages that the speed is increased by more than 60%, and the system consumption is small; flow analysis asset information is used as a supplement of combined scanning, so that assets are more comprehensive, and the port identification accuracy reaches 95% or above.

Description

Technical Field

[0001] The present invention belongs to the field of network space mapping, and specifically relates to an improved study of network asset mapping technology based on traffic analysis. Background Art

[0002] With the development of network technology, the resources in cyberspace are becoming more and more abundant, including traditional hardware and software infrastructure such as equipment and logical topology, as well as dynamic and changeable virtual resources such as network users and application services. Since the boundaries of cyberspace are relatively vague, it is necessary to use technical means to map out a "cyberspace map" similar to a geographical map (Lori Perri. What's New in the 2022 Gartner Hype Cycle for Emerging Technologies [EB / OL]. (2022-08-10) [2023-09-23, due to the current severe situation of network security and the frequent occurrence of various network security incidents (Guo Li, Cao Yanan, Su Majing, et al. Cyberspace Resource Mapping: Concepts and Technologies [J]. Journal of Information Security, 2018, 3 (04).), reasonably mapping out cyberspace maps is conducive to effectively maintaining the security of network assets, and at the same time it also has the effect of preventing network attacks. Therefore, cyberspace mapping technology is born.

[0003] Cyberspace mapping is to measure, integrate, analyze, and infer the various components of cyberspace and their relationships to discover the evolution of cyberspace and draw a holographic network map to comprehensively describe and display cyberspace information, and can provide data and technical support for various applications (such as network asset discovery and evaluation, equipment vulnerability detection, asset security operation management, etc.) (Chen Qing, Li Han, Du Yuejin, Zhang Yirong. Practice and thinking of cyberspace mapping technology [J]. Information and Communication Technology and Policy, 2021, 47(8): 30-38.). Therefore, the use of cyberspace resource mapping technology and the comprehensive mapping of cyberspace characteristics and resource distribution are of great significance and value for promoting national economic development and ensuring national security.

[0004] After reviewing the Docker-related information in recent years, Docker deployment-cyberspace mapping was proposed for the first time. According to the Gartner report (Lori Perri. What's New in the 2022 Gartner Hype Cycle for Emerging Technologies [EB / OL]. (2022-08-10) [2023-09-23]), ASM (attack surface management) may become the hottest segment of network security in the next three to five years. Asset mapping, as the basis of attack surface management, has also become a hot topic in the field of network security. Some scholars have elaborated on the concept and technical system of asset mapping (Guo Li, Cao Yanan, Su Majing, et al. Cyberspace Resource Mapping: Concepts and Technologies [J]. Journal of Information Security, 2018, 3(04).), and explored the key technologies of asset mapping from the three levels of detection, analysis and mapping. Some scholars have also put forward specific suggestions on the application of network asset mapping technology from the perspectives of asset management and attack surface discovery (Chen Qing, Li Han, Du Yuejin, Zhang Yirong. Practice and Thinking of Cyberspace Mapping Technology [J]. Information and Communication Technology and Policy, 2021, 47(8): 30-38.). Summary of the invention

[0005] Purpose of the invention: To solve the problems of traditional scanners that cannot quickly map large quantities of network assets, cannot dynamically perceive asset risks, and have a single adaptation scenario. This invention proposes an improved method for asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis to improve the work efficiency of mapping large quantities of network assets and the accuracy of port identification.

[0006] Technical solution: Asynchronous scanning traffic analysis uses python-nmap for scanning and detection, scapy analyzes assets in traffic, and asyncio is implemented asynchronously; mirror traffic analysis captures traffic from the network card, uses filebeat to generate logs and filter asset information. It includes the following steps:

[0007] Step 1: Use the cyberspace resource mapping system, which is a cyclic process integrating detection, analysis, mapping and application, to collaboratively detect cyberspace resources, obtain data, conduct fusion analysis, form a cyberspace resource library and draw a holographic map;

[0008] Table 1: Asset Mapping System Function Table

[0009] Image Name Version Function redis latest Task dispatch and data recovery cache

[10] postgres latest Permanent storage of surveying and mapping data scanner latest Custom build, integrated nmap / masscan, etc. apache latest Web page service filebeat 8.0.0 Forward the logs obtained by zeek centos 7.9.2009 Main image, Python environment, asynchronous processing

[11]

[0010] The functions of the asset mapping system are shown in Table 1, which is an improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis.

[0011] Step 2: Map active assets through IP / port scanning, and obtain information such as asset survival and port opening, and implement functions such as ping survival detection, TCP full connection, SYN half connection, FIN, Null and UDP scanning, including docker scan image integration Nmap and Masscan, etc.; in terms of working mode, it can be divided into connection-oriented (synchronous mode, standard TCP three-way handshake) and connectionless (asynchronous mode, asynchronous sending and receiving mode), etc., including nmap asynchronous scanning, etc.;

[0012] Table 2: nmap asynchronous scanning table

[0013] Use the nmap.PortScannerAslync() class method in the python-nmap module import nmap nm = nmap.PortScannerAslync() nm.scan("192.168.1.2","20-200","-sS")

[0014] Table 2 is an example of nmap asynchronous scanning. In the embodiment, the nmap.PortScannerAsync() class method in the python-nmap module is used to perform asynchronous port scanning. Through this method, non-blocking port scanning of a specified IP address can be achieved, and a callback function is called to process the result after the scanning is completed.

[0015] In the specific implementation, the nm.scan("192.168.1.2", "20-200", "-sS", callback = callback_result command is used to perform a SYN scan on the port range from 20 to 200 of the IP address 192.168.1.2, thereby quickly detecting the status of the port. This asynchronous processing method ensures the efficient operation of the system during the scanning process and does not block other operations of the main process.

[0016] Step 3: Using the unique fingerprint features of the asset, the detailed information of the device, system, service and application to which the asset belongs is identified. The system combines the advantages of Whatweb and Wappalyzer, interacts with the target Web server by constructing HTTP requests, extracts fingerprint feature information from the response data packet information, and compares it with the fingerprint database, thereby obtaining the component information and version information of the Web server and application, such as the script=banner mode of the Nmap system of the present invention;

[0017] Table 3: Python3 recognizes CMS type function instance table

[0018]

[0019] Table 3 is an example of python3 identifying CMS type functions. Use the nmap tool to scan the specified IP address and port, and the execution command is nmap-sT-sV-T4--script=banner-n 192.168.1.1-p 5672. This command performs service and version detection on the target end 5672 through TCP connection. The result shows that port 5672 / tcp is open, the service is amqp, and the version is RabbitMQ 3.9.7 (0-9). At the same time, by parsing the returned Www-Authenticat header, the application name and version of the firewall are extracted. .*? (?P<app_name> PEPLINK (?P<app_version> [\w / \s]+)VPN[- / ]{1}Firewall), which is the regular expression for parsing. And use the re.compile(reg,re.I).search(reg) method to match, and finally get the application name PEPLINK NG500 VPN-Firewall and application version NG500.

[0020] In addition, in the detected application information, the category list corresponding to the application name is returned through the get_categories method, and the web page content is analyzed through the analyze method to return the detected application list.

[0021] Step 4: Vulnerabilities of mapped assets are discovered through vulnerability scanning, including version vulnerabilities and principle vulnerabilities (PoC vulnerabilities). The CPE number of the application version of the mapped asset is compared with the CPE number of the version with public vulnerabilities. If the CPE is in the list of CPEs with vulnerabilities, it proves that a version vulnerability exists. Principle vulnerabilities are detected through PoC plug-in scripts. If the mapped asset hits certain rules in the script and confirmation information is returned, it proves that a principle vulnerability exists.

[0022] Table 4: Router POC script example

[0023]

[0024] Table 4 is an example of a router POC script. The vulnerability of the target device is verified by sending an HTTP request (such as a GET request GET / status / status_deviceinfo.htm HTTP / 1.1). The request contains the authentication information of the target device (such as Authorization: BasicYWRtaW46YWRtaW4=), and detects potential vulnerabilities by obtaining device information. In addition, the PoC (Proof of Concept) plug-in script is used to detect principle-based vulnerabilities, and then the version-type vulnerabilities are verified by the CPE number of the device. This method can effectively identify the security risks of the device and provide further repair suggestions to ensure the security of the network environment.

[0025] Step 5: Perform traffic analysis by sniffing the data packets flowing in the network, analyze the protocol and content of the data packets, and implement passive detection; at the hardware layer, the asset type can be identified from the hardware differences of the devices; at the network layer and transport layer, the protocol header field can be used to identify the operating system and service port information; at the application layer, the fingerprint detection of application components or software can be implemented by deeply analyzing the load data or feature fields in the protocol;

[0026] Table 5: docker-compose.yml file table of filebeat

[0027]

[0028] Table 5 is an example of the docker-compose.yml file of filebeat. The system of the present invention first captures the traffic passing through the network card by the network traffic analysis tool Zeek, and generates corresponding log files (such as ssl.log, dhcp.log, ssh.log, etc.) according to different protocol types. These log files contain key information such as source address, destination address, port, etc. Subsequently, the system uses the Filebeat component deployed by containerization, which is responsible for forwarding the log files generated by Zeek to the Redis database through the standardized log collection process, so as to realize the centralized management and further data analysis of logs. This system effectively improves the real-time monitoring and analysis capabilities of network traffic through the linkage of Zeek and Filebeat.

[0029] Step 6: Container-based development and deployment. The core is to start a Docker image with a certain environment, mount the configuration file and working directory into the container, map the port in the container to the host machine

[16] , configure permissions and network, and ensure normal communication with other middleware containers.

[0030] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a method according to an embodiment of the present invention; DETAILED DESCRIPTION

[0032] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0033] The improved research method of network asset mapping technology based on traffic analysis described in the present invention has the following process: Figure 1 . First, the cyberspace resources are collaboratively detected, data is acquired, and fusion analysis is performed to form a cyberspace resource library and draw a holographic map. Then, active assets are mapped through IP / port scanning, and information such as asset survival and port opening is obtained. Then, the surviving assets are further detected through fingerprint recognition to obtain more detailed asset attributes. Vulnerability scanning is used to discover the vulnerabilities of mapped assets, sniff the data packets flowing in the network, and perform traffic analysis. Finally, based on container development and deployment, the ports in the container are mapped to the host machine. Compared with TCP scanning, the method used in the present invention is more than 60% faster and consumes less system energy. Traffic analysis makes asset information more comprehensive, and the port identification accuracy reaches more than 95%.

[0034] The asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis of the present invention comprises the following steps:

[0035] Step 1: Conduct collaborative detection of cyberspace resources, obtain data, conduct fusion analysis, form a cyberspace resource library and draw a holographic map;

[0036] Step 2: Map active assets through IP / port scanning and obtain information such as asset survival and port openness;

[0037] Step 3: Further detect the surviving assets through intelligent identification to obtain more detailed asset attributes;

[0038] Step 4: Vulnerability scanning is used to discover the vulnerabilities of the mapped assets;

[0039] Step 5: Perform traffic analysis by sniffing the data packets flowing in the network;

[0040] Step 6: Based on container development and deployment, map the ports in the container to the host machine.

[0041] Based on the above steps, the proposed asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis is more than 60% faster than TCP scanning, with less system consumption. Traffic analysis makes asset information more comprehensive, and the port identification accuracy reaches more than 95%, achieving faster and more accurate scanning and analysis in this field.

[0042] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An improved method for asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis, characterized in that: include: Using the cyberspace resource mapping system as a cyclic process integrating detection, analysis, mapping and application, the cyberspace resources are collaboratively detected, data is acquired, and integrated analysis is performed to form a cyberspace resource library and draw a holographic map; Active assets are mapped through IP / port scanning, and information such as asset survival and port openness is acquired to realize functions such as ping survival detection, TCP full connection, SYN half connection, FIN, Null and UDP scanning, including docker's scan image integration of Nmap and Masscan, etc.; In terms of working mode, it can be divided into connection-oriented mode: synchronous mode, standard TCP three-way handshake, and connectionless mode: asynchronous mode, asynchronous receiving and sending mode, etc., including nmap asynchronous scanning, etc.; Using the unique fingerprint features of the asset, the detailed information of the equipment, system, service and application to which the asset belongs is identified. The system combines the advantages of Whatweb and Wappalyzer, interacts with the target Web server by constructing HTTP requests, extracts fingerprint feature information from the response data packet information, and compares it with the fingerprint database, thereby obtaining component information and version information of the Web server and application, such as the script=banner mode of the Nmap system of the present invention; Vulnerability scanning is used to discover the vulnerabilities of mapped assets, including version vulnerabilities and principle vulnerabilities. The CPE number of the application version of the mapped asset is compared with the CPE number of the version with public vulnerabilities. If the CPE is in the list of CPEs with vulnerabilities, it proves that there is a version vulnerability; principle vulnerabilities are detected through Poc plug-in scripts. If the mapped asset hits certain rules in the script and confirmation information is returned, it proves that there is a principle vulnerability; by sniffing the data packets flowing in the network, traffic analysis is performed, and the protocol and content of the data packets are analyzed to achieve passive detection; at the hardware layer, the asset type can be identified from the hardware differences of the equipment; At the network layer and transport layer, protocol header fields can be used to identify operating system and service port information; At the application layer, fingerprint detection of application components or software can be achieved by deeply analyzing the payload data or feature fields in the protocol; The core of container-based development and deployment is to start a Docker image with a certain environment, mount the configuration file and working directory into the container, map the port in the container to the host machine[16], configure permissions and networks, and ensure normal communication with other middleware containers.

2. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that: The asset mapping system is the functional implementation of the asset mapping system.

3. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that Use Docker's scan image integration, including: Nmap[12]; Masscan.

4. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that It supports multiple connection modes, including: synchronous mode, such as the standard TCP three-way handshake, and connectionless mode; asynchronous mode, such as asynchronous sending and receiving mode.

5. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that: Adopt Nmap's script=banner mode to identify the application and version of the scanned port and use the python re library to perform regular matching on the string.

6. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that: Version vulnerabilities are detected through cpe number proof, and principle vulnerabilities are detected through Poc plug-in scripts.

7. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that: Use zeek to capture the traffic passing through the network card, generate log files according to the traffic protocol, such as ssl.log / dhcp.log / ssh.log, etc., analyze the source address, destination address, port and other information, and then use filebeat to forward the log to redis.

8. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that: Use deep packet inspection technology to analyze the application layer content of traffic data packets to further refine and identify the specific application scenarios to which the assets belong and the types of protocols they use.

9. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that: The system optimizes the feature extraction process in traffic analysis based on machine learning algorithms, thereby improving the accuracy and efficiency of asset type identification.

10. The improved method of asset mapping technology based on asynchronous scanning traffic analysis and mirror traffic analysis according to claim 1 is characterized in that: The system dynamically adjusts the port scanning frequency and the sampling interval of traffic analysis through an automated scheduling mechanism to adapt to real-time changes in different network environments and ensure network stability during the surveying and mapping process.

Images